Monday, April 21, 2014

What are the criteria for determining success for Test Automation?

Test Automation is often thought of as a silver bullet that will solve the Teams testing problems. As a result there is a heavy investment of time, money, people in building an Automated Suite of Tests of different types which will solve all problems.

Is that really the case? We know the theoretical aspect of what is required to make Test Automation successful for the team. I want to know from practical perspective, with the context what worked or not for you.

I am currently writing about "Building an Enterprise-class Test Automation Framework", and am gathering experience reports based on what others in the industry have seen.
 
I am looking for people to share stories from their current / past experiences of full or limited success of Test Automation, answering the below questions (at a minimum):
 

Context:

  • What is the product under test like? (small / med / large / enterprise) (web / desktop / mobile / etc.)
  • How long is the the Test Automation framework envisioned to be used? (few months, a year or two, more than a few years, etc.)
  • What is the team (complete and test automation) size?
  • Is the testing team co-located or distributed? 
  • What are the tools / technologies used for testing?
  • Are the skills and capabilities uniform for the team members?
  • Is domain a factor for determining success criteria?  

Framework related:

  • What are the factors determining the success / failure of Test Automation implementation?
  • What worked for you? 
  • What did not work as well?
  • What could have been different in the above to make Test Automation a success story?
  • What are the enablers in your opinion to make Test Automation successful?
  • What are the blockers / anchors in your opinion that prevented Test Automation from being successful?
  • Does it matter if the team is working in Waterfall methodology or Agile methodology?
Anything else you can think of?