We all know that automation is one of the key enablers for those on the CI-CD journey.
Most teams are:
implementing automation
talking about its benefits
up-skilling themselves
talking about tooling
etc.
However, many a times I feel we are blinded because of the theoretical value test automation provides, or because everyone says it adds value, or because of the shiny tools / tech-stacks we get to use , or ...
To try and understand more about this, can you answer the below questions?
In your experience, or in your current project:
Does your functional automation really add value?
What makes you say it does / or does not?
How long does it take for tests to run and generate reports?
In most cases, the product-under-test is available on multiple platforms – ex: Android & iOS Native, and on Web. In such cases, for the same scenario that needs to be automated, is the test implemented once for all platforms, or once per platform?
How easy is it to debug and get to the root cause of failures?
How long does it take to update an existing test?
How long does it take to add a new test?
Do your tests run automatically via CI on a new build, or do you need to “trigger” the same?
What is the test passing percentage?
Do you “rerun” the failing tests to see if this was an intermittent issue?
Is there control on the level of parallel execution and switch to sequential execution based on context?
How clean & DRY is the code?
In my experience, unfortunately most of the functional automation that is built is: · not optimal · not fit-for-purpose · does not run fast enough · gives inconsistent feedback, hence unreliable
Hence, for the amount of effort invested in implementing automation,
Are you really getting the value from this activity?
Here are the key takeaways mentioned in the article -
There are many criteria to be considered before building framework / selecting tools for Functional Test Automation
It is very important to prioritise framework / tools capabilities needed for the software-under-test
A good, scalable Test Automation Framework that provides fast and reliable feedback to the team enables collaboration and CI/CD
Debugging / RCA (root cause analysis) and support for libraries / tools used is an afterthought in most cases. Do not fall in that trap.
There are some promising commercial tools that fit seamlessly in the Agile way of working. Depending on the complete context, these tools may be a good choice over building your own framework for Functional Automation.
On 29th November 2015, I spoke in Agile Noida on "Enabling Continuous Delivery (CD) in Enterprises with Testing". Below is the abstract, slides and video from the talk.
Abstract
The key objectives of any organization is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!
There are various practices that organizations need to implement to enable CD. Changes in requirements (a reality in all projects) needs to be managed better. Along with this, processes and practices need to be tuned based on the team capability, skills and distribution.
Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful.But, this is tricky and requires a lot of discipline, rigor and hard work by all the team members involved the product delivery.
All the challenges faced in smaller organizations get amplified when it comes to Enterprises. There are various reasons to this - but most common reasons are - scale, complexity of the domain, complexity of the integrations (to internal / external system), involvement of various partners / vendors, long product life-cycles, etc.
In such situations, the Testing complexity and challenges also increase exponentially!
Learn, via a case study of an Enterprise, a large Bank, the Testing approach required to take them on the journey to achieving CD.
