Test Driven Development via Agile Testing

I will be giving a talk in the "Next Generation Testing Conference" held in Bangalore on 21st June 2012. 


The topic and abstract is as mentioned below. See you at the conference!

Title:

Test Driven Development via Agile Testing

Abstract covering main features of the talk:

In this talk, I will cover the following topics and answer questions related to: ·       Overview of Agile Testing ·       The Test Pyramid ·       Different flavors of TDD o   BDD – Behavior Driven Development? o   ATDD – Acceptance Driven Development? o   BDT – Behavior Driven Testing? §  Difference between BDD and BDT §  Tools that support BDT §  The value proposition

vodQA NCR

vodQA NCR is being held on Saturday, June 23, 2012.


http://testing.thoughtworks.com/events/the-testing-spirit

See you there!

Keeping your test suites "green"

My article on Keeping your test suites "green" has been published in SiliconIndia's QA City. Looking forward for your comments.


Same article quoted below:


In days where we are talking and thinking more and more on how to achieve "Continuous Delivery" in our software projects, Test Automation plays an even more crucial role. 

To reap the benefits of test automation, you want to run it as often as possible. However, just putting your test automation jobs in some CI tool like Hudson / Jenkins / GO / etc., and have it run every so often is of little value, unless, the tests are passing, or the failures are identified and analyzed immediately, AND, proper action is taken based on the failures. 

If the number of failures / jobs are quite a few, then the test failure analysis and test maintenance activity takes a lot of time. Also, as a result, the development / product / project team may start losing confidence in the automation suite because the CI always shows the jobs in red. Eventually, test automation may lose priority and value, which is not a good sign. 

Before I explain a technique that may help keep your test suites "green" - and reduce the test failure analysis and maintenance time, let us understand why we get into this problem.

I have seen the functional tests failing for 3 main reasons:

1. The product has undergone some "unexpected change". As a result, the test has caught a regression bug as the product has changed when it was not supposed to.
2. The product has undergone some "expected" change and the test has not yet been updated to keep up with the new functionality.
3. There is an intermittent issue - maybe related to environment / database / browser / network / 3rd party integration / etc.
Regardless of the reason, if there is even 1 failure in your CI job, it means the whole job fails and turns "red". 

This is painful and more importantly, this does not provide the correct picture of the health of the system.

To determine the health of the system, we now need to:

• Spend dedicated time per test run to ensure the failures in the jobs are analyzed and accounted for,
• In case of genuine failures, defects are reported against the product, or,
• In case of test failures based on expected product changes, update the tests to be in accordance with the new functionality, or,
• In case of intermittent failures, rerun the test again to confirm the failure was indeed due to an intermittent issue.

This is not a trivial task to keep doing on every test run. So can something be done to keep your test suites green, and provide a true representation of what the health of the product under test?

Here is a strategy, which will reduce the manual analysis of your test runs, and, provide a better understanding into how the product conforms to what its supposed to do:

Lets make some assumptions:

1. Say, you have 5 jobs of various types in your CI
2. Each job uses a specific tag / annotation to run specific types of tests.

Now here is what you do:

1. Create appropriate commands / tasks in your test framework to execute tests with a new "failing_tests" tag / annotation.
2. Create a new job in CI - "Failing Tests" and point it to run the tests with tag / annotation "failing_tests".
3. Analyze all your existing / earlier jobs, and for all tests that have failed for any of the reasons mentioned earlier, comment out the original tag / annotation, and instead, add the tag / annotation "failing_tests" to such tests.

Run all the tests again and the now the following should be seen:

• The above steps have ensured the tests that pass, will continue to pass, with the added benefit of making the CI job green. 
• The tests that fail, will continue to fail - but in another, special "Failing Tests" CI job. 
• As a result, all the original 5 jobs you had in CI, will now turn GREEN and you just need to monitor the "Failing Tests" job.

This means now that your effort of test analysis has been reduced from 5 jobs to just 1 job. 

When a failing test passes, replace the "failing_tests" tag with the original tag back to it.

If you want to categorize the failing tests in a better way, you could potentially create separate category "Failing Tests" jobs like:

• "Failing Tests - Open Defects"
• "Failing Tests - Test updates needed"
• "Failing Tests - Intermittent / environment issues"

Regardless of your approach, the solution should be simple to implement, and you should be saving time at the end of the day, to focus on more important testing activities, instead of just analyzing the test failures.

One of my colleagues asked:
"What if a smoke test is failing? Should we move that also to a Failing Tests job?"

My answer was: 

"As with most things, you cannot apply one rule for everything. In this case also, you should not apply one strategy to all problems. As each problem is different in nature, you need to create a correct strategy that solves the problem in the best possible way.

That said, fundamentally, the smoke suite should always be "green". If there is any reason it is not, then we need to stop everything else, and make sure this is a passing test suite.

However, if you have various jobs representing the smoke suite, then you could potentially create a "Smoke - Failing Suite" on the above mentioned lines IF that helps reduce time wasted in test result analysis and provides the correct product health representation quickly, and consistently."

To summarize:

• Create a failing tests CI job and run all the failing tests as part of this job
• All existing CI jobs should turn "green"
• Monitor the failing tests and fix / update them as necessary
• If any of the passing tests fail at any point, first move them to the "Failing Tests" job to ensure the other jobs remain "green"
• When a failing test passes, move that test back from the "Failing Tests" job to the original job.

I have been profiled

SiliconIndia's QA City portal has put up my career profile on their site. You can see that here.

Theoretical Vs Practical knowledge

http://dilbert.com/strips/comic/2012-04-30/

Funny ... but on the other hand, at times this is true. Just because something has been written about, does not necessarily mean it is always true. Things change, evolve, and we need to change and move accordingly. At times, we need to flow against the tide for what we think and believe is the right thing to do.


This definitely applies to what I have seen in my career so far ... so keep thinking in innovative and creative ways - even if at times you have to swim against the tide!

Multi-tasking .... good or bad?

Many a times I end up trying to do too many things at almost the same time. I have got mixed results out of this approach.


I think off late, more often than not, I have not been too successful at juggling many things together ... this could be because of mental fatigue and burnout. 


As a result I have consciously tried to take a step away from items of relatively lower priority. This has helped me tremendously.  Also, I came across this post (http://blogs.hbr.org/schwartz/2012/03/the-magic-of-doing-one-thing-a.html)- which talks about techniques how to be more effective in your work. See if this helps you too!

vodqa Pune (17th Mar '12) videos, pictures, slides, feedback ...

Dear Testing enthusiasts,

Our recently concluded vodQA organized by ThoughtWorks Pune, on 17th March 2012 (http://testing.thoughtworks.com/events/testing-and-beyond), was a huge success. Your participation, energy, questions, thoughts, comments and feedback raised the level of this event to great heights! We thank you for that.

You can join the following groups to keep up-to-date with what’s happening in vodQA, to know when the next vodQA is happening, to connect with fellow testing enthusiasts, share thoughts related with testing, etc.

LinkedIn group: vodQA (http://linkd.in/IaHcFG)
Facebook group: vodQA (http://on.fb.me/HUPAX9)
Twitter: @vodqa

Here is what happened in this 7th edition of vodQA (4th in Pune):

Statistics:
375+ attendee registrations
35+ speaker registrations
130+ attendees

All videos from this edition of vodQA are available here: http://bit.ly/JRRTtH, and pictures are available here: http://on.fb.me/Jq4Qyh

Based on feedback received, here are the topics that attendees found most useful:
- Open Space
- Mobile Testing
- BDT  [ Behavior Driven Testing ]

Summary of feedbacks received:- video feedbacks (http://bit.ly/Jm88Qd)
- Impressive, please continue conducting such events
- I have attended vodQA for first time, it is an amazing experience
- Nice initiative taken by ThoughtWorks
- Have this event once in Quarter
- Parallel tracks (Participants found it difficult to choose one session over the other)
- Each speaker should have a Slide with email and contact details
- Should have more experienced speakers
- More time for Open Space
- Make arrangements for Parking
- Food can be improved

Want to hear more of:
- Cloud Computing, Security Testing, Malware, Ethical Hacking, Agile methodologies, mobile tech
- Current industry trends and topics

Suggestions given by attendees for next vodQA: http://bit.ly/HXGZ9I


External blogs by:
Anand Bagmar: http://bit.ly/HU78r5
Savita Munde: http://bit.ly/FOS6ll
Srinivas Chillara: http://bit.ly/IaVRAB
Sessions/Topics details:
- vodQA opening
YouTube: http://bit.ly/Jb3MPl

- Opening note by Chaitanya Nadkarny
YouTube: http://bit.ly/JckWcR

- Quiz
YouTube: http://bit.ly/I8o5sv

- Testing is Dead. Long Live Testing - Shrinivas Kulkarni
Synopsis: Last year, three leading software doctors pronounced testing dead. As we mourn the alleged demise of our craft - questions raise as what to do next? This talk analyses the meaning and impact of death of testing. The talk then deliberates on potential next steps and challenges ahead of us.
YouTube: http://bit.ly/JpXI4O
Slides: http://slidesha.re/IrBL4o

- Testing a Massively Multi-player Online Game Server - Nirmalya Sengupta, Srinivas Chilllara
Synopsis: An online game server's functional and non-functional features lead to non-standard challenges for both architecting as well as testing. This talk starts with an overview and then discusses one testing scenario in depth. We stress particularly on testing the asynchronous nature of the application's method calls. A few general approaches of testing such applications terms are alluded to at the end.
YouTube: http://bit.ly/HWO1NP
Slides: http://slidesha.re/JRZKHQ

- Virtualization Impact on Software Testing - Parthasarthi T
Synopsis: Virtualization Impact on Software Testing
YouTube: http://bit.ly/HWjFrj
Slides: http://slidesha.re/Jq4bNe

- Mobile Testing: Challenges and Solutions - Ashwini Phalle
Synopsis: Different testing requirements that mobile applications have, challenges and solutions.
Challenges:
1. Complex mobile testing matrix, Expensive test environment
2. Repetitive testing
3. Mobile testing for devices located at various locations
Solutions:
1. Risk Based Testing approach
2. Using Mobile device emulators
3. Use of Automation tools
4. Leveraging external services
YouTube: http://bit.ly/Jb4hZw
Slides: http://slidesha.re/Jb9Rer

- Open Space discussions
YouTube: http://bit.ly/Jb4k7O

- The Marshmallow Challenge - Sneha Kadam
Synposis: This is a fun and instructive design exercise that encourages teams to experience simple but profound lessons in collaboration, innovation & creativity. It challenges you to find hidden assumptions in business requirements & learn to Fail-Fast-Fail-Often! In 18 minutes, teams must build the tallest free-standing structure out of spaghetti, tape, string and one marshmallow which must be on top.
YouTube: Part 1: http://bit.ly/HR4XiX
YouTube: Part 2: http://bit.ly/J7t8zJ

- Mobile Testing: In and Out - Sudeep Somani
YouTube: http://bit.ly/JifEQH

- BDT (Behaviour Driven Testing) - Anand Bagmar
Synopsis: What is Behavior Driven Testing (BDT)? How does it differ from Behavior Driven Development? What tools support this kind of testing? The value proposition BDT offers.
YouTube: http://bit.ly/HUSuuY
Slides: http://slidesha.re/I69BNK

- Code Coverage of Function Testing Automation Scripts - Aakash Tyagi
Synopsis: Challenge As the product is a vast product that provides so regression suite of this was very big. It was taking about 14 days to execute and with every release it was increasing. The main challenge was to keep regression suite comprehensive as well small so that it can be executed many time. Solution Emma was used to find code coverage of product code then redesign the regression suite.
Slides: http://slidesha.re/HWQJCQ

- Negative Testing, in a positive vein - Srinivas Chillara
Synopsis: How to think about "negative testing", and why it may not be truly negative.
YouTube: http://bit.ly/IaKOr4

- Virtual Communication and Testers - Archana Dhingra
Synopsis: What is Virtual communication and its importance in IT industry. - The common mistakes we all commit while communicating in a Virtual environment - How to effectively manage and communicate with virtual teams. - Conflict resolution in a Virtual setup.
YouTube: http://bit.ly/Jq3uUc

- Automation Reusable Framework based on QC - Vysali Alaparthi
Synopsis: About ART:A hybrid framework named as ART (Automation Reusable Test) is used for end-to-end automation as ART framework supports automation of web, windows, and AS/400 applications. ART framework uses automation tool owned by HP i.e. Quick Test Professional (QTP) for execution of automated keyword-driven test scripts.Key Achievements: Efforts involved in test cases/scripts integration has reduced.
YouTube: http://bit.ly/Jlb8MH
Slides: http://slidesha.re/IrBtua

Closing note: Shalabh Varma
YouTube: http://bit.ly/Jb9fFO

Looking forward for your continued comments, feedback, thoughts and support to make vodQA more successful, and the QA community more vibrant and connected!

See you in the next vodQA.

Thank you.

vodQA Team.
vodqa-pune@thoughtworks.com

vodQA Pune - another big success

Thoughtworks Pune organized another successful vodQA on 17th March 2012

Statistics:

375+ attendee registrations

35+ speaker registrations

100+ external attendees

20+ Thoughtworkers

Sessions/Topics covered:

- Code Coverage of Function Testing Automation Scripts - Aakash Tyagi
- Mobile Testing: Challenges and Solutions - Ashwini Phalle
- Business Analysis - Beyond Technical & Communication Skills - Anil Dagia
- Testing is Dead. Long Live Testing - Shrinivas Kulkarni
- Testing a Massively Multi-player Online Game Server - Nirmalya Sengupta, Srinivas Chilllara
- Behaviour Driven Testing - Anand Bagmar
- Virtualisation Impact on Software Testing - Parthasarthi T

- Negative Testing, in a positive vein - Srinivas Chillara
- Virtual Communication and Testers - Archana Dhingra
- Automation Reusable Framework based on QC - Vysali Alaparthi

Workshops:

- Mobile Testing: In and Out - Sudeep Somani
- The Marshmallow Challenge - Sneha Kadam

Some feedbacks received:

• Impressive, please continue conducting such events
• I have attended vodQa for first time, its Amazing experince
• Some forum about Innovation & "Testing process"
• Nice initiative taken by Thoughtworks
• Have this event once in Quarter

External blog: http://savitamunde.wordpress.com/2012/03/18/thought-provoking-vodqa-pune-a-testing-conference/

Please join our LinkedIn Group - vodQA and our FaceBook group - vodQA..

We will be uploading the pictures, videos and slides to a common place soon for everyone to see.

Thank you all who attended for making this a successful event.

vodQA Pune - registrations open

The registrations for attendee and speakers for vodQA Pune on 17th March 2012 is now open. Looking forward to see you there :)

vodQA Chennai starts off with a century!

I attended vodQA in Chennai on 21st Jan 2012. The event was great. Over 100 passionate testers from Chennai testing community turned up and made sure people presenting were on their toes with excellent questions and great interactions.


One of the participants already blogged about it here. She raises valid observations - and I wish I had the opportunity to speak with her directly to address some of the questions / concerns she raised. 


In this session, I presented a topic - "What is WAAT?" based on my open-source project - WAAT. The slides used in this session are available here.


I am already looking forward to the next vodQA in Chennai. For now, I am preparing for vodQA Bangalore - "Agile Testing for Teams and Enterprises". on Feb 11, and then vodQA Pune on Mar 17.