Thursday, May 17, 2012

Keeping your test suites "green"

My article on Keeping your test suites "green" has been published in SiliconIndia's QA City. Looking forward for your comments.


Same article quoted below:


In days where we are talking and thinking more and more on how to achieve "Continuous Delivery" in our software projects, Test Automation plays an even more crucial role. 

To reap the benefits of test automation, you want to run it as often as possible. However, just putting your test automation jobs in some CI tool like Hudson / Jenkins / GO / etc., and have it run every so often is of little value, unless, the tests are passing, or the failures are identified and analyzed immediately, AND, proper action is taken based on the failures. 

If the number of failures / jobs are quite a few, then the test failure analysis and test maintenance activity takes a lot of time. Also, as a result, the development / product / project team may start losing confidence in the automation suite because the CI always shows the jobs in red. Eventually, test automation may lose priority and value, which is not a good sign. 

Before I explain a technique that may help keep your test suites "green" - and reduce the test failure analysis and maintenance time, let us understand why we get into this problem.

I have seen the functional tests failing for 3 main reasons:

1. The product has undergone some "unexpected change". As a result, the test has caught a regression bug as the product has changed when it was not supposed to.
2. The product has undergone some "expected" change and the test has not yet been updated to keep up with the new functionality.
3. There is an intermittent issue - maybe related to environment / database / browser / network / 3rd party integration / etc.
Regardless of the reason, if there is even 1 failure in your CI job, it means the whole job fails and turns "red". 

This is painful and more importantly, this does not provide the correct picture of the health of the system.

To determine the health of the system, we now need to:

• Spend dedicated time per test run to ensure the failures in the jobs are analyzed and accounted for,
• In case of genuine failures, defects are reported against the product, or,
• In case of test failures based on expected product changes, update the tests to be in accordance with the new functionality, or,
• In case of intermittent failures, rerun the test again to confirm the failure was indeed due to an intermittent issue.

This is not a trivial task to keep doing on every test run. So can something be done to keep your test suites green, and provide a true representation of what the health of the product under test?

Here is a strategy, which will reduce the manual analysis of your test runs, and, provide a better understanding into how the product conforms to what its supposed to do:

Lets make some assumptions:


1. Say, you have 5 jobs of various types in your CI
2. Each job uses a specific tag / annotation to run specific types of tests.

Now here is what you do:

1. Create appropriate commands / tasks in your test framework to execute tests with a new "failing_tests" tag / annotation.
2. Create a new job in CI - "Failing Tests" and point it to run the tests with tag / annotation "failing_tests".
3. Analyze all your existing / earlier jobs, and for all tests that have failed for any of the reasons mentioned earlier, comment out the original tag / annotation, and instead, add the tag / annotation "failing_tests" to such tests.

Run all the tests again and the now the following should be seen:

• The above steps have ensured the tests that pass, will continue to pass, with the added benefit of making the CI job green
• The tests that fail, will continue to fail - but in another, special "Failing Tests" CI job. 
• As a result, all the original 5 jobs you had in CI, will now turn GREEN and you just need to monitor the "Failing Tests" job.

This means now that your effort of test analysis has been reduced from 5 jobs to just 1 job. 

When a failing test passes, replace the "failing_tests" tag with the original tag back to it.

If you want to categorize the failing tests in a better way, you could potentially create separate category "Failing Tests" jobs like:

• "Failing Tests - Open Defects"
• "Failing Tests - Test updates needed"
• "Failing Tests - Intermittent / environment issues"

Regardless of your approach, the solution should be simple to implement, and you should be saving time at the end of the day, to focus on more important testing activities, instead of just analyzing the test failures.

One of my colleagues asked:
"What if a smoke test is failing? Should we move that also to a Failing Tests job?"


My answer was: 


"As with most things, you cannot apply one rule for everything. In this case also, you should not apply one strategy to all problems. As each problem is different in nature, you need to create a correct strategy that solves the problem in the best possible way.

That said, fundamentally, the smoke suite should always be "green". If there is any reason it is not, then we need to stop everything else, and make sure this is a passing test suite.

However, if you have various jobs representing the smoke suite, then you could potentially create a "Smoke - Failing Suite" on the above mentioned lines IF that helps reduce time wasted in test result analysis and provides the correct product health representation quickly, and consistently."

To summarize:

• Create a failing tests CI job and run all the failing tests as part of this job
• All existing CI jobs should turn "green"
• Monitor the failing tests and fix / update them as necessary
• If any of the passing tests fail at any point, first move them to the "Failing Tests" job to ensure the other jobs remain "green"
• When a failing test passes, move that test back from the "Failing Tests" job to the original job.

I have been profiled

SiliconIndia's QA City portal has put up my career profile on their site. You can see that here.

Monday, April 30, 2012

Theoretical Vs Practical knowledge

http://dilbert.com/strips/comic/2012-04-30/

Funny ... but on the other hand, at times this is true. Just because something has been written about, does not necessarily mean it is always true. Things change, evolve, and we need to change and move accordingly. At times, we need to flow against the tide for what we think and believe is the right thing to do.


This definitely applies to what I have seen in my career so far ... so keep thinking in innovative and creative ways - even if at times you have to swim against the tide!

Multi-tasking .... good or bad?

Many a times I end up trying to do too many things at almost the same time. I have got mixed results out of this approach.


I think off late, more often than not, I have not been too successful at juggling many things together ... this could be because of mental fatigue and burnout. 


As a result I have consciously tried to take a step away from items of relatively lower priority. This has helped me tremendously.  Also, I came across this post (http://blogs.hbr.org/schwartz/2012/03/the-magic-of-doing-one-thing-a.html)- which talks about techniques how to be more effective in your work. See if this helps you too!

Friday, April 20, 2012

vodqa Pune (17th Mar '12) videos, pictures, slides, feedback ...

Dear Testing enthusiasts,

Our recently concluded vodQA organized by ThoughtWorks Pune, on 17th March 2012 (http://testing.thoughtworks.com/events/testing-and-beyond), was a huge success. Your participation, energy, questions, thoughts, comments and feedback raised the level of this event to great heights! We thank you for that.

You can join the following groups to keep up-to-date with what’s happening in vodQA, to know when the next vodQA is happening, to connect with fellow testing enthusiasts, share thoughts related with testing, etc.

LinkedIn group: vodQA (http://linkd.in/IaHcFG)
Facebook group: vodQA (http://on.fb.me/HUPAX9)
Twitter: @vodqa

Here is what happened in this 7th edition of vodQA (4th in Pune):

Statistics:
375+ attendee registrations
35+ speaker registrations
130+ attendees

All videos from this edition of vodQA are available here: http://bit.ly/JRRTtH, and pictures are available here: http://on.fb.me/Jq4Qyh

Based on feedback received, here are the topics that attendees found most useful:
- Open Space
- Mobile Testing
- BDT  [ Behavior Driven Testing ]

Summary of feedbacks received:- video feedbacks (http://bit.ly/Jm88Qd)
- Impressive, please continue conducting such events
- I have attended vodQA for first time, it is an amazing experience
- Nice initiative taken by ThoughtWorks
- Have this event once in Quarter
- Parallel tracks (Participants found it difficult to choose one session over the other)
- Each speaker should have a Slide with email and contact details
- Should have more experienced speakers
- More time for Open Space
- Make arrangements for Parking
- Food can be improved

Want to hear more of:
- Cloud Computing, Security Testing, Malware, Ethical Hacking, Agile methodologies, mobile tech
- Current industry trends and topics

Suggestions given by attendees for next vodQA: http://bit.ly/HXGZ9I


External blogs by:
Anand Bagmar: http://bit.ly/HU78r5
Savita Munde: http://bit.ly/FOS6ll
Srinivas Chillara: http://bit.ly/IaVRAB

Sessions/Topics details:
- vodQA opening
YouTube: http://bit.ly/Jb3MPl

- Opening note by Chaitanya Nadkarny
YouTube: http://bit.ly/JckWcR

- Quiz
YouTube: http://bit.ly/I8o5sv

- Testing is Dead. Long Live Testing - Shrinivas Kulkarni
Synopsis: Last year, three leading software doctors pronounced testing dead. As we mourn the alleged demise of our craft - questions raise as what to do next? This talk analyses the meaning and impact of death of testing. The talk then deliberates on potential next steps and challenges ahead of us.
YouTube: http://bit.ly/JpXI4O
Slides: http://slidesha.re/IrBL4o

- Testing a Massively Multi-player Online Game Server - Nirmalya Sengupta, Srinivas Chilllara
Synopsis: An online game server's functional and non-functional features lead to non-standard challenges for both architecting as well as testing. This talk starts with an overview and then discusses one testing scenario in depth. We stress particularly on testing the asynchronous nature of the application's method calls. A few general approaches of testing such applications terms are alluded to at the end.
YouTube: http://bit.ly/HWO1NP
Slides: http://slidesha.re/JRZKHQ

- Virtualization Impact on Software Testing - Parthasarthi T
Synopsis: Virtualization Impact on Software Testing
YouTube: http://bit.ly/HWjFrj
Slides: http://slidesha.re/Jq4bNe

- Mobile Testing: Challenges and Solutions - Ashwini Phalle
Synopsis: Different testing requirements that mobile applications have, challenges and solutions.
Challenges:
1. Complex mobile testing matrix, Expensive test environment
2. Repetitive testing
3. Mobile testing for devices located at various locations
Solutions:
1. Risk Based Testing approach
2. Using Mobile device emulators
3. Use of Automation tools
4. Leveraging external services
YouTube: http://bit.ly/Jb4hZw
Slides: http://slidesha.re/Jb9Rer

- Open Space discussions
YouTube: http://bit.ly/Jb4k7O

- The Marshmallow Challenge - Sneha Kadam
Synposis: This is a fun and instructive design exercise that encourages teams to experience simple but profound lessons in collaboration, innovation & creativity. It challenges you to find hidden assumptions in business requirements & learn to Fail-Fast-Fail-Often! In 18 minutes, teams must build the tallest free-standing structure out of spaghetti, tape, string and one marshmallow which must be on top.
YouTube: Part 1: http://bit.ly/HR4XiX
YouTube: Part 2: http://bit.ly/J7t8zJ

- Mobile Testing: In and Out - Sudeep Somani
YouTube: http://bit.ly/JifEQH

- BDT (Behaviour Driven Testing) - Anand Bagmar
Synopsis: What is Behavior Driven Testing (BDT)? How does it differ from Behavior Driven Development? What tools support this kind of testing? The value proposition BDT offers.
YouTube: http://bit.ly/HUSuuY
Slides: http://slidesha.re/I69BNK

- Code Coverage of Function Testing Automation Scripts - Aakash Tyagi
Synopsis: Challenge As the product is a vast product that provides so regression suite of this was very big. It was taking about 14 days to execute and with every release it was increasing. The main challenge was to keep regression suite comprehensive as well small so that it can be executed many time. Solution Emma was used to find code coverage of product code then redesign the regression suite.
Slides: http://slidesha.re/HWQJCQ

- Negative Testing, in a positive vein - Srinivas Chillara
Synopsis: How to think about "negative testing", and why it may not be truly negative.
YouTube: http://bit.ly/IaKOr4

- Virtual Communication and Testers - Archana Dhingra
Synopsis: What is Virtual communication and its importance in IT industry. - The common mistakes we all commit while communicating in a Virtual environment - How to effectively manage and communicate with virtual teams. - Conflict resolution in a Virtual setup.
YouTube: http://bit.ly/Jq3uUc

- Automation Reusable Framework based on QC - Vysali Alaparthi
Synopsis: About ART:A hybrid framework named as ART (Automation Reusable Test) is used for end-to-end automation as ART framework supports automation of web, windows, and AS/400 applications. ART framework uses automation tool owned by HP i.e. Quick Test Professional (QTP) for execution of automated keyword-driven test scripts.Key Achievements: Efforts involved in test cases/scripts integration has reduced.
YouTube: http://bit.ly/Jlb8MH
Slides: http://slidesha.re/IrBtua

Closing note: Shalabh Varma
YouTube: http://bit.ly/Jb9fFO

Looking forward for your continued comments, feedback, thoughts and support to make vodQA more successful, and the QA community more vibrant and connected!

See you in the next vodQA.

Thank you.

vodQA Team.
vodqa-pune@thoughtworks.com

Thursday, March 22, 2012

vodQA Pune - another big success


Thoughtworks Pune organized another successful vodQA on 17th March 2012

Statistics:
375+ attendee registrations
35+ speaker registrations
100+ external attendees
20+ Thoughtworkers

Sessions/Topics covered:
- Code Coverage of Function Testing Automation Scripts - Aakash Tyagi
- Mobile Testing: Challenges and Solutions - Ashwini Phalle
- Business Analysis - Beyond Technical & Communication Skills - Anil Dagia
- Testing is Dead. Long Live Testing - Shrinivas Kulkarni
- Testing a Massively Multi-player Online Game Server - Nirmalya Sengupta, Srinivas Chilllara
- Behaviour Driven Testing - Anand Bagmar
- Virtualisation Impact on Software Testing - Parthasarthi T
- Negative Testing, in a positive vein - Srinivas Chillara
- Virtual Communication and Testers - Archana Dhingra
- Automation Reusable Framework based on QC - Vysali Alaparthi

Workshops:
- Mobile Testing: In and Out - Sudeep Somani
- The Marshmallow Challenge - Sneha Kadam

Some feedbacks received:
  • Impressive, please continue conducting such events
  • I have attended vodQa for first time, its Amazing experince
  • Some forum about Innovation & "Testing process"
  • Nice initiative taken by Thoughtworks
  • Have this event once in Quarter



Please join our LinkedIn Group - vodQA and our FaceBook group - vodQA..

We will be uploading the pictures, videos and slides to a common place soon for everyone to see.

Thank you all who attended for making this a successful event.

Monday, January 23, 2012

vodQA Chennai starts off with a century!

I attended vodQA in Chennai on 21st Jan 2012. The event was great. Over 100 passionate testers from Chennai testing community turned up and made sure people presenting were on their toes with excellent questions and great interactions.


One of the participants already blogged about it here. She raises valid observations - and I wish I had the opportunity to speak with her directly to address some of the questions / concerns she raised. 


In this session, I presented a topic - "What is WAAT?" based on my open-source project - WAAT. The slides used in this session are available here.


I am already looking forward to the next vodQA in Chennai. For now, I am preparing for vodQA Bangalore - "Agile Testing for Teams and Enterprises". on Feb 11, and then vodQA Pune on Mar 17.

Tuesday, January 3, 2012

[Date Updated] Announcing the next vodQA event in Pune

[Date updated to 17th March 2012]


After a long delay, ThoughtWorks is happy to announce the next edition of vodQA - THE TESTING SPIRIT! on 17th March, 2012 in ThoughtWorks Pune office.


Watch this space for more details.


Contact me if you are interested in helping make this a great event!


Thanks.


Anand

Assertions and Validations in Page Objects

A colleague recently asked me a very nice set of questions - 

  • "Have any of you designed tests with assertions happening in page objects and not in the tests? 
  • If yes, have you faced any specific problem with this approach? 
  • What would be the drawback in moving the assertions inside the page's methods."

Here are my thoughts on this.

Test Framework Design

I follow a few principles when designing my test framework:
  • Test code should be of Production Quality!
  • Since the test code should be of Production Quality, it means it needs to be designed and built using design patterns.
  • This well-designed test framework should have proper abstraction layers. These abstraction layers help in many different ways:
    • Decouple test specification from test implementation
    • Provides greater level of re-usability
    • Easier in re-factoring
    • Easier to maintain and evolve the framework
    • Easier for all team members to ramp-up and work in a collaborative way on specific abstraction layers.
  • Evolve your framework functionality and implementation. Keeping the end goal in mind, develop the framework as per requirements at that point in time. Do not attempt to build all the functionality in a single shot. More likely than not, you will end up building something that is not going to satisfy the future requirement.
See the diagram below for reference on different possible layers of a Test Framework.



What is Page-Object pattern?


Page-Object pattern is one of the powerful ways of designing a good, reusable, extensible and maintainable test framework. 

This article as great explanation and examples of Page Objects (http://code.google.com/p/selenium/wiki/PageObjects)

Assertions and validations in Page Objects?

The page object is a code representation of the actual page. It has accessors and modifiers (getters and setters) for various objects in that page. It only knows how to perform actions on the page object, and retrieve data / values from the page. 


If there is any problem in the page under test, or the page object representing the page, then the test will fail automatically (in most cases) because of functionality mismatch. 


Assertions / verifications are essentially business rules for the product under test.


The page-object should not have assertions or verifications in them. The business rules of the product do not belong in the page object layer, but instead in the layer above it.

In many cases the business rules remain the same where as the underlying implementation evolves. If you have the changes isolated and decoupled, then updating the framework becomes easy and much quicker. this also makes more sense in larger and distributed projects where everyone may not be on the same page.



I refrain from having any form of assertions in the page's object. It mixes the pure implementation of visibility of the page's functionality with the business logic. This in turn makes both, the framework and the tests brittle.

The impact of having these rules and guidelines of how the test framework is structured is greatly seen as the framework matures and when new people come on the team, the learning curve is not as steep.

If the project is small, or if the test framework is going to be thrown away after some time,  then you can probably get away by building the framework any which way you want. 



Thinking that I will take the easy way out for now and then will come back to "do this right" is a trap!!! More often than not, you are never going to get time to come back and make things right. So might as well, spend a little extra effort in the beginning and build your test framework the right way! Remember - "A stitch in time saves nine"!

Tuesday, November 29, 2011

Effective Strategies for Distributed Testing - slides available

I finally figured a way around the problem PowerPoint was giving me in converting the slides for this presentation to pdf format. 


The slides are now uploaded in slideshare.net  and available here.


The video recording of the webinar is available here.

Wednesday, November 16, 2011

vodQA - NCR

I am very happy and proud to share that vodQA conference is going national. 


ThoughtWorks will be hosting its first vodQA out of Pune ... this time in Gurgaon on 3rd December 2010.


I will be pairing with Manish Kumar on a topic "Distributed Agile Testing for Enterprises"


See this page for more information.

Thursday, November 10, 2011

Effective Strategies of Distributed Testing - recording available

On 9th November, 2011, I presented my first webinar with my colleague - Manish Kumar, on the topic "Effective Strategies for Distributed Testing".


We spoke based on our experiences and shared tips and strategies on various different themes, like 
- agile testing principles that matter!
- need of distributed teams,
- testing challenges in distributed teams,
- mindset, 
- communication, 
- collaboration, 
- tools & technologies,
- ATDD,
- test automation,
- and many others ...


All distributed teams are working on the same product ... hence it is extremely important to treat all these distributed teams as ONE TEAM!


Watch the video recording of this webinar available here. 


(http://testing.thoughtworks.com/events/effective-strategies-distributed-testing).

Sunday, November 6, 2011

Effective Strategies for Distributed Testing - webinar

Come, join Manish Kumar and me for a webinar on 9th November, 2011 on "Effective Strategies for Distributed Testing". 


We will be sharing tips and techniques on how you can make testing in distributed teams more effective.


More details on the webinar are available here.

Thursday, September 22, 2011

Asking the right question


This Dilbert strip says it all!!


For those not able to see the link it correctly, here it is:






If you don't ask the right question, the topic can digress in any direction ... resulting in waste of time and effort of all involved.


So remember - its not just important to ask questions, but it is more important to ask the right questions!!

Monday, August 29, 2011

Does extrapolation in estimation work?

How do you estimate test automation efforts for building a regression suite for functionality that is already in Production?


The approach I am following is:
> Get some level of understanding about the domain
> Look at a subset of existing test cases of varying complexities and sizes
> Estimate the effort for each of them and also identify the complexity involved in each, with the risks and assumptions.
> Identify some obvious spikes / tech tasks that will be needed
> Extrapolate the estimates
> Add 10-15% buffer in the estimates for unknowns.


And presto!! .... I have my estimates for the task on hand.


Is this approach right? 
Is extrapolating the estimates going to give valid results?
What can be different / better ways of getting the results required?

Thursday, August 18, 2011

To test or not to test ... do you ask yourself this question?

For people involved in Test Automation, I have seen that quite a few of us get carried away and start automating anything and everything possible. This inherently happens at the expense of ignoring / de-prioritizing other equally important activities like Manual (exploraratory / ad-hoc) testing.

Also, as a result, the test automation suite gets very large and unmaintainable, and in all possibilities, not very usable, with a long feedback cycle.

I have found a few strategies work well for me when I approach Test Automation:
  • Take a step back and look at the big picture.
  • Ask yourself the question - "Should I automate this test or not? What value will the product get by automating this?"
  • Evaluate what test automation will truly provide good and valuable feedback.
  • Based on the evaluation, build and evolve your test automation suite.
One technique that is simple and quick to evaluate what tests should be automated or not, is to do a Cost Vs Value analysis of your identified tests using the graph shown below.


This is very straight forward to use.

To start off, analyze your tests and categorize them in the different quadrants of the above graph.
  1. First automate tests that provide high value, and low cost to build / maintain = #1 in the graph. This is similar to the 80/20 rule.
  2. Then automate tests that provide high value, but have a high cost to build / maintain = #2 in the graph.
  3. Beyond this, IF there is more time available, then CONSIDER automating tests that have low value, and low cost. I would rather better utilize my time at this juncture to do manual exploratory testing of the system.
  4. DO NOT automate tests that have low value and high cost to build / maintain.

    Thursday, August 11, 2011

    How do you create / generate Test Data?

    Testing (manual or automation) depends upon data being present in the product.


    How do you create Test Data for your testing?


    • Manually create it using the product itself?
    • Directly use SQL statements to create the data in the DB (required knowledge of the schema)?
    • Use product apis / developer help to seed data directly? (using product factory objects)?
    • Use production snapshot / production-like data?
    • Any other way?

    How do you store this test data?


    • Along with the test (in code)?
    • In separate test data files - eg: XML, yml, other types of files?
    • In some specific tools like Excel?
    • In a separate test database?
    • Any other way?

    Do you use the same test data that is also being used by the developers for their tests?


    What are the advantages / disadvantages you have observed in the approach you use (or any other methods)?


    Looking forward to knowing your strategy on test data creation and management!

    Thursday, August 4, 2011

    Lessons from a 1-day training engagement - a Trainer perspective

    I was a Trainer for bunch of smart QAs recently. The training was about "Agile QA". This term is very vague, and if you think about it more deeply, it is also a very vast topic.

    The additional complexity was that this training was to be done in 1 day.

    So we broke it down to what was really essential to be covered in this duration, and what could be covered in this duration.

    We came down to 2 fundamental things:
    1. How can the QA / Test team be agile and contribute effectively to Agile projects?
    2. How can you be more effective in Test Automation - and that too start this activity in parallel with Story development?

    So we structured our thoughts, discussions and presentations around this.

    At the end of the exhausting day (both, for the Trainers as well as the Trainees), a couple of things stood out (not in any particular order):
    • What is the "right" Agile QA process?
    • Roles and responsibilities of a QA on an Agile team
    • Sharing case studies and the good & not-so-good experiences around them
    • Effectiveness of process and practices
    • Value of asking difficult questions at the right time
    • Taboo game - playing it, and reflecting on its learnings
    • What to automate and what NOT to automate?
    • Discussion around - How do you create / generate Test Data?

    Tuesday, July 26, 2011

    WAAT-Ruby - ready for use

    WAAT-Ruby is now ready for use. 

    Project hosted on github - http://github.com/anandbagmar/WAAT-Ruby
    WAAT-Ruby gem available for download from here.
    Documentation is available (on WAAt-Ruby wiki pages) here.

    Since WAAT-Ruby uses WAAT-Java under the covers, I have kept the same version numbers for both platforms. The latest version is 1.4.

    I have not yet pushed it out on rubygems.org. Will update once that is done.

    So far I have tested this on the following environments:
    • Windows 7 - 64-bit with Ruby 1.8.6
    • RHEL 6 - 64-bit with Ruby 1.8.6 (I had difficulty in getting jpcap deployed on this environment). But once that was done, WAAT worked smoothly out of the box.
    • Ubuntu 10.x - 32-bit with Ruby 1.8.7
    • Ubuntu 10.x - 32-bit with Ruby 1.9.1
    One important note:
    If you are using WAAT (Java or Ruby) on any Linux-based environment, please note the Jpcap requirement for execution.
    WAAT uses Jpcap for capturing network packets. Jpcap needs administrative privileges to do this work. See the Jpcap FAQs for more information about the same.


    For all WAAT related blog posts, click here.

    Wednesday, July 20, 2011

    WAAT - Ruby .... are we there yet?

    The WAAT ruby gem is almost ready. My colleagues are helping testing it out in different environments and am updating the documentation accordingly.

    Once done, this will be available as a Ruby gem from WAAT-Ruby github project, and also from rubygems.org.

    Contact me if you are interested in trying this out before release.

    Thursday, July 14, 2011

    What is your expiry date?

    Recently when doing some online transaction using my credit card, something struck me ... I realized that the form asking for my credit card information was quite weird, and probably incorrect.

    Here is a sample layout of what I am talking about:


    Here, I am asked to enter the details in this order:
    • Credit card number 
    • Card holder's name
    • Expiry date 
    • And so on ...
    As I was entering the information, I ended up questioning myself ... whose Expiry Date??? The card's or mine??? 

    Simply based on the flow of information asked for, it is quite easy to associate the Expiry Date with the earlier field - the Card holder's name. Right?

    Wouldn't the layout be better this way instead:
    • Card Holder name
    • Card number
    • Expiry date
    • CVV number

    Or, another way can be:
    • Card number
    • Expiry date
    • CVV number
    • Card Holder name


    I checked all my 10-15 (credit / debit / membership) cards that I have. All of them have the issue date / expiry date / validity period associated with the card number, and not the card holder's name.

    This leads me to believe that no one did a usability check, or, in this context, shall we call it a reality check when designing the credit card form like the one shown above. 

    I would have not let this design / layout get through. What would you do?

    Thursday, July 7, 2011

    Ruby Test Automation Framework Technology Stack Survey

    WAAT  - Web Analytics Automation Testing Framework is currently available for java based Test Automation Frameworks. (http://essenceoftesting.blogspot.com/search/label/waat)

    I am now working on making this available as a Ruby gem.

    In order to support WAAT for a good-mix of test environments, I would like to understand the different environments and technology stacks that are typically used by teams in their Test Automation Framework.

    Thank you for your time and help in providing this information.



    Wednesday, July 6, 2011

    RubyMine (and Cucumber) caching issue

    I use RubyMine to write and implement my Cucumber features on Linux.
    I have noticed one weird behavior at times.

    Though my step definition is correct, and the test also runs fine, RubyMine flags the step as not implemented. For some reason, it is not able to find the corresponding implementation in the .rb step definition file.

    On a haunch, I selected the "Invalidate Cache" in RubyMine's File menu, and selected the "Invalidate and Restart" option. Presto .... things started working properly again. 

    Now I am wondering why did the RubyMine cache get messed up in the first place .....

    Monday, June 27, 2011

    WAAT for Ruby on its way

    I have started work on creating a Ruby gem for WAAT. This is going to sit on top of the version created for Java. Hopefully will be able to get it out soon.

    Watch this space for more information.

    Friday, June 24, 2011

    Test Trend Analyzer (TTA)

    There are many tools and utilities that provide ways to do test result reporting and analysis of those results. However, I have not found a good, generic way of doing some Trend Analysis of those results. 


    Why do I need to Trend Analysis of the test results?

    Long(er) duration projects / enterprise products need to know the state of the quality of the product over time. One also may need to know various other metrics around the testing - like number of tests, pass %, failure %, etc. over time.

    The reports I have seen are very good about analyzing the current test results. However, I have not really come across a good generic tool that can be used in most environments for the Test Trend Analysis over a period of time.

    I am thinking about developing a tool which can address this space. I call this - the Test Trend Analyzer (TTA).

    Here is what I think TTA should do:

    Supports:
    • Work with reports generated by common unit-test frameworks (jUnit, nUnit, TestNG, TestUnit, style of reports)
    • Provides Web Service interface to upload results to TTA
    • Test Results uploaded will be stored in db
    • Will work on Windows and Linux

      Dashboard:
      • Creates default TTA dashboard
      • Customizable TTA dashboard
      • Dashboard will be accessible via the browser

        My questions to you:
        • Do you think this will help? 
        • What other features would you like to see in TTA?
        • What other type of support would you like to have in TTA?

        Tuesday, June 21, 2011

        WAAT and HTTPS

        While most sites use http to report tags to the web analytic tool, there are some cases where http is disabled and all traffic is using https only.

        In such cases, there may be a problem in using the generic solution provided by WAAT.

        I did some research, analysis and experimentation and here are my findings:
        1. jpcap captures raw packets. It does not differentiate about http / https
        2. There is no problem in WAAT. All it does it matches packets based on patterns you specify in the tests.
        3. Since the requests are https based, WAAT is not able to match the packets, unless you specify encrypted packet identifiers and encrypted data in the xml file. firebug / fiddler / ethereal / wireshark / charles / burp / etc. does something extra in this regard to decode the packet information and show the raw content in the browser / tool.

        So the question is what can be done next?
        1. If it is possible for you to get the configuration in the test environments changed to have the web analytics request sent out on http (maybe along with https) request, that can resolve the issue. Once in a while you can then verify manually if requests are going out on https.
        2. You can use Omniture Debugger - but the limitation in your case is that it will be available for Omniture only and not the other web analytic tools.
        3. You can extend the HttpSniffer class (,say HttpsSniffer), and provide implementation to decode the captured packets before doing the validation. However, note that this will be a expensive operation as you will be decoding all the captured packets on the network interfaces on your machine and the packet(s) of your interest will be fractional of those captured.

        Tuesday, April 19, 2011

        WAAT (Web Analytics Automation Testing Framework) is alive!!!

        [UPDATE]  Check related post here (http://essenceoftesting.blogspot.com/2011/04/waat-web-analytics-automation-testing.html)

        I am very happy to announce that the first release of WAAT is available for general use.

        WAAT is hosted on github (https://github.com/anandbagmar/WAAT)

        WAAT v1.3 can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent.

        Binaries:
        You can either get the code base from git@github.com:anandbagmar/WAAT.git, or, get the binaries available in the dist folder.

        Documentation:
        Documentation for using WAAT is available in various different formats:
        > WAAT Readme.docx
        > WAAT Readme.doc
        > WAAT Readme.pdf
        > WAAT Readme.html

        These files are available in the docs folder.

        The documentation is also part of the binary file downloads.

        I am looking forward for your usage and comments to make this better and more usable.

        Saturday, April 16, 2011

        WAAT release update

        I am almost ready with my first public release of WAAT. Some finishing touches remaining which is causing the delay for this.

        For those not aware, here is what WAAT is:
        > WAAT stands for Web Analytics Automation Testing Framework
        > Developed as a Java jar to be used in existing testing frameworks to do web analytics automation
        > Phase 1: Implemented for Omniture using Omniture Debugger -> Status: Completed
        > Phase 2: Can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent.
        -> Status: In progress. Documentation to be updated.
        > Phase 3: Make WAAT available for Ruby / .Net testing frameworks. -> Status: To be started.


        Look at my earlier post for more details on WAAT.

        Wednesday, April 13, 2011

        Interesting webinar coming up ... "Where Exploration and Automation meet: Leveraging..."

        There is a very interesting and informative webinar on how to utilize automated functional testing within your organization.

        This is scheduled for Thursday, April 21.

        See this link for more information.

        Saturday, April 9, 2011

        Agile QA Process

        After doing testing on multiple Agile projects, I have come to realize certain aspects about the process and techniques that are common across projects. Some things I have learned along the way, some, by reflection on the mistakes / sub-optimal things that I did.

        I have written and published my thoughts around the "Agile QA Process", more particularly what techniques can be used to test effectively in the Iterations. The pdf is available here for your reading. (http://www.slideshare.net/abagmar/agile-qa-process)

        Note: A process is something that should be tweaked and customized based on the context you are in. The process mentioned in the document should be taken in the same light.

        Friday, April 8, 2011

        WAAT - Web Analytics Automation Testing Framework

        [UPDATE]  See my post about how you can get WAAT here (http://essenceoftesting.blogspot.com/2011/04/waat-is-alive.html).

        Problem statement:

        On one of the projects that I worked on, I needed to test if Omniture reporting was done correctly.

        The client relied a lot on Omniture reports to understand and determine the direction of their business. They have a bunch of Omniture tags reported for a lot of different actions on the site. Manual testing was the only way this functionality could be done verified. But given the huge number of tags, it was never possible to be sure that all tags were being reported correctly on a regular basis.

        So I came up with a strategy to remove this pain-point.

        Approach:
        I created a framework in our existing automation framework to do Omniture testing. The intention of creating this framework was:
        1. There is minimal impact on existing tests.
        2. There should be no need to duplicate the tests just to do Omniture testing.
        3. Should be easy to use (specify Omniture tags for various different actions, enable testing, etc.)

        How it helped us?
        1. We provided a huge and reliable safety net to the client and the development team by having Omniture testing automated.
        2. Reduced the manual testing effort required for this type of testing, and instead got some bandwidth to focus on other areas.

        Next Steps:
        I am making this into a generic framework - a.k.a. WAAT - Web Analytics Automation Testing Framework to enable others doing Omniture testing to easily automate this functionality. This project will be hosted on github.

        Phase 1 of this implementation will be for Omniture Debugger and input data provided in xml format. This framework will be available as a jar file.

        Phase 2 also now complete includes support for any Web Analytic tool. I have tested this with Google Analytics as well as Omniture (NOT using Omniture Debugger). This uses a generic mechanism to capture packets from the network layer and processes them appropriately. Given this generic approach to work with any Web Analytic tool, the framework does become OS dependent.

        Watch this space for more information (instructions, links to github, etc). Also, please do contact me with ideas / suggestions / comments about the same.

        Tuesday, January 18, 2011

        My article on "Future of Test Automation Tools and Infrastructure" featured in The Smart Techie magazine

        My article on the Future of Test Automation Tools and Infrastructure is featured in "The Smart Techie" magazine. You can check it out here or download the pdf from here.

        Wednesday, December 8, 2010

        MEDIA ALERT: Live Web Event: The End of the Free Internet

        Paul Jay, senior editor of The Real News Network, will moderate a virtual panel discussion promoting a dialogue for the technology community about the technological and legal ramifications of the WikiLeaks shutdown.
           
        Topics to be discussed include:

            * What does the cut off of service to WikiLeaks mean for the future of the Internet? 

            * Will digital journalism be less protected?

            * Was WikiLeaks afforded procedural protections before its website and DNS entries were shut down? What process should be required?

            * The Internet is vulnerable to internal threats. What technical innovations are needed?

            * Can leaks ever be stopped? Is it worth the price?


        Click here (http://bit.ly/ejXRNX) for more information.

        Tuesday, November 23, 2010

        Focus and Positivity is the key!

        There are certain things I have observed quite a few times in various different situations while testing. These things seem more like a pattern than exceptions, and many a times demotivate me to do the right thing, for myself, and the organization.


        What are these things?

        • Testing is considered as a step child - at times not given enough "listening ear" as required.
        • Testing usually seems lower in priority (compared to other issues faced).
        • Clients / organization seem OK with bugs / defects going live or, in other words, they are OK with pushing code in Production without testing it properly.
        • QA team is not involved in all stages of the life cycle - leading to disconnect between team members, invalid / incorrect assumptions, etc.
        • QA capacity is not adequate for the amount of work actually needed to be done.
        • Team members say - "This [issue] is not my responsibility. Talk to someone else."
        • Team members say - "This [issue] is not your [QA] responsibility. You don't need to provide suggestions / solutions for it. Let the appropriate owners think / talk about it."

        Tips to remain focused on the right thing
        • Don't be hesitant to raise issues (time and again).
        • Identify and keep highlighting risks - in current situation, their implications and the potential mitigation points.
        • Keep doing what is right for the project / client.
        • Identify and engage with the "correct" people to ensure issues are understood, highlighted and prioritized.
        • Know what is in your control, and what is not. Strive to effect a change in what is in your control. For things out of your control, raise it with the relevant people, and then feel satisfied that you have done your part.
        • Be positive.
        I strongly believe that the last 2 items in the above list are the most crucial to remaining focused and being successful in whatever one does ... in personal life and in professional life.


        Saturday, November 20, 2010

        Generic Test Automation Frameworks - Good or Bad?

        Many a times I have been part of discussions that we should build a generic test automation framework which all other project teams can leverage. Also, another thought on this has been that building test automation framework is not really a QA job, but a developer job.

        I have different thoughts on this.
        1. From a capability building perspective, if we build a generic framework, how is the QA team going to learn to do things from ground-up, and also make it better?
        2. From a business perspective, if I as an organization have a generic framework, then am I going to give it for free to the client? If no, then it is as good as saying I have built the automation framework as a standalone product - which is fine. But, if I am giving it free to the client, then what value has it brought to me as an organization by designing and building this framework, and then I just give it off free?
        3. Developers can build a great framework - it is another product for them. However, the users of this framework are QA. More often than not, this framework that is built by the developers will be difficult to use by the QAs.
        • I have seen an example of a client-project, where our developers and QAs built the framework together. However, only the developers were able to make any core framework changes because of its complexity. Also, when the time came to hand-over the framework to the client, it was the developer(s) who had to provide training for this - the QAs were not even involved in this, because ...
        So to cut the answer short, I don't like the idea of generic test automation frameworks!

        Monday, October 11, 2010

        vodQA 2 - plan so hard for it, and its over so soon!

        We were planning for vodQA 2 since about 6-7 weeks. But we suffered our first hiccup when I had to go to the US on a QA Coaching assignment for 3 weeks. Fortunately, my return was a few days before vodQA (7th October 2010).

        It was a tough time being away ... constantly thinking about if all things are taken care off or not, if we are getting adequate help for the core organizers or not, what blockers are existing, and so on and on.

        However, our core team here in ThoughtWorks office really slogged it out, at times pulling people by their ears :) to ensure we are on track.

        All this meant that I didn't have to do much when I came back.

        On THE day, just before the event though, we suffered another hiccup - our audio-visual vendor refused to come up till a good few guests had already arrived. So it was awkward start to the event ... but things settled down pretty fast and then the event did go through quite smoothly.

        Pictures from vodQA 2 are available here

        A lot of positives from the event:
        1. More than 101 external guests arrived. Of these there were a significant number of people who had also attended our first vodQA event on June 10, 2010.
        2. The talks were very interesting, and very apt to the changing dynamics of the industry.
        3. In the initial feedback that we have gone through, people seem to be craving for such type of events - which is a very positive sign for us that we are doing something right, giving something to the community that will help them share, learn, grow and connect.
        4. We incorporated the feedback received in our first vodQA to make this more tuned to what people wanted to experience.

        A few people mentioned in the event - "this is our QA family, and we want to see it grow".

        Finally, the event came, and went - leaving behind lot of good memories. I will post the presentation and the videos links as soon as they are posted.

        Meanwhile, here are the topics that were covered in vodQA 2:

        Speaker / topic list:

        Full Length talks
        1. Deepak Gole & Saager Mhatre - Sapna Solutions
        Automated acceptance testing for iPhone

        2.Parul Mody & Vijay Khatke - BMC Software
        Cloud Testing

        3. Ashwini Malthankar - ThoughtWorks
        Effective use of Continuous integration by QA

        4. Vijay and Supriya - ThoughtWorks
        Test your service not your UI

        5. Ananthapadmanabhan R - ThoughtWorks
        Twist : Evolving test suites over test cases

        Lightning Talks
        1. Satish Agrawal - Persistent Systems
        Leadership and Innovation in a commoditized industry

        2. Anay Nayak - ThoughtWorks
        Fluent Interfaces and Matches Fish bowl topic: DNA of Test Automation