Real-time Trend and Failure Analysis using Test Trend Analyzer (TTA)

Real-time Trend and Failure Analysis using Test Trend Analyzer (TTA)

Anand Bagmar

Summary

Organizations have long running products / programs. They need to understand the health of their products / projects at a quick glance, instead of having a team of people manually scrambling frantically to collate and collect the information needed to get a sense of quality
about the products they support. 

TTA is an open source product that becomes the source of information to give you real-time insights into the health of the product portfolio using the Test Automation results, in form of Trends, Comparative Analysis, Failure Analysis and Functional Performance Benchmarking.

The Dream

The statement "I have a dream" is a very famous quote by American activist Martin Luther King Jr.

I resonate very closely with that. Here is why and how ...

Sometime in 2011, I had a dream ... a vision about a product that can help those working in large organizations understand the health of their products / projects at a quick glance, instead of having a team of people manually scrambling frantically to collate and collect the information needed to get a sense of quality about the products they support….

I called this dream - Test Trend Analyzer - TTA

What is TTA?

In a nutshell, given all various types of Test Automation is done in your organization, TTA is a product that stores and parses the test run results, and then displays various Trend Analysis charts and also does Test Failure analysis for you. Based on the context of the product under test, the viewer can then make more meaning of the data presented, and more importantly, take meaningful actions / next steps.

Why do I need to Trend Analysis of the test results?

Automation (Unit / Integration / Functional / etc.) is a key factor in ensuring the success, quality and time-to-market for products.

Since Automated tests are executed via CI (Continuous Integration), a lot of trend analysis and test failure analysis is already be done by the CI tool itself.

However, the ability of CI doing this is limited for the following reasons:

• The typical archival duration in CI is in the range from 15-45 days.
• Only trends can be seen after grouping relevant jobs in the CI tool.
• It is difficult to group all related product jobs in CI – because of the sheer volume of tests.
• The grouping of jobs becomes more challenging if the number of products / projects / vendors or partners / environments / etc. are more in number.
• The projects / products are long running (many months to years). It is not practical to archive the results for such duration in CI.

I have seen first-hand many of the use cases listed below from real scenarios, where we need a unique and different product to solve some Testing Specific problems:

• A Business Manager / Test Director overseeing multiple products development in the organization may want to see the overall health of all the products in his / her portfolio, in real time.
• A Product Owner / PM / Test Manager overseeing the product development / testing of a specific product in the organization may want to see the overall health of the product, in real time.
• Individual team members (Tech Leads / QAs / Developers / etc.) want to do quick test failure analysis in order to decide the correct priority of next set of tasks.

Vision for TTA

With the above considerations in mind, I came up with the following vision statement for TTA:

•       A single point, visual solution to gauge the health of your product portfolio using Test Automation results by means of –

–      Trends

–      Failure analysis

•       And providing

–      Drill-down reports

–      Customizable reports

•       So that

–      Different stakeholders can get single click view of the health status and potential issues

–      A project team can decide if automation is useful or not.

–      Automated data collation and trending to avoid manual data aggregation and interpretation

•       With the stakeholders being

–      QA Directors / Managers / Leads / hands-on-tester

–      Developers

–      Tech-Ops

How does TTA work?

TTA is developed as an independent RoR product. It uses MySQL as the database. You will need to install TTA (instructions available on TTA github wiki) on an independent (virtual) machine.

TTA is a decoupled product. It does not depend on any specific CI (Continuous Integration) Tool, programming language, test framework, etc.

CI Jobs typically call some build tool – example ant, maven, gradle, etc. The command called by the CI job does the test setup, and then executes the tests. After execution, the results are sent back to CI, and the test run completed.

After the test execution is completed, to integrate automatic reporting of results to TTA, we need to:

• Zip the log folder, and,
• Send the results with test meta-data information

Current set of Features for TTA

·       Test Pyramid view (/pyramid) - to see how your project's automation effort aligns with the Test Automation philosophy

·       Comparative Analysis view (/comparative_analysis) - to see the trend of your test automation results over a period of time, and if any patterns emerge

·       Failure Analysis view (/defect_analysis) - to make better meaning of the test failures, and help you prioritize which failures should be fixed first.

·       Integration of external dashboards (add from /admin page, see integration on /home page) - this allows one to integrate different existing dashboards into TTA - to make it a one stop place for seeing all Testing related information. Example: You can integrate your defect reports from Mingle / Jira / etc., or, you can also integrate your specific Continuous Integration (CI) dashboard from Go / Jenkins / Hudson / Bamboo / etc.

·       Test Execution Trend - to see the benchmarking of specific test execution over a time period

·       Compare test runs (/compare_runs) - to compare specific test runs

o   what are the common failures, 

o   what are the unique failures, 

o   what failed on date 1, but passed on date 2

o   what failed on date 2, but passed on date 1

·       Upload Test Run Data manually (/upload) - to manually upload test data in case if you have not uploaded test data automatically to TTA, but still want to use TTA

·       TTA Statistics Page (/stats) - to know usage of TTA by different projects / teams in your organization

Refer to my blog or the TTA-github-wiki for other information, including screenshots about TTA.

Current state

TTA is available as an open-source product via github. There are a couple of clients (internal to ThoughtWorks, and external) using TTA in their projects.

How can you contribute?

Given that we have implemented only a few basic features right now, and there are many more in the backlog, here is how you can help:

• Suggest new ideas / features that will help make TTA better
• Use TTA on your project and provide feedback
• More importantly, help in implementing these features

Contact information

Contact Anand Bagmar (anand.bagmar@thoughtworks.com / abagmar@gmail.com) for more information about TTA.

Test Execution Trend feature added in TTA (0.5.7)

A new version of Test Trend Analyzer (TTA) is now available on github.

These are the new features in it:

• Test Execution Trend - to see the benchmarking of specific test execution over a time period

• Failure Analysis View

• Modified view to show failures for specific Test category along with specific test run

• Modified view to show failures for All test categories with latest test run

• Minor Refactoring for Compare Runs view

• Query optimisations + minor defect fixes.

For those that missed the earlier announcement - summary of features of TTA is available here, and source code is available on github.

PS: There is a lot of work (features / UI) planned for TTA. If you wish to contribute, we would love your help!!

TTA (0.5.4) feature information and screenshots

Here are some screenshots from the current functionality supported by TTA v0.5.4.

See the TTA Wiki page or Contact me for more information on TTA.

Manual upload of Test data

Test Pyramid

Comparative Analysis

Compare different test runs

Test Failure Analysis

Integrate External Dashboards with TTA

TTA Statistics page

TaaS @Agile2013

[UPDATE] Final update: The slides AND the audio recording (mp3) of my talk in Agile2013 are now available.

[UPDATE] Slides for the presentation are now available here.
 
So close, yet so far .... dreading the 15.30 hour non-stop flight from Mumbai to Newark, yet super excited to speak in Agile 2013, Integration testing in Enterprises using TaaS - via Case Study, on 5th August, from 2pm-3.15pm. Complete schedule is available here.

TaaS and BDT - back to back at WCNGT, Bangalore

I had a great time talking about TaaS and BDT at WCNGT conference in Bangalore on 11th July 2013.

Here is more information about the talks:

Integrated Testing in Enterprises using TaaS:

http://essenceoftesting.blogspot.in/2013/07/taas-in-wcngt-bangalore-and-agile-2013.html
The audio recording of the talk is available here.
The slides for the talk are available here.
TaaS is available on github and as a gem on rubygems.org.

Building the "right" regression suite using Behavior Driven Testing (BDT)

http://essenceoftesting.blogspot.com/2013/07/building-right-regression-suite-using.html
The audio recording of the talk is available here.
The slides for the talk are available here.

Building the "right" regression suite using Behavior Driven Testing (BDT)

As a last minute surprise, I have been requested to talk about Building the "right" regression suite using Behavior Driven Testing (BDT) in the World Conference on Next Generation Testing on 11/12 July 2013 in Bangalore. This is another of those topics I feel very strongly about and am very excited at this opportunity.

What this does mean though is that I have back-2-back 2 1-hour sessions at WCNGT2013 - first - Integrated Testing in Enterprises using TaaS, and immediately post lunch - the BDT talk. 

Should be fun ... and hopefully I don't mix up the content in the talks :)

Here is the high-level agenda of the topic: Building the "right" regression suite using Behavior Driven Testing (BDT):

• Overview of Agile Testing
• The Test Pyramid
• Different flavors of TDD
• BDD – Behavior Driven Development?
• ATDD – Acceptance Driven Development?
• BDT – Behavior Driven Testing?
• Difference between BDD and BDT
• Tools that support BDT
• The value proposition for the team - live documentation, business rules, onboarding, etc.

TaaS in WCNGT, Bangalore and Agile 2013, Nashville, USA

A busy couple of weeks for me as I talk about TaaS in World Conference on Next Generation Testing in Bangalore, on 11/12th July 2013, and in Agile 2013 in Nashville, USA, on 4-9th August 2013.
 
My session details are as follows:

In WCNGT - Integrated Testing In Enterprises Using TaaS, on 11th July, from 12pm-1pm. Detail program is available here.

In Agile 2013, Integration testing in Enterprises using TaaS - via Case Study, on 5th August, from 2pm-3.15pm. Complete schedule is available here.

I will be speaking about TaaS in Agile 2013 in Nashville, TN, USA

Very happy and excited to say that my talk on "Integration testing in Enterprises using TaaS - via Case Study" has been selected in Agile 2013. This conference is scheduled from 5-9th August in Nashville, TN, USA, and I get to present it as the 2nd talk on the 1st day of the conference!

TaaS is available on rubygems.org

I have finally created and released the TaaS gem to rubygems.org. You can find it here.

To know more about TaaS - see all related blog posts here, or refer to the github project for the same.

Introducing Test Trend Analyzer (TTA)

The statement "I have a dream" is a very famous quote by American activist Martin Luther King Jr.

I resonate very closely with that. Here is why and how ...

Almost 2 years ago, I had a dream ... a vision about a product that can help those working in large organizations understand the health of their products / projects at a quick glance, instead of having a team of people manually scrambling frantically to collate and collect the information needed to get a sense of quality about the products they support. I chose to call this product Test Trend Analyzer - TTA

Given that Automation is a key factor in ensuring the success, quality and time-to-market for products, I took that as a baseline requirement and came up with a basic high-level diagram for TTA:

TTA - first diagram

Finally, a couple of months ago, I found a bunch of passionate people, who also had some time, to try and implement this tool.

We came up with this vision for TTA:

TTA Vision

Accordingly, we planned, prioritized, spiked, did some prototypes, did demos and showcases and got a base usable product completed.
This is a open-source project, available on github with more information about it on the github wiki.

The technology stack used is: RoR with mysql db. 

TTA falls in the Big Data + Visualization space - specific to Testing.




Watch this space for more information about TTA and the currently supported features. Email out to me if you need more specific information, or have questions on how can you use TTA, etc.

How can you contribute?

Given that we have implemented only a few basic features right now, and there are many more in the backlog, here is how you can help:

• Suggest new ideas / features that will help make TTA better
• Use TTA on your project and provide feedback
• More importantly, help in implementing these features

TaaS (Test-as-a-Service) coming to Sri Lanka and Delhi

Riding on the good response received on TaaS in 2012 and early 2013, I will be speaking more on the same in Colombo, Sri Lanka on 7th March 2013, and in Delhi on 14th March 2013. Looking forward to more feedback on TaaS!

rubygems.org & WAAT

With the recent problems the rubygems.org site is facing, you may not be able to get to the WAAT gem. In that case, you can also download the gem directly from the github repository. All information about WAAT can be found from my blog as before.

TaaS received pretty well in Pune conf

As mentioned in my earlier post, I spoke about TaaS - Test-as-a-Service in UNICOM's Next Generation Testing Conference in Pune on 23rd Jan. The talk went pretty well. Since this talk was for a slightly smaller duration, I had to use a scaled down version of the same slides that I used in the TaaS talk in Mumbai. The audio playback of the talk in Pune is available here.

There were a lot of interesting questions that came up this time:
1. Can I use TaaS to test in the cloud?
2. Can I use TaaS to do load / performance testing?
3. What are the security implications of using TaaS? 
4. Can I use TaaS to do mobile testing across different devices?

I will try to address these questions in subsequent posts.

WAAT-Ruby crosses 1000 downloads on rubygems

Feeling very happy - the WAAT-ruby gem has crossed 1000 downloads from rubygems.org.

Taking video recordings of your test execution

I have always wanted to take video recordings of my test execution ... discard the video if the test passes, and save it if the test fails. 

Recently, I found that the headless gem does that for you. There is also some sample code to make the integration of this gem in your test framework very seamless.

While using the gem and its api's are very straightforward, this did not work very well for me.

Reasons:

1. I run my tests from various different types of machines / os, namely, mac, RHEL and Windows
2. The total number of machines I can run my tests from is huge (>20)  … mainly triggered via CI
3. The RHEL environment is controlled and configured by puppet.
4. There is a bunch of underlying library requirements for the video capture

So though it is easy to use the gem, it is not really easy to setup ALL test execution environments with these dependent libraries for this to work - at least for me.

Does anyone know of a better / easier solution?

TaaS (Test-as-a-Service) coming to Pune

2012 was a great year for me. I got the opportunity to share my learnings and open-source contributions in various conferences. 

2013 also seems to be starting off well. I will be talking in UNICOM's Next Generation Testing Conference, this time in Pune, on Wednesday, 23rd Jan. I will be repeating my talk from Mumbai on TaaS - Test-as-a-Service.

End-2-End automated Integration Testing using TaaS - Test as a Service

I spoke in Mumbai recently about an open-source framework created to assist in End-2-End automated Integration Testing using TaaS - Test as a Service. The slides for the talk are available here and audio here.

The session was planned for 1 hour. I managed to finish the talk in 50ish minutes, then then the Q&A went on for another 30ish minutes.

Plan is to create and publish TaaS as a gem on rubygems.org.

Looking forward to your comments / feedback for the same!!

WAAT v1.5.0 released to rubygems.org

The newest version of WAAT (Web Analytics Automation Testing framework) for Java and Ruby is now available with a new plugin - JsSniffer.

Following the announcement of the new JsSniffer plugin for WAAT, I have now completed the testing for the same, and released the gem - WAAT-1.5.0.gem to rubygems.org.

Please take a look at the FAQs and the major changes on Ruby / major changes on Java section for some known issues, potential solutions for the same - as some of these changes could break your tests.

WAAT (Java & Ruby) with JS_SNIFFER is out of the box

I have pushed in the latest changes to WAAT to get over the limitation of not working in a pure https environment (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html).

The solution is creating a new type of plugin - called JS_SNIFFER.

This plugin requires the user of WAAT to do a little more work than before. 

They need (to work with their development team) to figure out what JS script they need to invoke in the browser to get the URL of interest that is sent as a pure https request over the wire. WAAT then takes this request, and does the tag matching for you.

This generation of the JS script is a one-time effort - unless the way the tags are reported changes in the product. Then the test framework can work in a seamless fashion as before to test this is working consistently in an automated fashion. 

Another advantage of this is that with this approach, we do not need to install jpcap or run the tests as a "super-user" - a restriction posed by the network packet capture library.

The WAAT_v1.5.0.jar is available on here (https://github.com/anandbagmar/WAAT/tree/master/dist) on github.

Similarly, I have also updated the WAAT-ruby gem (WAAT-1.5.0.gem). This gem is not yet pushed out to rubygems.org - as I am still testing it out. However, if you are interested, you can download it from here.

As usual, feedback / comments / suggestions most welcome!