rubygems.org & WAAT

With the recent problems the rubygems.org site is facing, you may not be able to get to the WAAT gem. In that case, you can also download the gem directly from the github repository. All information about WAAT can be found from my blog as before.

WAAT-Ruby crosses 1000 downloads on rubygems

Feeling very happy - the WAAT-ruby gem has crossed 1000 downloads from rubygems.org.

Speaking in Bangalore about WAAT and Agile Testing

As mentioned in my earlier post, I spoke last week in UNICOM's Next Generation Testing Conference in Bangalore on 13th December. My talk was about the "The What, Why and How of Web Analytics Testing". The slides from the talk are available here on slideshare.net. The audio recording of the talk is available here. (You can download and use a player like VLC to listen to the recording.)

I had a great time speaking in this conference. My talk was probably the only very technical talk in the conference. Another thing I observed from the audience is that not many of them knew about Web Analytics. I managed to finish up my talk in 40 min, and surprisingly, for an audience who didn't know much about Web Analytics, there were no questions. BUT, in the lunch break and networking session, a lot of people came up to me and said they really enjoyed my talk, and will look forward to "seeing" more on how Web Analytics is used in their organisation. 

Also, there was a lot of interest and questions about Agile, and Agile Testing. This is a topic I can talk about for hours at length - and I controlled myself to a great extent to let other esteemed speakers also talk and answer questions about the same from the audience.

I ended up encouraging a lot of fellow speakers and attendees to think with an "open-mind" and be innovative and see how service organisations can provide "a good and true value" to customers, and why it is important to provide "good" solutions and ask the right and tough questions to the same! Also shared the concepts of TaaS with a few folks who potentially would be in similar soup with the "common test framework" concept.

All in all, a very good time in Bangalore. Now looking forward for the same in Mumbai on 18th December, where I will be talking about "Integration Testing in Enterprises - using TaaS (Test as a Service)".

December 2012 conferences - WAAT & TaaS it is!

December seems to be a busy time related to conferences for me.

First, I attended test-ed 2012 conference hosted by moolya where I had the opportunity to meet James Bach and a few other great speakers. 

I was a little let-down by James talk. It seemed like more of a marketing pitch for moolya - and somehow I felt I expected more from a person of his caliber! None-the-less, I am sure he inspired a lot of folks in the auditorium to become a free and innovative tester! 

I also got to talk with him 1-1 about BDD and what situations it works well, and more importantly, when it does not work well. Also spoke with him on how BDT (Behavior Driven Testing) helps in building the "right automated regression suite" and the challenges facing the Testers in India in order to become the "free-spirited, creative and innovative testers" he spoke about.

Next, on 13th December, I am speaking in UNICOM's Next Generation Testing Conference in Bangalore about "The What, Why, and How of Web Analytics Testing". I will be talking about my open-source framework - "WAAT - the Web Analytics Automation Testing", and how that can ease the manual drudge of web analytics testing.

To close the year, I will be speaking on 18th December in another UNICOM's Next Generation Testing Conference in Mumbai. Here I will be talking about "Integration testing in Enterprises using TaaS (Test-as-a-Service)- Via Case Study". This is about another open-source framework I have created - TaaS - Test as a Service.

Hope to see you all in Bangalore / Mumbai.

WAAT v1.5.0 released to rubygems.org

The newest version of WAAT (Web Analytics Automation Testing framework) for Java and Ruby is now available with a new plugin - JsSniffer.

Following the announcement of the new JsSniffer plugin for WAAT, I have now completed the testing for the same, and released the gem - WAAT-1.5.0.gem to rubygems.org.

Please take a look at the FAQs and the major changes on Ruby / major changes on Java section for some known issues, potential solutions for the same - as some of these changes could break your tests.

WAAT (Java & Ruby) with JS_SNIFFER is out of the box

I have pushed in the latest changes to WAAT to get over the limitation of not working in a pure https environment (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html).

The solution is creating a new type of plugin - called JS_SNIFFER.

This plugin requires the user of WAAT to do a little more work than before. 

They need (to work with their development team) to figure out what JS script they need to invoke in the browser to get the URL of interest that is sent as a pure https request over the wire. WAAT then takes this request, and does the tag matching for you.

This generation of the JS script is a one-time effort - unless the way the tags are reported changes in the product. Then the test framework can work in a seamless fashion as before to test this is working consistently in an automated fashion. 

Another advantage of this is that with this approach, we do not need to install jpcap or run the tests as a "super-user" - a restriction posed by the network packet capture library.

The WAAT_v1.5.0.jar is available on here (https://github.com/anandbagmar/WAAT/tree/master/dist) on github.

Similarly, I have also updated the WAAT-ruby gem (WAAT-1.5.0.gem). This gem is not yet pushed out to rubygems.org - as I am still testing it out. However, if you are interested, you can download it from here.

As usual, feedback / comments / suggestions most welcome!

New version of WAAT-ruby gem available

I finally got around to pushing out a new version of WAAT-ruby (1.4.1) on rubygems.org. The only change in this version is the removal of a dependency on a particular version of bundler. See the WAAT-ruby project on github for more information.

Watch this space for a new version of WAAT-ruby that overcomes the limitation of doing web analytics automation for https urls.

Whats next for WAAT?

It has been quite some time that I updated WAAT. The released version has been working well - but it does have its limitations as listed in the FAQs on github.

The biggest limitation I feel about the current release version of WAAT is that it does not work in a pure https kind of an environment. (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html)

Of late I have been spiking out different ways to overcome this limitation. I have experimented to create a HttpsSniffer, and hit various different road-blocks in that. That has forced me to look at another strategy. 

So I have changed direction in coming to a solution. I am looking at creating something like a JSInjector / JSSniffer plugin - which executes a javascript in the browser from where the action is invoked. This is not as straight forward to use as the earlier approaches. The user of this plugin will need to understand the DOM and some javascripts better, maybe take help from the development team, but then once the way to retrive the basic information is known, then we are in calm waters again :)

If you are facing this similar issue in a pure https environment for web analytics testing, look out for more information in this space.

My plan is to update WAAT, followed-by WAAT-Ruby and then lastly release a new version of the WAAT-Ruby gem following that.

WAAT-Ruby crosses 300 downloads

I was very happy to see that WAAT-Ruby (Web Analytics Automation Testing framework) has crossed 300 downloads on rubygems.org.

Feedback on WAAT

I am considering adding some functionality to WAAT. However, before I do that, I would like to know what your opinion is.


So, to all those who are either using, tried using, or, want to use WAAT: Can you please provide me some feedback based on the following questions:

• Which flavor of WAAT do you use? 
• Java
• Ruby
• Both
• Have you faced any problems using WAAT? 
• If yes, what problems? How did you resolve them?
• WAAT using the httpSniffer approach has known limitations (namely: does not support https request capturing, and on non-Windows platform you need to run the tests using root access). 
• Have you run into these limitations? 
• How did you resolve the issue?
• Do you find the WAAT wiki useful?
• If not, what if done differently will provide more value?
• Any other thoughts / comments on how WAAT can be made better?

Looking forward for your comments.

Thanks.

Anand

vodQA Chennai starts off with a century!

I attended vodQA in Chennai on 21st Jan 2012. The event was great. Over 100 passionate testers from Chennai testing community turned up and made sure people presenting were on their toes with excellent questions and great interactions.


One of the participants already blogged about it here. She raises valid observations - and I wish I had the opportunity to speak with her directly to address some of the questions / concerns she raised. 


In this session, I presented a topic - "What is WAAT?" based on my open-source project - WAAT. The slides used in this session are available here.


I am already looking forward to the next vodQA in Chennai. For now, I am preparing for vodQA Bangalore - "Agile Testing for Teams and Enterprises". on Feb 11, and then vodQA Pune on Mar 17.

WAAT-Ruby - available on RubyGems and opensourcetesting

I am very happy to announce that WAAT is now available on RubyGems.org and is also linked from Open source functional testing tools.

Here are the links for the same:

 

https://rubygems.org/gems/WAAT

http://www.opensourcetesting.org/functional.php#WAAT-Ruby

http://www.opensourcetesting.org/functional.php#WAAT-Java

WAAT-Ruby - ready for use

WAAT-Ruby is now ready for use. 

Project hosted on github - http://github.com/anandbagmar/WAAT-Ruby

WAAT-Ruby gem available for download from here.

Documentation is available (on WAAt-Ruby wiki pages) here.

Since WAAT-Ruby uses WAAT-Java under the covers, I have kept the same version numbers for both platforms. The latest version is 1.4.

I have not yet pushed it out on rubygems.org. Will update once that is done.

So far I have tested this on the following environments:

• Windows 7 - 64-bit with Ruby 1.8.6
• RHEL 6 - 64-bit with Ruby 1.8.6 (I had difficulty in getting jpcap deployed on this environment). But once that was done, WAAT worked smoothly out of the box.
• Ubuntu 10.x - 32-bit with Ruby 1.8.7
• Ubuntu 10.x - 32-bit with Ruby 1.9.1

One important note:

If you are using WAAT (Java or Ruby) on any Linux-based environment, please note the Jpcap requirement for execution.

WAAT uses Jpcap for capturing network packets. Jpcap needs administrative privileges to do this work. See the Jpcap FAQs for more information about the same.


For all WAAT related blog posts, click here.

WAAT - Ruby .... are we there yet?

The WAAT ruby gem is almost ready. My colleagues are helping testing it out in different environments and am updating the documentation accordingly.

Once done, this will be available as a Ruby gem from WAAT-Ruby github project, and also from rubygems.org.

Contact me if you are interested in trying this out before release.

Ruby Test Automation Framework Technology Stack Survey

WAAT  - Web Analytics Automation Testing Framework is currently available for java based Test Automation Frameworks. (http://essenceoftesting.blogspot.com/search/label/waat)

I am now working on making this available as a Ruby gem.

In order to support WAAT for a good-mix of test environments, I would like to understand the different environments and technology stacks that are typically used by teams in their Test Automation Framework.

Thank you for your time and help in providing this information.

WAAT for Ruby on its way

I have started work on creating a Ruby gem for WAAT. This is going to sit on top of the version created for Java. Hopefully will be able to get it out soon.

Watch this space for more information.

WAAT and HTTPS

While most sites use http to report tags to the web analytic tool, there are some cases where http is disabled and all traffic is using https only.

In such cases, there may be a problem in using the generic solution provided by WAAT.

I did some research, analysis and experimentation and here are my findings:

1. jpcap captures raw packets. It does not differentiate about http / https
2. There is no problem in WAAT. All it does it matches packets based on patterns you specify in the tests.
3. Since the requests are https based, WAAT is not able to match the packets, unless you specify encrypted packet identifiers and encrypted data in the xml file. firebug / fiddler / ethereal / wireshark / charles / burp / etc. does something extra in this regard to decode the packet information and show the raw content in the browser / tool.

So the question is what can be done next?

1. If it is possible for you to get the configuration in the test environments changed to have the web analytics request sent out on http (maybe along with https) request, that can resolve the issue. Once in a while you can then verify manually if requests are going out on https.
2. You can use Omniture Debugger - but the limitation in your case is that it will be available for Omniture only and not the other web analytic tools.
3. You can extend the HttpSniffer class (,say HttpsSniffer), and provide implementation to decode the captured packets before doing the validation. However, note that this will be a expensive operation as you will be decoding all the captured packets on the network interfaces on your machine and the packet(s) of your interest will be fractional of those captured.

WAAT (Web Analytics Automation Testing Framework) is alive!!!

[UPDATE]  Check related post here (http://essenceoftesting.blogspot.com/2011/04/waat-web-analytics-automation-testing.html)

I am very happy to announce that the first release of WAAT is available for general use.

WAAT is hosted on github (https://github.com/anandbagmar/WAAT)

WAAT v1.3 can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent.

Binaries:

You can either get the code base from git@github.com:anandbagmar/WAAT.git, or, get the binaries available in the dist folder.

Documentation:

Documentation for using WAAT is available in various different formats:

> WAAT Readme.docx

> WAAT Readme.doc

> WAAT Readme.pdf

> WAAT Readme.html

These files are available in the docs folder.

The documentation is also part of the binary file downloads.

I am looking forward for your usage and comments to make this better and more usable.

WAAT release update

I am almost ready with my first public release of WAAT. Some finishing touches remaining which is causing the delay for this.

For those not aware, here is what WAAT is:
> WAAT stands for Web Analytics Automation Testing Framework
> Developed as a Java jar to be used in existing testing frameworks to do web analytics automation
> Phase 1: Implemented for Omniture using Omniture Debugger -> Status: Completed
> Phase 2: Can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent. -> Status: In progress. Documentation to be updated.
> Phase 3: Make WAAT available for Ruby / .Net testing frameworks. -> Status: To be started.

Look at my earlier post for more details on WAAT.

WAAT - Web Analytics Automation Testing Framework

[UPDATE]  See my post about how you can get WAAT here (http://essenceoftesting.blogspot.com/2011/04/waat-is-alive.html).

Problem statement:

On one of the projects that I worked on, I needed to test if Omniture reporting was done correctly.

The client relied a lot on Omniture reports to understand and determine the direction of their business. They have a bunch of Omniture tags reported for a lot of different actions on the site. Manual testing was the only way this functionality could be done verified. But given the huge number of tags, it was never possible to be sure that all tags were being reported correctly on a regular basis.

So I came up with a strategy to remove this pain-point.

Approach:

I created a framework in our existing automation framework to do Omniture testing. The intention of creating this framework was:

1. There is minimal impact on existing tests.

2. There should be no need to duplicate the tests just to do Omniture testing.

3. Should be easy to use (specify Omniture tags for various different actions, enable testing, etc.)

How it helped us?

1. We provided a huge and reliable safety net to the client and the development team by having Omniture testing automated.

2. Reduced the manual testing effort required for this type of testing, and instead got some bandwidth to focus on other areas.

Next Steps:

I am making this into a generic framework - a.k.a. WAAT - Web Analytics Automation Testing Framework to enable others doing Omniture testing to easily automate this functionality. This project will be hosted on github.

Phase 1 of this implementation will be for Omniture Debugger and input data provided in xml format. This framework will be available as a jar file.

Phase 2 also now complete includes support for any Web Analytic tool. I have tested this with Google Analytics as well as Omniture (NOT using Omniture Debugger). This uses a generic mechanism to capture packets from the network layer and processes them appropriately. Given this generic approach to work with any Web Analytic tool, the framework does become OS dependent.

Watch this space for more information (instructions, links to github, etc). Also, please do contact me with ideas / suggestions / comments about the same.