WAAT and HTTPS

While most sites use http to report tags to the web analytic tool, there are some cases where http is disabled and all traffic is using https only.

In such cases, there may be a problem in using the generic solution provided by WAAT.

I did some research, analysis and experimentation and here are my findings:

1. jpcap captures raw packets. It does not differentiate about http / https
2. There is no problem in WAAT. All it does it matches packets based on patterns you specify in the tests.
3. Since the requests are https based, WAAT is not able to match the packets, unless you specify encrypted packet identifiers and encrypted data in the xml file. firebug / fiddler / ethereal / wireshark / charles / burp / etc. does something extra in this regard to decode the packet information and show the raw content in the browser / tool.

So the question is what can be done next?

1. If it is possible for you to get the configuration in the test environments changed to have the web analytics request sent out on http (maybe along with https) request, that can resolve the issue. Once in a while you can then verify manually if requests are going out on https.
2. You can use Omniture Debugger - but the limitation in your case is that it will be available for Omniture only and not the other web analytic tools.
3. You can extend the HttpSniffer class (,say HttpsSniffer), and provide implementation to decode the captured packets before doing the validation. However, note that this will be a expensive operation as you will be decoding all the captured packets on the network interfaces on your machine and the packet(s) of your interest will be fractional of those captured.

WAAT (Web Analytics Automation Testing Framework) is alive!!!

[UPDATE]  Check related post here (http://essenceoftesting.blogspot.com/2011/04/waat-web-analytics-automation-testing.html)

I am very happy to announce that the first release of WAAT is available for general use.

WAAT is hosted on github (https://github.com/anandbagmar/WAAT)

WAAT v1.3 can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent.

Binaries:

You can either get the code base from git@github.com:anandbagmar/WAAT.git, or, get the binaries available in the dist folder.

Documentation:

Documentation for using WAAT is available in various different formats:

> WAAT Readme.docx

> WAAT Readme.doc

> WAAT Readme.pdf

> WAAT Readme.html

These files are available in the docs folder.

The documentation is also part of the binary file downloads.

I am looking forward for your usage and comments to make this better and more usable.

WAAT release update

I am almost ready with my first public release of WAAT. Some finishing touches remaining which is causing the delay for this.

For those not aware, here is what WAAT is:
> WAAT stands for Web Analytics Automation Testing Framework
> Developed as a Java jar to be used in existing testing frameworks to do web analytics automation
> Phase 1: Implemented for Omniture using Omniture Debugger -> Status: Completed
> Phase 2: Can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent. -> Status: In progress. Documentation to be updated.
> Phase 3: Make WAAT available for Ruby / .Net testing frameworks. -> Status: To be started.

Look at my earlier post for more details on WAAT.

Interesting webinar coming up ... "Where Exploration and Automation meet: Leveraging..."

There is a very interesting and informative webinar on how to utilize automated functional testing within your organization.

This is scheduled for Thursday, April 21.

See this link for more information.

Agile QA Process

After doing testing on multiple Agile projects, I have come to realize certain aspects about the process and techniques that are common across projects. Some things I have learned along the way, some, by reflection on the mistakes / sub-optimal things that I did.

I have written and published my thoughts around the "Agile QA Process", more particularly what techniques can be used to test effectively in the Iterations. The pdf is available here for your reading. (http://www.slideshare.net/abagmar/agile-qa-process)

Note: A process is something that should be tweaked and customized based on the context you are in. The process mentioned in the document should be taken in the same light.

WAAT - Web Analytics Automation Testing Framework

[UPDATE]  See my post about how you can get WAAT here (http://essenceoftesting.blogspot.com/2011/04/waat-is-alive.html).

Problem statement:

On one of the projects that I worked on, I needed to test if Omniture reporting was done correctly.

The client relied a lot on Omniture reports to understand and determine the direction of their business. They have a bunch of Omniture tags reported for a lot of different actions on the site. Manual testing was the only way this functionality could be done verified. But given the huge number of tags, it was never possible to be sure that all tags were being reported correctly on a regular basis.

So I came up with a strategy to remove this pain-point.

Approach:

I created a framework in our existing automation framework to do Omniture testing. The intention of creating this framework was:

1. There is minimal impact on existing tests.

2. There should be no need to duplicate the tests just to do Omniture testing.

3. Should be easy to use (specify Omniture tags for various different actions, enable testing, etc.)

How it helped us?

1. We provided a huge and reliable safety net to the client and the development team by having Omniture testing automated.

2. Reduced the manual testing effort required for this type of testing, and instead got some bandwidth to focus on other areas.

Next Steps:

I am making this into a generic framework - a.k.a. WAAT - Web Analytics Automation Testing Framework to enable others doing Omniture testing to easily automate this functionality. This project will be hosted on github.

Phase 1 of this implementation will be for Omniture Debugger and input data provided in xml format. This framework will be available as a jar file.

Phase 2 also now complete includes support for any Web Analytic tool. I have tested this with Google Analytics as well as Omniture (NOT using Omniture Debugger). This uses a generic mechanism to capture packets from the network layer and processes them appropriately. Given this generic approach to work with any Web Analytic tool, the framework does become OS dependent.

Watch this space for more information (instructions, links to github, etc). Also, please do contact me with ideas / suggestions / comments about the same.

My article on "Future of Test Automation Tools and Infrastructure" featured in The Smart Techie magazine

My article on the Future of Test Automation Tools and Infrastructure is featured in "The Smart Techie" magazine. You can check it out here or download the pdf from here.

MEDIA ALERT: Live Web Event: The End of the Free Internet

Paul Jay, senior editor of The Real News Network, will moderate a virtual panel discussion promoting a dialogue for the technology community about the technological and legal ramifications of the WikiLeaks shutdown.
   
Topics to be discussed include:

    * What does the cut off of service to WikiLeaks mean for the future of the Internet? 

    * Will digital journalism be less protected?

    * Was WikiLeaks afforded procedural protections before its website and DNS entries were shut down? What process should be required?

    * The Internet is vulnerable to internal threats. What technical innovations are needed?

    * Can leaks ever be stopped? Is it worth the price?


Click here (http://bit.ly/ejXRNX) for more information.

Focus and Positivity is the key!

There are certain things I have observed quite a few times in various different situations while testing. These things seem more like a pattern than exceptions, and many a times demotivate me to do the right thing, for myself, and the organization.


What are these things?

• Testing is considered as a step child - at times not given enough "listening ear" as required.
• Testing usually seems lower in priority (compared to other issues faced).
• Clients / organization seem OK with bugs / defects going live or, in other words, they are OK with pushing code in Production without testing it properly.
• QA team is not involved in all stages of the life cycle - leading to disconnect between team members, invalid / incorrect assumptions, etc.
• QA capacity is not adequate for the amount of work actually needed to be done.
• Team members say - "This [issue] is not my responsibility. Talk to someone else."
• Team members say - "This [issue] is not your [QA] responsibility. You don't need to provide suggestions / solutions for it. Let the appropriate owners think / talk about it."

Tips to remain focused on the right thing

• Don't be hesitant to raise issues (time and again).
• Identify and keep highlighting risks - in current situation, their implications and the potential mitigation points.
• Keep doing what is right for the project / client.
• Identify and engage with the "correct" people to ensure issues are understood, highlighted and prioritized.
• Know what is in your control, and what is not. Strive to effect a change in what is in your control. For things out of your control, raise it with the relevant people, and then feel satisfied that you have done your part.
• Be positive.

I strongly believe that the last 2 items in the above list are the most crucial to remaining focused and being successful in whatever one does ... in personal life and in professional life.

Generic Test Automation Frameworks - Good or Bad?

Many a times I have been part of discussions that we should build a generic test automation framework which all other project teams can leverage. Also, another thought on this has been that building test automation framework is not really a QA job, but a developer job.

I have different thoughts on this.

1. From a capability building perspective, if we build a generic framework, how is the QA team going to learn to do things from ground-up, and also make it better?
2. From a business perspective, if I as an organization have a generic framework, then am I going to give it for free to the client? If no, then it is as good as saying I have built the automation framework as a standalone product - which is fine. But, if I am giving it free to the client, then what value has it brought to me as an organization by designing and building this framework, and then I just give it off free?
3. Developers can build a great framework - it is another product for them. However, the users of this framework are QA. More often than not, this framework that is built by the developers will be difficult to use by the QAs.

• I have seen an example of a client-project, where our developers and QAs built the framework together. However, only the developers were able to make any core framework changes because of its complexity. Also, when the time came to hand-over the framework to the client, it was the developer(s) who had to provide training for this - the QAs were not even involved in this, because ...

So to cut the answer short, I don't like the idea of generic test automation frameworks!