Any WAAT (Web Analytics Automation Testing Framework) users out there?

It has been over 2 years since any update to WAAT - Java or Ruby. Over the years, I have realised, and also received a lot of thoughts / feedback from users of WAAT around where it helps, and what challenges exist. 

Also, given the widespread IoT & Big Data based work going on around the world, (Web) Analytics now plays a much bigger role in guiding business take better decisions. 

WAAT (again) fits in the grand scheme of things very nicely as a framework to automate the validation of correct reporting of tags to any Web Analytics solution provider.

Hence, its a no-brainer for me - it is high time I work on some of the feedback and limitations of WAAT to make it usable again!

At the recently concluded Selenium Conference 2016 held in Bangalore, India, I got an idea of how to overcome a lot of challenges (listed below) and pain in using WAAT. 

What's next?

To implement my new idea, this does mean a couple of things:

• Existing plugins have limited use - and needs to be deleted.
• A new plugin would need to be created - which may mean different set of APIs, and also different way to specify the test data.

Questions for you

Before I go ahead making these changes - I would like to get answers to the below questions (please add your answers directly in the comments):

• Is anyone currently using WAAT? If yes - 
• which version (Java / Ruby)?
• which plugin
• Using HTTP / HTTPS?
• Which Web Analytic solution are you using? (ex: Google Analytics, WebTrends, etc?)
• Would you be interested in using the new WAAT? If yes - 
• Which language? Java / Ruby / JavaScript / Python / etc?
• Would you like to contribute to implementing this new WAAT? If yes - contact me! :)

-----------------------------------------------------------------------------------------------------------------

Current plugins available in WAAT:

• Omniture Debugger (WAAT-Java)
• Pros:
• OS independent
• Run using regular-test-user 
• Cons:
• Browser dependent - need to implement ScriptRunner for the UI-driver in use
• Web-Analytic solution dependent - only for Adobe Marketing Clout / Omniture SiteCatalyst
• HTTPSniffer (WAAT-Java, WAAT-Ruby)
• Pros
• Web-Analytic solution independent
• Browser independent
• UI-Driver independent
• Cons
• 3rd party libraries are OS dependent
• HTTPs is not supported out-of-the-box
• Run tests as "root"
• JSSniffer (WAAT-Java, WAAT-Ruby)
• Pros
• Web-Analytic solution independent
• Browser independent
• HTTPs supported out-of-the-box
• No 3rd party library dependency
• Cons
• Need to write JavaScript to get the URL from the browser context
• UI-Driver dependent

-----------------------------------------------------------------------------------------------------------------

WAAT 2.0 (BETA) available for use

Very excited to share that I, along with a few collaborators of WAAT did a workshop in vodQA, Pune on Sat, 27th Aug about Automating Web Analytics - Why? How?. 
We implemented the automation using WAAT.

Also, am very excited to announce that WAAT v2.0 BETA for Java is now available from the WAAT project.

The slides from the workshop are available here

Automating Web Analytics from Anand Bagmar

Here are some details of the workshop:

Abstract

Do you know –

• What is Web Analytics?

• How does Web Analytics work?

• Why is it important? How to test Web Analytics?

• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

 
We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

 
We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

 
Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Pre-Reqs

• JDK 1.8
• IDE (Eclipse / IntelliJ Idea / etc.)
• Git (and add it to path)
• Gradle 2.3 (and add it to path)
• Firefox browser compatible with Selenium v2.53.1 (v46)
• Also refer here for steps to run Appium tests using WAAT 

 

Sample Code

- Create a directory WAAT-Workshop and run
git clone git@github.com:anandbagmar/waat-sample-java.git
- If Using any existing framework: download WAAT-all-v2.0_BETA.jar
http://goo.gl/8bfyaJ
- To run sample code - from waat-sample-java directory - run the command -
gradle clean build 

Please reach out if you need help in using WAAT!

WAAT (Web Analytics Automation Testing Framework) is alive!!!

[UPDATE]  Check related post here (http://essenceoftesting.blogspot.com/2011/04/waat-web-analytics-automation-testing.html)

I am very happy to announce that the first release of WAAT is available for general use.

WAAT is hosted on github (https://github.com/anandbagmar/WAAT)

WAAT v1.3 can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent.

Binaries:

You can either get the code base from git@github.com:anandbagmar/WAAT.git, or, get the binaries available in the dist folder.

Documentation:

Documentation for using WAAT is available in various different formats:

> WAAT Readme.docx

> WAAT Readme.doc

> WAAT Readme.pdf

> WAAT Readme.html

These files are available in the docs folder.

The documentation is also part of the binary file downloads.

I am looking forward for your usage and comments to make this better and more usable.

WAAT-Ruby - ready for use

WAAT-Ruby is now ready for use. 

Project hosted on github - http://github.com/anandbagmar/WAAT-Ruby

WAAT-Ruby gem available for download from here.

Documentation is available (on WAAt-Ruby wiki pages) here.

Since WAAT-Ruby uses WAAT-Java under the covers, I have kept the same version numbers for both platforms. The latest version is 1.4.

I have not yet pushed it out on rubygems.org. Will update once that is done.

So far I have tested this on the following environments:

• Windows 7 - 64-bit with Ruby 1.8.6
• RHEL 6 - 64-bit with Ruby 1.8.6 (I had difficulty in getting jpcap deployed on this environment). But once that was done, WAAT worked smoothly out of the box.
• Ubuntu 10.x - 32-bit with Ruby 1.8.7
• Ubuntu 10.x - 32-bit with Ruby 1.9.1

One important note:

If you are using WAAT (Java or Ruby) on any Linux-based environment, please note the Jpcap requirement for execution.

WAAT uses Jpcap for capturing network packets. Jpcap needs administrative privileges to do this work. See the Jpcap FAQs for more information about the same.


For all WAAT related blog posts, click here.

Feedback on WAAT

I am considering adding some functionality to WAAT. However, before I do that, I would like to know what your opinion is.


So, to all those who are either using, tried using, or, want to use WAAT: Can you please provide me some feedback based on the following questions:

• Which flavor of WAAT do you use? 
• Java
• Ruby
• Both
• Have you faced any problems using WAAT? 
• If yes, what problems? How did you resolve them?
• WAAT using the httpSniffer approach has known limitations (namely: does not support https request capturing, and on non-Windows platform you need to run the tests using root access). 
• Have you run into these limitations? 
• How did you resolve the issue?
• Do you find the WAAT wiki useful?
• If not, what if done differently will provide more value?
• Any other thoughts / comments on how WAAT can be made better?

Looking forward for your comments.

Thanks.

Anand

WAAT (Java & Ruby) with JS_SNIFFER is out of the box

I have pushed in the latest changes to WAAT to get over the limitation of not working in a pure https environment (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html).

The solution is creating a new type of plugin - called JS_SNIFFER.

This plugin requires the user of WAAT to do a little more work than before. 

They need (to work with their development team) to figure out what JS script they need to invoke in the browser to get the URL of interest that is sent as a pure https request over the wire. WAAT then takes this request, and does the tag matching for you.

This generation of the JS script is a one-time effort - unless the way the tags are reported changes in the product. Then the test framework can work in a seamless fashion as before to test this is working consistently in an automated fashion. 

Another advantage of this is that with this approach, we do not need to install jpcap or run the tests as a "super-user" - a restriction posed by the network packet capture library.

The WAAT_v1.5.0.jar is available on here (https://github.com/anandbagmar/WAAT/tree/master/dist) on github.

Similarly, I have also updated the WAAT-ruby gem (WAAT-1.5.0.gem). This gem is not yet pushed out to rubygems.org - as I am still testing it out. However, if you are interested, you can download it from here.

As usual, feedback / comments / suggestions most welcome!

Whats next for WAAT?

It has been quite some time that I updated WAAT. The released version has been working well - but it does have its limitations as listed in the FAQs on github.

The biggest limitation I feel about the current release version of WAAT is that it does not work in a pure https kind of an environment. (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html)

Of late I have been spiking out different ways to overcome this limitation. I have experimented to create a HttpsSniffer, and hit various different road-blocks in that. That has forced me to look at another strategy. 

So I have changed direction in coming to a solution. I am looking at creating something like a JSInjector / JSSniffer plugin - which executes a javascript in the browser from where the action is invoked. This is not as straight forward to use as the earlier approaches. The user of this plugin will need to understand the DOM and some javascripts better, maybe take help from the development team, but then once the way to retrive the basic information is known, then we are in calm waters again :)

If you are facing this similar issue in a pure https environment for web analytics testing, look out for more information in this space.

My plan is to update WAAT, followed-by WAAT-Ruby and then lastly release a new version of the WAAT-Ruby gem following that.

WAAT release update

I am almost ready with my first public release of WAAT. Some finishing touches remaining which is causing the delay for this.

For those not aware, here is what WAAT is:
> WAAT stands for Web Analytics Automation Testing Framework
> Developed as a Java jar to be used in existing testing frameworks to do web analytics automation
> Phase 1: Implemented for Omniture using Omniture Debugger -> Status: Completed
> Phase 2: Can be used to test *almost any type of Web Analytic solution. Tested with Google Analytics and Omniture. This is platform dependent. -> Status: In progress. Documentation to be updated.
> Phase 3: Make WAAT available for Ruby / .Net testing frameworks. -> Status: To be started.

Look at my earlier post for more details on WAAT.

WAAT-Ruby - available on RubyGems and opensourcetesting

I am very happy to announce that WAAT is now available on RubyGems.org and is also linked from Open source functional testing tools.

Here are the links for the same:

 

https://rubygems.org/gems/WAAT

http://www.opensourcetesting.org/functional.php#WAAT-Ruby

http://www.opensourcetesting.org/functional.php#WAAT-Java

New version of WAAT-ruby gem available

I finally got around to pushing out a new version of WAAT-ruby (1.4.1) on rubygems.org. The only change in this version is the removal of a dependency on a particular version of bundler. See the WAAT-ruby project on github for more information.

Watch this space for a new version of WAAT-ruby that overcomes the limitation of doing web analytics automation for https urls.

WAAT - Web Analytics Automation Testing Framework

[UPDATE]  See my post about how you can get WAAT here (http://essenceoftesting.blogspot.com/2011/04/waat-is-alive.html).

Problem statement:

On one of the projects that I worked on, I needed to test if Omniture reporting was done correctly.

The client relied a lot on Omniture reports to understand and determine the direction of their business. They have a bunch of Omniture tags reported for a lot of different actions on the site. Manual testing was the only way this functionality could be done verified. But given the huge number of tags, it was never possible to be sure that all tags were being reported correctly on a regular basis.

So I came up with a strategy to remove this pain-point.

Approach:

I created a framework in our existing automation framework to do Omniture testing. The intention of creating this framework was:

1. There is minimal impact on existing tests.

2. There should be no need to duplicate the tests just to do Omniture testing.

3. Should be easy to use (specify Omniture tags for various different actions, enable testing, etc.)

How it helped us?

1. We provided a huge and reliable safety net to the client and the development team by having Omniture testing automated.

2. Reduced the manual testing effort required for this type of testing, and instead got some bandwidth to focus on other areas.

Next Steps:

I am making this into a generic framework - a.k.a. WAAT - Web Analytics Automation Testing Framework to enable others doing Omniture testing to easily automate this functionality. This project will be hosted on github.

Phase 1 of this implementation will be for Omniture Debugger and input data provided in xml format. This framework will be available as a jar file.

Phase 2 also now complete includes support for any Web Analytic tool. I have tested this with Google Analytics as well as Omniture (NOT using Omniture Debugger). This uses a generic mechanism to capture packets from the network layer and processes them appropriately. Given this generic approach to work with any Web Analytic tool, the framework does become OS dependent.

Watch this space for more information (instructions, links to github, etc). Also, please do contact me with ideas / suggestions / comments about the same.

WAAT and HTTPS

While most sites use http to report tags to the web analytic tool, there are some cases where http is disabled and all traffic is using https only.

In such cases, there may be a problem in using the generic solution provided by WAAT.

I did some research, analysis and experimentation and here are my findings:

1. jpcap captures raw packets. It does not differentiate about http / https
2. There is no problem in WAAT. All it does it matches packets based on patterns you specify in the tests.
3. Since the requests are https based, WAAT is not able to match the packets, unless you specify encrypted packet identifiers and encrypted data in the xml file. firebug / fiddler / ethereal / wireshark / charles / burp / etc. does something extra in this regard to decode the packet information and show the raw content in the browser / tool.

So the question is what can be done next?

1. If it is possible for you to get the configuration in the test environments changed to have the web analytics request sent out on http (maybe along with https) request, that can resolve the issue. Once in a while you can then verify manually if requests are going out on https.
2. You can use Omniture Debugger - but the limitation in your case is that it will be available for Omniture only and not the other web analytic tools.
3. You can extend the HttpSniffer class (,say HttpsSniffer), and provide implementation to decode the captured packets before doing the validation. However, note that this will be a expensive operation as you will be decoding all the captured packets on the network interfaces on your machine and the packet(s) of your interest will be fractional of those captured.

WAAT out of hibernation ...

I am resuming work on WAAT. If interested in contributing, look at https://waffle.io/anandbagmar/WAAT or https://github.com/anandbagmar/waat/issues

Looking forward to pull requests :)

WAAT v1.5.0 released to rubygems.org

The newest version of WAAT (Web Analytics Automation Testing framework) for Java and Ruby is now available with a new plugin - JsSniffer.

Following the announcement of the new JsSniffer plugin for WAAT, I have now completed the testing for the same, and released the gem - WAAT-1.5.0.gem to rubygems.org.

Please take a look at the FAQs and the major changes on Ruby / major changes on Java section for some known issues, potential solutions for the same - as some of these changes could break your tests.

WAAT Java v1.5.1 released today

After a long time, and with lot of push from collaborators and users of WAAT, I have finally updated WAAT (Java) and made a new release today. 

You can get this new version - v1.5.1 directly from the project's dist directory.

Once I get some feedback, I will also update WAAT-ruby with these changes.

Here is the list of changes in WAAT_v1.5.1:

Changes in v1.5.1

• Engine.isExpectedTagPresentInActualTagList in engine class is made public
  
• Updated Engine to work without creating testData.xml file, and directly sending exceptedSectionList for tags
  

  Added a new method
      Engine.verifyWebAnalyticsData(String actionName, ArrayList
   expectedSectionList, String[] urlPatterns, int minimumNumberOfPackets)

• Added an empty constructor for Section.java to prevent marshalling error
  
• Support Fragmented Packets
  
• Updated Engine to support Pattern comparison, instead of String contains
  

Do let me know if you see any problems / issues with this update.

Thanks.

vodQA Pune - Less Talk, Only Action - Agenda, Abstract and workshop pre-requisites

vodQA-Pune - Less Talk, Only Action!

When? - Saturday, 27th Aug 2016, 8.30am - 5.30pm

Where? 

ThoughtWorks, Pune 
6th Floor, Binarius Building, 
Beside Sales Tax Office, Shastrinagar, Yerawada 
Pune, Maharashtra 411006 

Map: http://goo.gl/maps/KfuJG

What?

vodQA Pune - Less Talk, Only Action! 
- Registration - 8.30am- 9.15am
- Welcome note - 9.15am

Agenda

Abstracts

1. Automating Web Analytics - Why? How?

Do you know –

• What is Web Analytics?

• How does Web Analytics work?

• Why is it important? How to test Web Analytics?

• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Pre-Reqs:

• JDK 1.8
• IDE (Eclipse / IntelliJ Idea / etc.)
• Git
• Firefox browser compatible with Selenium v2.53.1 (v46)
• Also refer here for steps to run Appium tests using WAAT

2. Performance testing with Gatling for Beginners

Gatling is a server side performance testing tool. This workshop aims at giving introduction to Gatling and facilitating participants to write their first performance tests using Gatling.

> Brief intro to Gatling

> Using Postman to check stub server(created using mountebank for workshop)

> Write a sample test using gatling (pre set up machines are provided)

Pre-Reqs:

• Participants should know at least one programming language well (Gatling tests are written in Scala. But participants need not know Scala)
• Prior knowledge of what 'Performance testing is?' would help.

System requirements:

• Homebrew (Package Manager for Mac)
• Java 8
• Gradle
• Intellij IDE(Add Scala plug in)

3. Game of Test Automation

We are going to use a game to work out the WHY, WHAT and HOW of test automation within the context of consumers, application, skillset, mindset, etc.

Pre-Reqs: None

4. Security Testing - Operation Vijay

It is now the days of the web. All businesses move their applications online for their customers to use.

Many of these applications contain critical data of the customers such as credit card details, their personal details and so on. These data are very valuable. If these data fall in the wrong hands, they can have disastrous consequences.

Attacks on these systems can destroy the trust that the customers have for the business, they can cause great losses to the customers as well as the business, and so on.

The motivation behind attacks could be different like earning money, earning popularity, destroying a competitor company, etc.

No matter what the intention of the attack is, we need to develop safe applications and we need to know the various vulnerabilities and the consequences of our decisions when we develop applications.

Similarly, we should be aware of the various vulnerabilities before we test the applications so that we can try and exploit it during the testing phase and ensure better quality and safer applications.

Pre-Reqs:

• Java
• Modern Browser like Firefox/Chrome

5. Automate your Mobile tests with Appium

Introduction to Appium.

Appium design for Android and iOS

How to locate elements on android and iOS applications(Inspector).

Hands-on code snippet for android and iOS(Wordpress as sample app)

Generating reports (ExtentReports)

Evolve code snippet written above into a Page Object Framework.

Pre-Reqs:

• IntelliJ
• Java 1.8
• MAVEN
• Node
• Adb - android debug bridge
• Appium installed through node(npm install -g appium). Once the installation goes well. Hit command “appium” without quote and you should see Welcome message from appium.
• Download & install Virtual Box from here
• Download the basic version on Genymotion Android Emulator from here
• Make sure you chose the basic version which is free for use(Go to individual tab).
• You will be asked to create account with Genymotion before downloading the Genymotion package/installer
• Confirm your account by pressing the link sent to your registered email id
• Once you download, install Genymotion Android Emulator by following the instructions here
• Once you install, open/start Genymotion and click on Add(+) link to add a new virtual device and press Signin and login
• Select Android Version 6.0.0 and Device model Google Nexus 5 or 6. For more details follow the instruction mentioned here
• Select from available option, Press Next and wait till download finishes.
• Press start button & you should see emulator running.

6. Increase Automation to REST

What are web services and why we use them?

How to test a web service in multiple ways?

Increased familiarity with automation

Tools used : rest client, postman (mention alternatives), unirest JAVA and requests python

Pre-Reqs:

• IntelliJ
• MAVEN
• CHROME and POSTMAN

7. Let's cook Cucumber

In this workshop we will be covering:

• Advantages of BDD through cucumber example.
• Framework setup along with JAVA and Selenium.
• Writing one end to end test case in real world.
• If time permits - will be covering basic Refactoring

Pre-Reqs:

• IntelliJ
• Java 1.8
• Maven
• Firefox 46

8. Mobile Automation using Espresso

Pre-Reqs:

• Ensure you have Java 7 or beyond, installed on your box. If not you can download it from here 
• Download & install Virtual Box from here
• Download and install Android Studio from here. 
• Also kindly download the android command line tools from the above link.
• If you open android studio, it will ask you download & setup Android API and build tools which is useful for debug & compile our apps so please follow the steps. Ensure you have following things installed before you come for workshop and downloading it takes lot of time. 
  • Android SDK build-tools v23.0
  • Android API 5.0 or above
  • Install Android Support library from Extras. 
  • Please google in case of any issues while installing SDK
• Download the basic version on Genymotion Android Emulator from here
  • Make sure you chose the basic version which is free for use(Go to individual tab).
  • You will be asked to create account with Genymotion before downloading the Genymotion package/installer
  • Confirm your account by pressing the link sent to your registered email id
  • Once you download, install Genymotion Android Emulator by following the instructions here
  • Once you install, open/start Genymotion and click on Add(+) link to add a new virtual device and press Signin and login 
  • Select Android Version 6.0.0 and Device model Google Nexus 5 or 6. For more details follow the instruction mentioned here
  • Select from available option, Press Next and wait till download finishes.
  • Press start button & you should see emulator running.
  • You’re all set for the event.