Analytics - The Brain of the Software

An Analogy 

I am not a doctor, nor did I enjoy biology too much in my curriculum as a student.  However, I do know that the body has many organs and each organ plays a vital role in the well being of the individual.

Each organ has to:

• function correctly (movement, senses, core functions, etc.)

• has to perform as per expectations in different conditions the individual may be going through (walking, running, swimming, etc.)

• has to be secure from external parameters (heat, cold, rain, what we eat / drink, etc.)

• has to have a proper user experience (ex: if the human hands had webs like ducks, would we be able to hold a pen correctly to write?

• I would like to think of the brain as the super computer which keeps track of what is going on in the body, if each piece playing its part correctly, or not. And if there is something unexpected going on, then there are mechanisms of giving that feedback internally and externally so that course correction would be possible.

How does this relate to software?

Software is similar in some ways. For any software product to work, the following needs to be done:

Functionality works as expected

• The architecture, testability of the system will allow for various types of testing activities to be performed on the software to ensure everything works as expected
• Test Automation practices will give you quick feedback

There is a plethora of open-source and commercial tools in this space to help in this regard - the most popular open-source tools being Selenium and Appium.

Software is performant

• We can do performance testing at various different levels to ensure at different loads and conditions, the users will be able to use the product in a seamless fashion
• There are many tools to assist in Performance Testing - some popular ones being JMeter and Gatling.

Software is secure

• Building and testing for security is critical as you do not want user information to be leaked or manipulated and neither do you want to allow external forces to control / manipulate your product behaviour and control
• The Test Automation Pyramid hence also includes NFRs

User experience is validated, and consistent

• In the age of CD (Continuous Delivery & Continuous Deployment), you need to ensure your user experience across all your software delivery means (browsers, mobile-browsers, native apps for mobiles and tablets, etc.) is consistent and users do not face the brunt of UI / look-and-feel issues in the software at the cost of new features
• This is a relatively new domain - but there are already many tools to help in this spaces as well - the most popular one (in terms of integration, usage and features) being the AI-powered Applitools

Visual Validation is the new tip of the Test Automation Pyramid!

What is the brain of the software?

The above is all good, and known in various ways. But what is the "brain" of the software? How does one know if everything is working fine or not? Who will receive the feedback and how do we take corrective action on this?

Analytics is that piece in the Software product that functions as the brain. It keeps collecting data about each important piece of software, and provides feedback on the same.

I have come across some extreme examples of Business / Organizations who have all their eggs in one basket - in terms of

• understand their Consumers (engagement / usage / patterns / etc.),
• understand usage of product features, and,
• do all revenue-related book-keeping

This is all done purely on Analytics! Hence, to say “Business runs on Analytics, and it may be OK for some product / user features to not work correctly, but Analytics should always work” - is not a myth!

What this means is Analytics is more important now, than before.

Unfortunately, Analytics is not known much to the Software Dev + Test community. We know it very superficially - and do what is required to implement it and quickly test it out. But what is analytics? Why is it important? What is the impact of this not working well? Not many think about this.

I have been testing Analytics since 2010 ... and the kind of insights I have been able to get about the product have been huge! I have been able to contribute back to the team and help build better quality software as a result.

But I have to be honest - it is painful to test Analytics. And that is why I created an open-source framework - WAAT - to help automate some of this testing activities.

I also do workshops to help people learn more about Analytics, its importance, and how they can automate this as well.

In the workshop, I do not assume anything and approach is to discuss and learn by example and practice, the following

• How does Analytics works (for Web and Mobile)?
• Test Analytics manually in different ways
• Test Analytics via the final reports
• Why some Automation strategies will work, and some WILL NOT WORK (based on my experience)!
• We will see a demo of the Automation running for the same.
• Time permitting, we will setup running some Automation scripts on your machine to validate the same

Takeaways from the workshop

We will learn by practicing the following:

• What is Analytics?
• Techniques to test analytics manually.
• How to automate the validation of analytics, via a demo, and if time permits, run the automation from your machine as well.

Hope this post helps you understand the importance of Analytics and why you need to know more about it. Do reach out to me if you want to learn more about it.

Next upcoming Analytics workshop is in TestBash Australia 2019. Let me know if you would be interested in attending the same

Recording from webinar on The Missing Feedback Loop now available

On 21st Nov, TestCraft.io hosted me in a webinar where I spoke about - "The Missing Feedback Loop - The Tools, Techniques, and Automation to Solve It". 

You can get the recording from here (https://hubs.ly/H0fBDN50).

Update: Upcoming webinar - The Missing Feedback Loop

Please note that the webinar will be on 21st November. 

Register here (https://hubs.ly/H0fp4by0)

Upcoming webinar - The Missing Feedback Loop

I am very excited to share that I am going to conduct a webinar hosted by testcraft.io on "The Missing Feedback Loop - The Tools, Techniques, and Automation to Solve It". 

You can register for the webinar from here (https://hubs.ly/H0fp4by0).

Date & Time:
Thursday, November 21, 2018 at 02:00 PM New-York (EDT), 11:00 AM San-Francisco (PDT) and 08:00 PM Amsterdam (UTC+2)

Agile Testing, Analytics Testing and Measuring Consumer Quality from Poland and USA

The last few weeks have been very hectic for me. In between my consulting assignments, I traveled to Krakow, Poland for Agile & Automation Days 2018, and then to Arlington, Virginia in USA for STPCon Fall 2018.

In the Agile & Automation Days 2018 conference, I spoke about "Measuring Consumer Quality - The Missing Feedback Loop" and conducted a 1/2 day workshop on "Analytics Rebooted - A Workshop".

In STPCon Fall 2018, I conducted 2 workshops - 1/2 day each - "Practical Agile Testing Workshop" and "Analytics Rebooted - A Workshop" and also spoke about "Measuring Consumer Quality - The Missing Feedback Loop"

Overall, I had a very good trip, amazing conversations and interactions with the attendees and the speakers. I would be lying if I say I am not tired and my throat has gone sore. But, would I do this again? Absolutely! Going to conferences and meeting people, sharing my experiences with them, and learning from their experiences gives me a lot of happiness and satisfaction.

Below are the abstracts of the workshops and the talk. 

Contact me via LinkedIn, or twitter, or my site - essenceoftesting.com if you need any additional information, or if you want help in learning / implementing these or other topics related to Quality / Testing / Automation.

Practical Agile Testing Workshop

Workshop Description:

The Agile Manifesto was published in 2001. It took the software industry a good few years to truly understand what the manifesto means, and the principles behind it. However, choosing and implementing the right set of practices to get the true value from working the Agile way has been the biggest challenge for most!

While Agile is now mainstream, and as we get better at the development practices to “being Agile”, Testing has still been lagging behind in most cases. A lot of teams are still working in the staggered fashion (a.k.a. Iterative waterfall way of working). Here teams may be testing after development completes, or Automation is done in the next Iteration / Sprint, etc.

In this workshop, we will learn and share various principles and practices which teams should adopt to be successful in testing (in-cycle) in Agile projects.

Workshop Agenda:

• What is Agile testing? - Learn what does it mean to Test on Agile Projects
• Effective strategies for Distributed Testing - Learn practices that help bridge the Distributed Testing gap!
• Test Automation in Agile Projects - Why? What? How? - Why is Test Automation important, and how do we implement a good, robust, scalable and maintainable Test Automation framework!
• Build the "right" regression suite using Behavior Driven Testing (BDT) - Behavior Driven Testing (BDT) is an evolved way of thinking about Testing. It helps in identifying the 'correct' scenarios, in form of user journeys, to build a good and effective (manual & automation) regression suite that validates the Business Goals. 

Key learning for participants in this workshop:

• Understand the Agile Testing Manifesto.
• Learn the essential Testing practices and activities essential for teams to adopt to work in Agile way of working.
• Discover techniques to do effective testing in distributed teams.
• Find out how Automation plays a crucial role in Agile projects.
• Learn how to build a good, robust, scalable and maintainable Functional Automation framework.
• Learn, by practice, how to identify the right types of tests to automate as UI functional tests - to get quick and effective feedback.

Analytics Rebooted – A Workshop

Workshop Description:

I have come across some extreme examples of Business / Organizations who have all their eggs in one basket - in terms of # understand their Consumers (engagement / usage / patterns / etc.), # understand usage of product features, and, # do all revenue-related book-keeping

This is all done purely on Analytics! Hence, to say “Business runs on Analytics, and it may be OK for some product / user features to not work correctly, but Analytics should always work” - is not a myth!

What this means is Analytics is more important now, than before.

In this workshop, we will not assume anything. We will discuss and learn by example and practice, the following:

• How does Analytics works (for Web & Mobile)? 
• Test Analytics manually in different ways 
• Test Analytics via the final reports
• Why some Automation strategies will work, and some WILL NOT WORK (based on my experience)!
• We will see demo of the Automation running for the same.
• Time permitting, we will setup running some Automation scripts on your machine to validate the same

Measuring Consumer Quality – The Missing Feedback Loop

Session Description:

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organization, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about “loosely” - is the feedback loop from your end-users to making better decisions.

SO, what is this feedback loop? Is it a myth? How do you measure it? Is there a “magic” formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers
• How do you know your product is working well?
• How do you know your Consumers are engaged with your product?
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behavior, before making any code change?

Attendees will have deeper understanding and appreciation of the following:

• What is Consumer Quality and how does it help shape your business!
• Ways to measure Consumer Quality
• Why is understanding Consumer Engagement vital to the success of your product

Measuring Consumer Quality - The Missing Feedback Loop

I spoke in vodQA at ThoughtWorks, Pune on "Measuring Consumer Quality - the Missing Feedback Loop". 

This talk address the why and how from my earlier blog post on "Understanding, Measuring and Building Consumer Quality". I recommend you read that first, before going through the slides and video for this talk.

Abstract:

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about "loosely" - is the feedback loop from your end-users to making better decisions.

SO, What is this feedback loop? Is it a myth? How do you measure it? Is there a "magic" formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers 
• How do you know your product is working well? 
• How do you know your Consumers are engaged with your product? 
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behaviour, before making any code change? 

Video:

Slides can be found here.

Pictures:

Analytics - the forgotten child!

After a long time, I spoke about What, Why and How of Analytics Testing at Selenium Conference, Berlin 2017.

This talk was initially supposed to be focussed on Web Analytics only, with impact on / of IoT (Internet of Things) and Big Data, but my recent experiences made me realise, the learnings could easily be applied to Analytics from Mobile native apps as well.

So against better judgement, a full 30 minutes before I was supposed to go on stage, I started a revamp of the slides to include more content, which also meant a complete change of flow of the talk / slides. Talk about making stupid decisions, but thankfully, it turned out pretty ok!!

Abstract of the talk:

What is Web Analytics and why is it important? We'll walk through techniques for manually testing your data and automating the validation process.

Just knowing about Analytics is not sufficient for business now. There are new kids in town - IoT and Big Data - two of the most used and well-known buzz words in the software industry! With a creative mindset looking for opportunities to add value, the possibilities for IoT are infinite. With each such opportunity, there's a huge volume of data being generated which, if analysed and used correctly, can feed into creating more opportunities and increased value propositions.

There are 2 types of analysis that one needs to think about:

1. How is the end-user interacting with the product? - This will give some level of understanding into how to re-position and focus on the true value add features for the product.
2. What are the patterns in the data? - With the huge volume of data being generated by the end-user interactions, and the data being captured by all devices in the food-chain of the offering, it is important to identify patterns and find out new product and value opportunities based on these.

Video from the talk:

Slides from the talk:

The What, Why and How of Analytics Testing from Anand Bagmar

Analytics Testing

I recently spoke in an Agile Testing Conference on - The What, Why and How of (Web) Analytics Testing.

I was also part of a panel discussion having the theme - "What's not changed since moving to Agile Testing - The Legacy Continues!" There were some very interesting perspectives in this discussion.

The great part was that the audience was very involved and vocal throughout the day. This made is very interactive and good sharing of information and experiences for all!

Below is some information about the talk. I will try to add the link to the video soon.
 

Abstract

Analytics is changing the way products and services are being created and consumed.

In this session, we will learn

• What is Analytics?
• Why is it important to use Analytics in your product?
• The impact of Analytics not working as expected

We will also see some techniques to test Analytics manually and also automate that validation. But just knowing about Analytics is not sufficient for business now.

There are new kids in town - IoT and Big Data - two of the most used and heard-off buzz words in the Software Industry!

With IoT, with a creative mindset looking for opportunities and ways to add value, the possibilities are infinite. With each such opportunity, there is a huge volume of data being generated - which if analyzed and used correctly, can feed into creating more opportunities and increased value propositions.

There are 2 types of analysis that one needs to think about.

1. How is the end-user interacting with the product? This will give some level of understanding into how to re-position and focus on the true value add features for the product.

2. With the huge volume of data being generated by the end-user interactions, and the data being captured by all devices in the food-chain of the offering, it is important to identify patterns from what has happened, and find out new product / value opportunities based on usage patterns.

Slides

The What, Why and How of (Web) Analytics Testing (Web, IoT, Big Data) from Anand Bagmar

Pictures

WAAT out of hibernation ...

I am resuming work on WAAT. If interested in contributing, look at https://waffle.io/anandbagmar/WAAT or https://github.com/anandbagmar/waat/issues

Looking forward to pull requests :)

Slides from vodQA Pune - Less Talk, Only Action! now available

vodQA-Pune - Less Talk, Only Action! was held on on Saturday, 27th Aug 2016, 8.30am - 5.30pm at ThoughtWorks, Pune.

Agenda

Abstracts with Slides

1. Automating Web Analytics - Why? How?

Do you know –

• What is Web Analytics?How does Web Analytics work?
• Why is it important? How to test Web Analytics?
• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Slides: automating-web-analytics

2. Performance testing with Gatling for Beginners

Gatling is a server side performance testing tool. This workshop aims at giving introduction to Gatling and facilitating participants to write their first performance tests using Gatling.

• Brief intro to Gatling
• Using Postman to check stub server(created using mountebank for workshop)
• Write a sample test using gatling (pre set up machines are provided)

Slides: gatling-performance-workshop

3. Game of Test Automation

We are going to use a game to work out the WHY, WHAT and HOW of test automation within the context of consumers, application, skillset, mindset, etc.

4. Security Testing - Operation Vijay

It is now the days of the web. All businesses move their applications online for their customers to use.

Many of these applications contain critical data of the customers such as credit card details, their personal details and so on. These data are very valuable. If these data fall in the wrong hands, they can have disastrous consequences.

Attacks on these systems can destroy the trust that the customers have for the business, they can cause great losses to the customers as well as the business, and so on.

The motivation behind attacks could be different like earning money, earning popularity, destroying a competitor company, etc.

No matter what the intention of the attack is, we need to develop safe applications and we need to know the various vulnerabilities and the consequences of our decisions when we develop applications.

Similarly, we should be aware of the various vulnerabilities before we test the applications so that we can try and exploit it during the testing phase and ensure better quality and safer applications.

Slides: security-testing-operation-vijay

5. Automate your Mobile tests with Appium

• Introduction to Appium.
• Appium design for Android and iOS
• How to locate elements on android and iOS applications(Inspector).
• Hands-on code snippet for android and iOS(Wordpress as sample app)
• Generating reports (ExtentReports)
• Evolve code snippet written above into a Page Object Framework.
  

Slides: mobile-automation-using-appium

6. Increase Automation to REST

• What are web services and why we use them?
• How to test a web service in multiple ways?
• Increased familiarity with automation

Tools used : rest client, postman (mention alternatives), unirest JAVA and requests python

Slides: increase-automation-to-rest, api-webservice-setup-instructions

7. Let's cook Cucumber

In this workshop we will be covering:

• Advantages of BDD through cucumber example.
• Framework setup along with JAVA and Selenium.
• Writing one end to end test case in real world.
• If time permits - will be covering basic Refactoring
  

Slides: lets-cook-cucumber
 

8. Mobile Automation using Espresso

Imagine a situation where every commit spits out a build that can be deployed to production with confidence. In today's startup era, this can be a huge boost to business as it will reduce the time to market. UI Automation for mobile apps, be it native or hybrid, has been painful since long. But with mature frameworks coming up and Google/Apple realizing the importance of such tools, UI Automation is gaining traction in the mobile space. 

This talk is basically to understand what and why of espresso along with automating a simple scenario using espresso.

Slides: getting-high-on-espresso

WAAT 2.0 (BETA) available for use

Very excited to share that I, along with a few collaborators of WAAT did a workshop in vodQA, Pune on Sat, 27th Aug about Automating Web Analytics - Why? How?. 
We implemented the automation using WAAT.

Also, am very excited to announce that WAAT v2.0 BETA for Java is now available from the WAAT project.

The slides from the workshop are available here

Automating Web Analytics from Anand Bagmar

Here are some details of the workshop:

Abstract

Do you know –

• What is Web Analytics?

• How does Web Analytics work?

• Why is it important? How to test Web Analytics?

• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

 
We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

 
We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

 
Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Pre-Reqs

• JDK 1.8
• IDE (Eclipse / IntelliJ Idea / etc.)
• Git (and add it to path)
• Gradle 2.3 (and add it to path)
• Firefox browser compatible with Selenium v2.53.1 (v46)
• Also refer here for steps to run Appium tests using WAAT 

 

Sample Code

- Create a directory WAAT-Workshop and run
git clone git@github.com:anandbagmar/waat-sample-java.git
- If Using any existing framework: download WAAT-all-v2.0_BETA.jar
http://goo.gl/8bfyaJ
- To run sample code - from waat-sample-java directory - run the command -
gradle clean build 

Please reach out if you need help in using WAAT!

Agenda published for vodQA Pune - Less Talk, Only Action!

The agenda for the upcoming vodQA Pune - Less Talk, Only Action! is now published.


NOTE: For this event, we will require the participants to get their laptops (and chargers).

When? 

Saturday, 27th Aug 2016, 8.30am - 5.30pm

Where? 

ThoughtWorks, Pune 
6th Floor, Binarius Building, 
Beside Sales Tax Office, Shastrinagar, Yerawada 
Pune, Maharashtra 411006 

Map: 

http://goo.gl/maps/KfuJG

What?

vodQA Pune - Less Talk, Only Action! 
- Registration - 8.30am- 9.15am
- Welcome note - 9.15am

Topics covered: 

This vodQA will focus on hands-on activities. We are setting up a bunch of workshops for participants, please keep a watch for details and pre-requisites required for the workshops.

We will publish the agenda very soon in vodQA groups on 
Facebook (https://www.facebook.com/groups/vodqa/) and 
LinkedIn (https://www.linkedin.com/groups/4281359)

NOTE: For this event, we will require the participants to get their laptops (and chargers).

vodQA Pune - Less Talk, Only Action!

Very happy to announce the next edition of vodQA - 'Less Talk, Only Action!' with focus on hands-on activities. 

You are required to register & submit for this vodQA edition by filling in the below google form - https://goo.gl/forms/We3002DOlkQXmPsN2

There are limited seats for the workshops, available on first-come-first-served basis. 

NOTE: For this event, we will require the participants to get their laptops (and chargers).

When? 

Saturday, 27th Aug 2016, 8.30am - 5.30pm

Where? 

ThoughtWorks, Pune 
6th Floor, Binarius Building, 
Beside Sales Tax Office, Shastrinagar, Yerawada 
Pune, Maharashtra 411006 

Map: 

http://goo.gl/maps/KfuJG

What?

vodQA Pune - Less Talk, Only Action! 
- Registration - 8.30am- 9.15am
- Welcome note - 9.15am

Topics covered: 

This vodQA will focus on hands-on activities. We are setting up a bunch of workshops for participants, please keep a watch for details and pre-requisites required for the workshops.

We will publish the agenda very soon in vodQA groups on 
Facebook (https://www.facebook.com/groups/vodqa/) and 
LinkedIn (https://www.linkedin.com/groups/4281359)

NOTE: For this event, we will require the participants to get their laptops (and chargers).

Agenda will be published soon.

Any WAAT (Web Analytics Automation Testing Framework) users out there?

It has been over 2 years since any update to WAAT - Java or Ruby. Over the years, I have realised, and also received a lot of thoughts / feedback from users of WAAT around where it helps, and what challenges exist. 

Also, given the widespread IoT & Big Data based work going on around the world, (Web) Analytics now plays a much bigger role in guiding business take better decisions. 

WAAT (again) fits in the grand scheme of things very nicely as a framework to automate the validation of correct reporting of tags to any Web Analytics solution provider.

Hence, its a no-brainer for me - it is high time I work on some of the feedback and limitations of WAAT to make it usable again!

At the recently concluded Selenium Conference 2016 held in Bangalore, India, I got an idea of how to overcome a lot of challenges (listed below) and pain in using WAAT. 

What's next?

To implement my new idea, this does mean a couple of things:

• Existing plugins have limited use - and needs to be deleted.
• A new plugin would need to be created - which may mean different set of APIs, and also different way to specify the test data.

Questions for you

Before I go ahead making these changes - I would like to get answers to the below questions (please add your answers directly in the comments):

• Is anyone currently using WAAT? If yes - 
• which version (Java / Ruby)?
• which plugin
• Using HTTP / HTTPS?
• Which Web Analytic solution are you using? (ex: Google Analytics, WebTrends, etc?)
• Would you be interested in using the new WAAT? If yes - 
• Which language? Java / Ruby / JavaScript / Python / etc?
• Would you like to contribute to implementing this new WAAT? If yes - contact me! :)

-----------------------------------------------------------------------------------------------------------------

Current plugins available in WAAT:

• Omniture Debugger (WAAT-Java)
• Pros:
• OS independent
• Run using regular-test-user 
• Cons:
• Browser dependent - need to implement ScriptRunner for the UI-driver in use
• Web-Analytic solution dependent - only for Adobe Marketing Clout / Omniture SiteCatalyst
• HTTPSniffer (WAAT-Java, WAAT-Ruby)
• Pros
• Web-Analytic solution independent
• Browser independent
• UI-Driver independent
• Cons
• 3rd party libraries are OS dependent
• HTTPs is not supported out-of-the-box
• Run tests as "root"
• JSSniffer (WAAT-Java, WAAT-Ruby)
• Pros
• Web-Analytic solution independent
• Browser independent
• HTTPs supported out-of-the-box
• No 3rd party library dependency
• Cons
• Need to write JavaScript to get the URL from the browser context
• UI-Driver dependent

-----------------------------------------------------------------------------------------------------------------