Analytics - The Brain of the Software

An Analogy 

I am not a doctor, nor did I enjoy biology too much in my curriculum as a student.  However, I do know that the body has many organs and each organ plays a vital role in the well being of the individual.

Each organ has to:

• function correctly (movement, senses, core functions, etc.)

• has to perform as per expectations in different conditions the individual may be going through (walking, running, swimming, etc.)

• has to be secure from external parameters (heat, cold, rain, what we eat / drink, etc.)

• has to have a proper user experience (ex: if the human hands had webs like ducks, would we be able to hold a pen correctly to write?

• I would like to think of the brain as the super computer which keeps track of what is going on in the body, if each piece playing its part correctly, or not. And if there is something unexpected going on, then there are mechanisms of giving that feedback internally and externally so that course correction would be possible.

How does this relate to software?

Software is similar in some ways. For any software product to work, the following needs to be done:

Functionality works as expected

• The architecture, testability of the system will allow for various types of testing activities to be performed on the software to ensure everything works as expected
• Test Automation practices will give you quick feedback

There is a plethora of open-source and commercial tools in this space to help in this regard - the most popular open-source tools being Selenium and Appium.

Software is performant

• We can do performance testing at various different levels to ensure at different loads and conditions, the users will be able to use the product in a seamless fashion
• There are many tools to assist in Performance Testing - some popular ones being JMeter and Gatling.

Software is secure

• Building and testing for security is critical as you do not want user information to be leaked or manipulated and neither do you want to allow external forces to control / manipulate your product behaviour and control
• The Test Automation Pyramid hence also includes NFRs

User experience is validated, and consistent

• In the age of CD (Continuous Delivery & Continuous Deployment), you need to ensure your user experience across all your software delivery means (browsers, mobile-browsers, native apps for mobiles and tablets, etc.) is consistent and users do not face the brunt of UI / look-and-feel issues in the software at the cost of new features
• This is a relatively new domain - but there are already many tools to help in this spaces as well - the most popular one (in terms of integration, usage and features) being the AI-powered Applitools

Visual Validation is the new tip of the Test Automation Pyramid!

What is the brain of the software?

The above is all good, and known in various ways. But what is the "brain" of the software? How does one know if everything is working fine or not? Who will receive the feedback and how do we take corrective action on this?

Analytics is that piece in the Software product that functions as the brain. It keeps collecting data about each important piece of software, and provides feedback on the same.

I have come across some extreme examples of Business / Organizations who have all their eggs in one basket - in terms of

• understand their Consumers (engagement / usage / patterns / etc.),
• understand usage of product features, and,
• do all revenue-related book-keeping

This is all done purely on Analytics! Hence, to say “Business runs on Analytics, and it may be OK for some product / user features to not work correctly, but Analytics should always work” - is not a myth!

What this means is Analytics is more important now, than before.

Unfortunately, Analytics is not known much to the Software Dev + Test community. We know it very superficially - and do what is required to implement it and quickly test it out. But what is analytics? Why is it important? What is the impact of this not working well? Not many think about this.

I have been testing Analytics since 2010 ... and the kind of insights I have been able to get about the product have been huge! I have been able to contribute back to the team and help build better quality software as a result.

But I have to be honest - it is painful to test Analytics. And that is why I created an open-source framework - WAAT - to help automate some of this testing activities.

I also do workshops to help people learn more about Analytics, its importance, and how they can automate this as well.

In the workshop, I do not assume anything and approach is to discuss and learn by example and practice, the following

• How does Analytics works (for Web and Mobile)?
• Test Analytics manually in different ways
• Test Analytics via the final reports
• Why some Automation strategies will work, and some WILL NOT WORK (based on my experience)!
• We will see a demo of the Automation running for the same.
• Time permitting, we will setup running some Automation scripts on your machine to validate the same

Takeaways from the workshop

We will learn by practicing the following:

• What is Analytics?
• Techniques to test analytics manually.
• How to automate the validation of analytics, via a demo, and if time permits, run the automation from your machine as well.

Hope this post helps you understand the importance of Analytics and why you need to know more about it. Do reach out to me if you want to learn more about it.

Next upcoming Analytics workshop is in TestBash Australia 2019. Let me know if you would be interested in attending the same

Test Automation Pyramid

Quality & Release Strategy for Native Android & iOS Apps at AppiumConf 2019

What an amazing time speaking at the first AppiumConf 2019 in Bangalore, India. I spoke about my experiences in setting "Quality & Release Strategy for Native Android & iOS Apps"

Abstract:
Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

Slides:

Quality & Release Strategy for Native Android & iOS Apps from Anand Bagmar

Visual Validation - The Missing Tip of the Automation Pyramid at QuaNTA NXT at Globant

I spoke about Visual Validation - The Missing Tip of the Automation Pyramid at QuaNTA NXT event organised by Globant India Pvt. Ltd.

The event was very well organised and I had the opportunity to interact with a full house, and also later meet and talk with a lot of interesting people - curious about current state of testing, test automation and how AI can impact it in the future.

Agenda:

Below is the abstract of my talk:

The Test Automation Pyramid is not a new concept. While Automation helps validate functionality of your product, the look & feel / user-experience (UX) validation is still mostly manual.

With everyone wanting to be Agile, doing quick releases, this look & feel / UX validation becomes the bottleneck, and also is a very error-prone activity which causes brand, revenue and leads diluting your user-base.

In this session, we will explore why Automated Visual Validation is now essential in your Automation Strategy and also look at how an AI-powered tool - Applitools Eyes, can solve this problem.

Recording from the talk:

Some pictures:

.

Collaboration - A Taboo!

In AgileIndia 2019 in Bangalore, as part of the Agile Mindset theme, I played a tweak of the Taboo game - to make it a Collaboration game.

Abstract: 

When one has fun at work, work becomes fun. However, daily pressures, metrics, KPIs, and what not, have dissolved the fun, and made work drudgery in various ways. 

This creates stress for individuals, in teams, and across teams, there is mistrust, unnecessary competition, blame, finger-pointing ….

What better way to learn, and re-learn the basics of life, work, team-work - than to play a game, have fun, and correlate it with how life and work indeed should be treated as a game, and we should have fun in this journey. Only then can people truly succeed, and so can organisations.

Here, we will play a game – “Collaboration - A Taboo!” – where you will 

• Re-learn collaboration techniques via a game! 
• Learning applicable for individuals and teams, in small or big organisations
• Re-live your childhood when playing this game

Be prepared for a twist which will leave you thinking!

Slides:

Collaboration - A Taboo! from Anand Bagmar



.

Visual validation - The Missing Tip of the Automation Pyramid

At yet-another-vodQA at ThoughtWorks, this time in the Pune edition on 16th March 2019, I spoke about Visual validation - The Missing Tip of the Automation Pyramid

Abstract:

The Test Automation Pyramid is not a new concept. The top of the pyramid is our UI / end-2-end functional tests - which should cover the breadth of the product.

What the functional tests cannot capture though, is the aspects of UX validations that can only be seen and in some cases, captured by the human eye. This is where the new buzzwords of AI & ML can truly help.

In this session, we will explore why Visual Validation is an important cog in the wheel of Test Automation and also different tools and techniques that can help achieve this. We will also see a demo of Applitools Eyes - and how it can be a good option to close this gap in automation!

Slides are available from here

Visual Validation - The Missing Tip of the Automation Pyramid from Anand Bagmar

Video is available here:

Thanks to Priyank Shah for this pic!

I also received some awesome feedback for the same.

Thanks vodQA Team! Till next time, adios!

Talks and workshops in Agile India 2019

In the upcoming Agile India 2019 in Bangalore, I will be speaking about:

If you have not yet registered, you can use this code to get a discount on your registration - anand-10di$c-agile 

In addition, there are some great pre and post conference workshops as well. I will be participating in "Facilitating for Effective Collaboration...One Nudge at a Time" workshop - conducted by Deborah Hartmann Preuss and Ellen Grove. 

This is going to be one amazing conference to learn, network and share ideas and experiences. See you there!

.

Test Automation in the World of AI and ML

My article on "Test Automation in the World of AI & ML" recently got published on InfoQ.


Here are the key takeaways mentioned in the article -

• There are many criteria to be considered before building framework / selecting tools for Functional Test Automation
• It is very important to prioritise framework / tools capabilities needed for the software-under-test
• A good, scalable Test Automation Framework that provides fast and reliable feedback to the team enables collaboration and CI/CD
• Debugging / RCA (root cause analysis) and support for libraries / tools used is an afterthought in most cases. Do not fall in that trap.
• There are some promising commercial tools that fit seamlessly in the Agile way of working. Depending on the complete context, these tools may be a good choice over building your own framework for Functional Automation.

You can read the full article from here. 

Looking forward to comments on the same!

.

Recording from webinar on The Missing Feedback Loop now available

On 21st Nov, TestCraft.io hosted me in a webinar where I spoke about - "The Missing Feedback Loop - The Tools, Techniques, and Automation to Solve It". 

You can get the recording from here (https://hubs.ly/H0fBDN50).

A blog about my blog

With the risk of this ending up becoming a recursion, I want to share interesting statistics about my blog - essenceoftesting.blogspot.com and content shared on my SlideShare account.

• First blog: Future of Test Automation Tools & Infrastructure, published on 25th Aug 2010
• Number of blogs published: 218, not counting this one
• Number of viewers: 400K+

Here are some charts from Analytics of my SlideShare and blog:

Top content from my SlideShare:

• Agile QA Process - 75K+ views, 2.75K+ downloads
• Career Path of a Tester - 1.4K views, 50 downloads

Blog Overview:

Audience:

Popular posts:

• Career Path of a Tester!
• Introducing MAD LAB - for Mobile Automation
• WAAT - Web Analytics Automation Testing Framework
• Any WAAT (Web Analytics Automation Testing Framework) users out there?
• Taking Protractor to the next level
• Features of my Android Test Automation Framework

Referrers:

Update: Upcoming webinar - The Missing Feedback Loop

Please note that the webinar will be on 21st November. 

Register here (https://hubs.ly/H0fp4by0)

Is the Future of Test Automation I predicted already here?

Today, almost at the end of 2018, I have come across many tools focused on making Test Automation, easier, faster, reliable and more valuable to the teams & the product - like testim.io, testcraft.io, katalon, cypress.io, mabl, test.ai, etc. These tools are very interesting and very promising for the value proposition they are bringing to the table. 

As I reflect on these shiny new tools, my mind wanders back to 2009 / 2010 when I was toying with the idea of what would be next in Test Automation Tools & Infrastructure space. I had penned my thoughts and published an article on ThoughtWorks Insights with the title - "Future of Test Automation Tools & Infrastructure" (https://www.thoughtworks.com/insights/blog/future-test-automation-tools-infrastructure). 

If we look deeper in my post, the tools I mentioned above (and many others that I probably am unaware of), are conceptually on the lines of what I had sort-of thought in 2009 / 2010. They are using a very interesting blend of past experiences, in some cases advanced technology like AI & ML, in some cases leveraging cloud / SaaS model, and more importantly - pushing the boundaries to do things differently! I am personally very happy to see this happen.

That gets another set of questions in my mind now - if what I had thought of back then is now true, and a reality, then what is next? What will the next generation of new, interesting, shiny tools look like in the next 5 years?

Upcoming webinar - The Missing Feedback Loop

I am very excited to share that I am going to conduct a webinar hosted by testcraft.io on "The Missing Feedback Loop - The Tools, Techniques, and Automation to Solve It". 

You can register for the webinar from here (https://hubs.ly/H0fp4by0).

Date & Time:
Thursday, November 21, 2018 at 02:00 PM New-York (EDT), 11:00 AM San-Francisco (PDT) and 08:00 PM Amsterdam (UTC+2)

Agile Testing, Analytics Testing and Measuring Consumer Quality from Poland and USA

The last few weeks have been very hectic for me. In between my consulting assignments, I traveled to Krakow, Poland for Agile & Automation Days 2018, and then to Arlington, Virginia in USA for STPCon Fall 2018.

In the Agile & Automation Days 2018 conference, I spoke about "Measuring Consumer Quality - The Missing Feedback Loop" and conducted a 1/2 day workshop on "Analytics Rebooted - A Workshop".

In STPCon Fall 2018, I conducted 2 workshops - 1/2 day each - "Practical Agile Testing Workshop" and "Analytics Rebooted - A Workshop" and also spoke about "Measuring Consumer Quality - The Missing Feedback Loop"

Overall, I had a very good trip, amazing conversations and interactions with the attendees and the speakers. I would be lying if I say I am not tired and my throat has gone sore. But, would I do this again? Absolutely! Going to conferences and meeting people, sharing my experiences with them, and learning from their experiences gives me a lot of happiness and satisfaction.

Below are the abstracts of the workshops and the talk. 

Contact me via LinkedIn, or twitter, or my site - essenceoftesting.com if you need any additional information, or if you want help in learning / implementing these or other topics related to Quality / Testing / Automation.

Practical Agile Testing Workshop

Workshop Description:

The Agile Manifesto was published in 2001. It took the software industry a good few years to truly understand what the manifesto means, and the principles behind it. However, choosing and implementing the right set of practices to get the true value from working the Agile way has been the biggest challenge for most!

While Agile is now mainstream, and as we get better at the development practices to “being Agile”, Testing has still been lagging behind in most cases. A lot of teams are still working in the staggered fashion (a.k.a. Iterative waterfall way of working). Here teams may be testing after development completes, or Automation is done in the next Iteration / Sprint, etc.

In this workshop, we will learn and share various principles and practices which teams should adopt to be successful in testing (in-cycle) in Agile projects.

Workshop Agenda:

• What is Agile testing? - Learn what does it mean to Test on Agile Projects
• Effective strategies for Distributed Testing - Learn practices that help bridge the Distributed Testing gap!
• Test Automation in Agile Projects - Why? What? How? - Why is Test Automation important, and how do we implement a good, robust, scalable and maintainable Test Automation framework!
• Build the "right" regression suite using Behavior Driven Testing (BDT) - Behavior Driven Testing (BDT) is an evolved way of thinking about Testing. It helps in identifying the 'correct' scenarios, in form of user journeys, to build a good and effective (manual & automation) regression suite that validates the Business Goals. 

Key learning for participants in this workshop:

• Understand the Agile Testing Manifesto.
• Learn the essential Testing practices and activities essential for teams to adopt to work in Agile way of working.
• Discover techniques to do effective testing in distributed teams.
• Find out how Automation plays a crucial role in Agile projects.
• Learn how to build a good, robust, scalable and maintainable Functional Automation framework.
• Learn, by practice, how to identify the right types of tests to automate as UI functional tests - to get quick and effective feedback.

Analytics Rebooted – A Workshop

Workshop Description:

I have come across some extreme examples of Business / Organizations who have all their eggs in one basket - in terms of # understand their Consumers (engagement / usage / patterns / etc.), # understand usage of product features, and, # do all revenue-related book-keeping

This is all done purely on Analytics! Hence, to say “Business runs on Analytics, and it may be OK for some product / user features to not work correctly, but Analytics should always work” - is not a myth!

What this means is Analytics is more important now, than before.

In this workshop, we will not assume anything. We will discuss and learn by example and practice, the following:

• How does Analytics works (for Web & Mobile)? 
• Test Analytics manually in different ways 
• Test Analytics via the final reports
• Why some Automation strategies will work, and some WILL NOT WORK (based on my experience)!
• We will see demo of the Automation running for the same.
• Time permitting, we will setup running some Automation scripts on your machine to validate the same

Measuring Consumer Quality – The Missing Feedback Loop

Session Description:

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organization, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about “loosely” - is the feedback loop from your end-users to making better decisions.

SO, what is this feedback loop? Is it a myth? How do you measure it? Is there a “magic” formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers
• How do you know your product is working well?
• How do you know your Consumers are engaged with your product?
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behavior, before making any code change?

Attendees will have deeper understanding and appreciation of the following:

• What is Consumer Quality and how does it help shape your business!
• Ways to measure Consumer Quality
• Why is understanding Consumer Engagement vital to the success of your product

Conference season here is - talks, workshops, travelling, networking!

September & October 2018 is a busy conference season for me.

On 27th September, I played a game - "Collaboration - A Taboo!" at ATA GTR 2018 with an audience of 100+ people. There was absolute chaos in the game - a lot of it self-inflicted ... and thankfully - exactly was I wanted it to be. So much fun, energy and enthusiasm in the room meant there was no one feeling drowsy in the post lunch session! 

Typically I play this game in 45-min to 1 hour duration. At ATA GTR 2018 though, I had only 30 min to play the game, and add my own twist on top of it. But, never have I ever taken more than the allocated time - and I managed to get the objectives of the game achieved as well in these 30 min.

Below are some pictures from the game.




Then on 28th September, I spoke on "Measuring Consumer Quality - The Missing Feedback Loop" at StepIn's PSTC 2018. Slides from that talk can be found here.

In October, I will be off to Agile & Automation Days in Krakow, Poland. Here I will be speaking about "Measuring Consumer Quality - The Missing Feedback Loop" and also conducting a workshop on - "Analytics Rebooted - A Workshop". See detailed schedule here. 

Then I fly directly to Arlington, VA to participate in STPCon Fall 2018. Here I will be conducting 2 workshops - "Analytics Rebooted - A Workshop" and "Practical Agile Testing Workshop". I am also speaking about "Measuring Consumer Quality - The Missing Feedback Loop".

Will share experiences from these conferences soon!

Interesting reads - Data, Analytics, Security, DevOps, Python

Here is some interesting reading material I came across today - related to Data, Analytics, Security, DevOps and Python

• AI & Analytics in Production  
• Case Studies: Machine Learning
• Putting the Power of Kafka into the Hands of Data Scientists
• Building a Real-Time Bike-Share Data Pipeline with StreamSets, Kafka and MapD
• Real-time data foundations: Kafka - The what, why, and how of Kafka
• A Short Summary of Smoothing Algorithms
• AVA: The Art and Science of Image Discovery at Netflix
• Deploy only what you trust: introducing Binary Authorization for Google Kubernetes Engine
• A Clean Start: Finding Vulnerabilities in your Docker Base Images
• The Best User Behavior Analytics for Your Web or Mobile Product
• Mozilla identifies 10 open source personas: What you need to know
• 18 Python programming books for beginners and veterans
• Making the Adobe Cloud More Resilient — with Chaos
• 6 Patterns for Continuous Delivery with Serverless Applications
• How Kubernetes Is Transforming into a Universal Scheduler
• InSpec: Auditing and Testing Framework
• ChaoSlingr: Introducing Security into Chaos Testing
• Retire.js - What you require you must also retire

Happy Reading to you!

Testing in the Agile World

Thanks to ThoughtWorks, I was introduced to many things - 

• Agile
• Blogging
• Speaking at conferences
• Contributing to Open-source
• ...
• ...

The list is actually quite long - but that is not the intention of this post.

The main takeaway in my learning at ThoughtWorks though, is how to Test better, and be more effective in that for the end-user. 

Even before my time at ThoughtWorks, I never agreed with the thought process that Functional Automation can / should be done only when the feature is stable. But at ThoughtWorks, I did learn many more tips and tricks and techniques and processes how to do this Functional Automation in a better way, as the product is evolving.

On 9th April 2011, I had written a detailed blog post / article regarding how can we test better in the Agile world. 

This post was titled - "Agile QA Process", and the document was uploaded to slideshare with the name - "Agile QA Process". I am very pleasantly surprised that till date, that document has had over 74K views and almost 2.7K downloads, and is still my topmost viewed post on slideshare.

When I look back at the document, it still seems very relevant and applicable, to me! 

What do you think?

Some good examples of Data Science, AI & ML

Following up on my earlier post about ODSC - Data Science, AI, ML - Hype, or Reality?, I thought it is good to also share some of the good examples of work happening in the field.

Here are some of the examples I got to hear in the ODSC conference, most of which are available to the common human:

• Amazing work done in the complex field of Speech recognition 
• Why complex? Think about languages, dialects, multiple conversations at the same time, different speed of talking, etc.
• Text to speech
• Ex: This is especially very helpful for people with disabilities
• Speech to Text
• Ex: Alexa, Google Voice, etc. type of applications
• Traffic control / Routes / Navigation
• Ex: Google Maps
• Recommendation engines
• Ex: eCommerce products
• Preventive maintenance
• Lot of advanced vehicles have a number of sensors that can alert the driver / car manufacturer about potential issues coming up / service due for the vehicle
• Autonomous vehicles
• Ex: Self driving vehicles
• Ex: Optimizing Cab scheduling / routing - There was a good session on how OLA manage its complexity in scheduling and routing - which is very applicable to eCommerce, Aviation industry, Hotel industry, etc.
• I recently also saw a video about Volvo truck driver getting out of the truck in a difficult terrain, and walking in front of the truck, controlling its movement using a game-like controller
• Medical equipment / gadgets for preventive / alerting health-care products

Also, Dr. Ravi Mehrotra, from IDeaS made a very powerful statement in his keynote - that I loved!! 

He said - "Best way to learn, is to forget what is not important".

This statement resonates a lot with what I think .... one needs to forget what is not (as) important, in order to focus and prioritize on what is important and can add value.

Especially true for Testers to keep in mind!