Visual validation - The Missing Tip of the Automation Pyramid

At yet-another-vodQA at ThoughtWorks, this time in the Pune edition on 16th March 2019, I spoke about Visual validation - The Missing Tip of the Automation Pyramid

Abstract:

The Test Automation Pyramid is not a new concept. The top of the pyramid is our UI / end-2-end functional tests - which should cover the breadth of the product.

What the functional tests cannot capture though, is the aspects of UX validations that can only be seen and in some cases, captured by the human eye. This is where the new buzzwords of AI & ML can truly help.

In this session, we will explore why Visual Validation is an important cog in the wheel of Test Automation and also different tools and techniques that can help achieve this. We will also see a demo of Applitools Eyes - and how it can be a good option to close this gap in automation!

Slides are available from here

Visual Validation - The Missing Tip of the Automation Pyramid from Anand Bagmar

Video is available here:

Thanks to Priyank Shah for this pic!

I also received some awesome feedback for the same.

Thanks vodQA Team! Till next time, adios!

Is the Future of Test Automation I predicted already here?

Today, almost at the end of 2018, I have come across many tools focused on making Test Automation, easier, faster, reliable and more valuable to the teams & the product - like testim.io, testcraft.io, katalon, cypress.io, mabl, test.ai, etc. These tools are very interesting and very promising for the value proposition they are bringing to the table. 

As I reflect on these shiny new tools, my mind wanders back to 2009 / 2010 when I was toying with the idea of what would be next in Test Automation Tools & Infrastructure space. I had penned my thoughts and published an article on ThoughtWorks Insights with the title - "Future of Test Automation Tools & Infrastructure" (https://www.thoughtworks.com/insights/blog/future-test-automation-tools-infrastructure). 

If we look deeper in my post, the tools I mentioned above (and many others that I probably am unaware of), are conceptually on the lines of what I had sort-of thought in 2009 / 2010. They are using a very interesting blend of past experiences, in some cases advanced technology like AI & ML, in some cases leveraging cloud / SaaS model, and more importantly - pushing the boundaries to do things differently! I am personally very happy to see this happen.

That gets another set of questions in my mind now - if what I had thought of back then is now true, and a reality, then what is next? What will the next generation of new, interesting, shiny tools look like in the next 5 years?

Testing in the Agile World

Thanks to ThoughtWorks, I was introduced to many things - 

• Agile
• Blogging
• Speaking at conferences
• Contributing to Open-source
• ...
• ...

The list is actually quite long - but that is not the intention of this post.

The main takeaway in my learning at ThoughtWorks though, is how to Test better, and be more effective in that for the end-user. 

Even before my time at ThoughtWorks, I never agreed with the thought process that Functional Automation can / should be done only when the feature is stable. But at ThoughtWorks, I did learn many more tips and tricks and techniques and processes how to do this Functional Automation in a better way, as the product is evolving.

On 9th April 2011, I had written a detailed blog post / article regarding how can we test better in the Agile world. 

This post was titled - "Agile QA Process", and the document was uploaded to slideshare with the name - "Agile QA Process". I am very pleasantly surprised that till date, that document has had over 74K views and almost 2.7K downloads, and is still my topmost viewed post on slideshare.

When I look back at the document, it still seems very relevant and applicable, to me! 

What do you think?

Essence of Testing - A new beginning

In my career so far, I have been very fortunate to have got great mentors, and a variety of opportunities to learn, add value, and share my experiences with others around me.

Here are some of these experiences:

• Worked in various sized organisations across the globe in the past couple of decades
• The teams have been big and small
• Played a variety of roles - Quality Analyst, SDET (Software Development Engineer in Test), Product Quality Engineer (PQE), Automation Engineer, Consultant, Coach, Project Manager, Director - Quality, Support Engineer, etc.
• Worked with teams having products in different domains - Health care, eCommerce, Banking / Finance, Retail, Entertainment (OTT), Research, etc. 
• With in organisations (B2B and B2C space): WebMD, Borland Software, Microsoft (Redmond, USA), AmberPoint (USA / India), ThoughtWorks, Vuclip 
• Shared experiences with other via Meetups and Conferences world-wide
• Created opensource tools like WAAT, TTA, TaaS

After working as an employee in these organisations for almost a couple of decades now, I have taken a plunge to do something different.

I am now looking forward to work with Organisations and Teams, to help co-create optimized solutions towards shipping a quality product. Leave me a message on my blog, or send me an email at abagmar@gmail.com to talk more on how we can work together!

As part of this journey, even before I was able to buy my own laptop, and warm it up, I got an interesting consulting assignment, thanks to a dear friend.

This assignment was exactly what I needed to get started - it was a Discovery Workshop, with the following objective:

• Learn and understand the current state of the team and the product they were building
• Understand current (perceived) challenges
• Suggest improvements for the team in areas of:
• Tweaks in the current processes
• Practices to be adopted / got better at / or stopped (as anti-patterns and not adding value)
• Identify opportunities to Test better, and early
• Suggest a Test Automation Pyramid that is fit-for-purpose for the team
• Suggest strategy, tools and approach for end-2-end (e2e) functional automation

As I usually do, I started off the workshop with my favorite tool for note taking - Mind Maps! As the conversations evolved and got specific, I started with a Balanced Mind-Map  to Fishbone Mind-Map.

Eventually I consolidated my thoughts and created the Discovery Workshop Report for the client in the form of a slide deck, using the following approach.

• Learning from Discovery
• Talk with the Team members
• See & understand the project management tool, quality of requirements, etc.
• See the code - understand complexity, types of tests written, quality of tests, etc.
• See the Testing related artifacts - test cases, test execution strategy, exploratory testing, etc.
• See the CI server - how deployments happen, what causes the builds to fail, etc.
• Recommendations
• For all the above areas, I created recommendations on what different aspects may help the team move ahead in a better way

Discovery and Recommendations were based on each specific activity:

• Process
• Architecture
• Requirements
• Development
• Testing
• Deployments

Once the learning from the Learning and Recommendation stages were well understood (by me), I then created a Suggested Plan of Execution (in phases / milestones)

• From the recommendations, I created milestone based plans on what can be started immediately, and what decisions the team needs to take to move forward in other areas

In this 2 week time, unknowingly, in retrospect, the whole engagement was quite agile. There were periodic demos of the POCs, regular progress sharing, and changing direction of discovery and recommendations based on learning.

The Client also appreciated the quality of conversations, and the results that were shared.

All in all, a very satisfying beginning to my new journey with Essence of Testing!

Measuring Consumer Quality - The Missing Feedback Loop

I spoke in vodQA at ThoughtWorks, Pune on "Measuring Consumer Quality - the Missing Feedback Loop". 

This talk address the why and how from my earlier blog post on "Understanding, Measuring and Building Consumer Quality". I recommend you read that first, before going through the slides and video for this talk.

Abstract:

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about "loosely" - is the feedback loop from your end-users to making better decisions.

SO, What is this feedback loop? Is it a myth? How do you measure it? Is there a "magic" formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers 
• How do you know your product is working well? 
• How do you know your Consumers are engaged with your product? 
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behaviour, before making any code change? 

Video:

Slides can be found here.

Pictures:

Agile Testing & Patterns for a good Test Automation Frameworks

2018 started with a bang! I got an opportunity to speak and share my experiences in 2 rocking meetups.

Patterns of a "good" Test Automation Framework

TechnoWise meetup on 13th Jan on Patterns of a "good" Test Automation Framework went very well, with lot of interaction and discussions along the way.

Patterns of a “good” test automation framework from Anand Bagmar

What is Agile Testing? How does Automation Help?

Then, there was an impromptu meetup setup by a very proactive ISQA community at GO-JEK office in Jakarta, Indonesia on 16th Jan. The topic there was What is Agile Testing? How does Automation help? In this meetup, I also covered aspects of Career Path of a Tester, and QA Skills and Capabilities.

There were many amazing experiences from this meetup -

• The ISQA community is very active. The meetup was setup in literally a few days and there were over 150+ attendees
• The GO-JEK office space is a very fun place. They actually have an auditorium in the office to host meetups and similar activities, apparently, once a week!!
• The questions / interactions with the attendees were very insightful

Here are slides shared in the meetup. I will share the link of the video as well once available.

What is Agile Testing? How does Automation help? from Anand Bagmar

Understanding, Measuring and Building Consumer Quality

It has been a long time since I posted anything on my blog. For those who don't know, I am working in Vuclip, a B2C company in the OTT space, where we have millions of consumers using our product via Android and iOS native apps, and the Browser too.

In the past few months, I have been in deep water taking on a new and very exciting initiative. Before I share what that is - here is a traditional approach to Quality.


Typically practices, processes, tools are chosen and implemented to help build a good Quality Product for the end-user. Evolving from Waterfall methodology to Agile methodology has been challenging for many (organizations and individuals), but has proven to be a huge step forward to achieving the goal of building a good and usable product.

In this course of time, we have (thankfully) changed the thought process of considering QA to be the "gate-keeper of Quality" to QA being a "Quality Advocate and Quality Enabler" for the team and the product. A very important change as a result has been changing the focus of QA from "finding defects" to "preventing defects".

And rightly so! After-all, why should the QA be the gate-keeper and -

• take the responsibility and blame of someone giving poor / incomplete requirements? or,
• someone writing bad code during development?

The QA is not a scavenger meant to clean up mess created by others. The QA instead is an enabler who -

• helps bring all stakeholders together through the life-cycle of the product - from conceptualization to end-delivery, 
• asks a lot of questions to find gaps, clarify assumptions, etc.
• helps find and radiate information including risks, and,
• is an active part of doing whatever it takes to prevent defects coming into the system

The Agile practices help do this in a collaborative way, getting features to completion in an incremental fashion, and iterating / pivoting based on the feedback received. This is also what practices related to Continuous Delivery enables us to do well.

But this is nothing new, at least for me. After all, during my fantastic journey at ThoughtWorks, I would say that these were basic tenets of why and how we worked.

That said, my eye-opener in the past few months has been to take this thought process many steps forward.

My agenda has been - how can I help influence and raise the bar of quality in such a way that we not only build a quality product, but also be in a position to predict how our millions of consumers will be able to use it.

This initiative we are calling as Consumer Quality -

• how do we understand Quality (= value) of the product as perceived by our Consumers, 
• what data can be relevant to understand this, how can we be proactive about looking at this data while building a quality product, and,
• the Nirvana stage - how can we predict what actions taken will have desired impact on Consumer Quality!

I hope to be able to share with you more of this in 2018!

Happy New Year everyone! Keep Learning, Keep Sharing!

Why I needed to build my own MAD-LAB

I spoke about "Build your own MAD-LAB - for Mobile Test Automation" at vodQA - The Saga Continues! at Vuclip in collaboration with ThoughtWorks on Sat, 29th July 2017.

Join the vodQA group on facebook / LinkedIn to be part of the vodQA community.

Here are details of the talk:

Description

Building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!


Setting the stage - I am coordinating all Testing activities for VIU - an OTT (over-the-top entertainment) product available on Android, iOS and WAP platforms. This product delivers high quality, popular video content in various different languages for consumers in various different regions. One of the main items in my charter is to implement functional test automation for consumer / user functionalities, and to provide quick feedback to the team and stakeholders on the “true” state of the product on all supported platforms for VIU.


In this talk, using the above set context, I will be sharing the following:

• The automation strategy
• Chosen tech-stack
• How (and why) no cloud-based solution worked for me
• Implementation details - MAD-LAB - which arose from the learnings of the failed experiments - which resulted in setting up my own real-device in-house lab.
• The core implementation (code) of MAD-LAB (already open-sourced)

Takeaways for attendees

• Learning from my experiments (what worked, or didn’t)
• Approach to testing an OTT (entertainment domain) product
• How to build a Test Automation Framework using cucumber-jvm / Appium
• Implementation details to Manage Devices, Optimizing Test Execution via distribution, Appium server, Custom Reporting etc., enabling automatic test execution via CI on each new app build, and more.

Slides

Build your own MAD-LAB - for Mobile Test Automation from Anand Bagmar

Video (talk starts at 04m:45s)

https://www.youtube.com/watch?v=yiRbfhOqQuc#t=04m45s

vodQA - The Saga Continues in Pune!

After a long break, vodQA returns to Pune. This time, ThoughtWorks & Vuclip are jointly hosting vodQA.

At vodQA, we have always strived to focus on the art and practice of Testing. This edition of vodQA is no different. Hence the theme for this vodQA - "The Saga Continues". 

We welcome all roles interested in the (Software) Quality to participate in this conference.

The event will be held on Sat, 29th July 2017 at Vuclip office in Pune. 

Agenda is as below:

Address:

1st Floor, 

Nanasaheb Gaikwad Information Technology Park, 

Sarjaa Rd, Aundh, Pune, Maharashtra 411007

Above Croma

Link to Map - https://goo.gl/maps/4yWaX6atutw

A new beginning - entertainment on mobile

After 7+ years, I finally took the heavy step and moved out of ThoughtWorks.

The past 7+ years have been awesome. I had loads of fun, learnt many new things, made a lot of friends and found inspiration and guidance from a lot of mentors.

Thank you ThoughtWorks and ThoughtWorkers! Wouldn’t have been who I am today without you and you all will always be a huge part of me!

Taking the decision was tougher than I thought it would be ... but new challenges were waiting for me, and the time had come.

On 1st December, 2016, I started my next stint as "Directory - Quality" at Vuclip, Inc for the Viu product. You can also find us via the PlayStore or AppStore.

Day 1 at vuclip, barring the first 2 hours of paperwork, was getting right into action. With the planning for 2017 in full swing, there was no time to settle - but instead had to hit the ground running.

The charter starting Day 1 for me was:

• Define & execute test strategy for Viu - for multiple platforms, for multiple regions & partners
• Build team to help execute the above (see section below on what I am looking out for)
• In scope - functional testing, automation, performance, analytics, benchmarking, infrastructure, tooling, etc.
• Out of scope - nothing

And so the fun has begun.

So, here is what I need to learn and execute immediately (looking forward to suggestions, links, feedback on how you have done it in the past)

• Is it worth setting up a mobile lab (real devices + simulators) in-house or use external services for running automated tests  exploratory tests? 
• If the latter to the above, what services have you used in the past? What have the results been?
• Is it possible (and worth) automating the checks for memory / processor / battery usage when running tests against the native app (on Android & iOS)?
• How to do native app performance testing (client-side) for Android & iOS?

Also, I am looking to build a strong testing team with team members having the primary skills & capabilities -

• Open-minded, quick learner
• A good Testing-mindset
• Mobile Testing experience (non-automated + test automation)
• Performance Testing (client-side & server-side)

Contact me if you are interested in being part of my team to work on this challenging product.

Slides from vodQA Pune - Less Talk, Only Action! now available

vodQA-Pune - Less Talk, Only Action! was held on on Saturday, 27th Aug 2016, 8.30am - 5.30pm at ThoughtWorks, Pune.

Agenda

Abstracts with Slides

1. Automating Web Analytics - Why? How?

Do you know –

• What is Web Analytics?How does Web Analytics work?
• Why is it important? How to test Web Analytics?
• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Slides: automating-web-analytics

2. Performance testing with Gatling for Beginners

Gatling is a server side performance testing tool. This workshop aims at giving introduction to Gatling and facilitating participants to write their first performance tests using Gatling.

• Brief intro to Gatling
• Using Postman to check stub server(created using mountebank for workshop)
• Write a sample test using gatling (pre set up machines are provided)

Slides: gatling-performance-workshop

3. Game of Test Automation

We are going to use a game to work out the WHY, WHAT and HOW of test automation within the context of consumers, application, skillset, mindset, etc.

4. Security Testing - Operation Vijay

It is now the days of the web. All businesses move their applications online for their customers to use.

Many of these applications contain critical data of the customers such as credit card details, their personal details and so on. These data are very valuable. If these data fall in the wrong hands, they can have disastrous consequences.

Attacks on these systems can destroy the trust that the customers have for the business, they can cause great losses to the customers as well as the business, and so on.

The motivation behind attacks could be different like earning money, earning popularity, destroying a competitor company, etc.

No matter what the intention of the attack is, we need to develop safe applications and we need to know the various vulnerabilities and the consequences of our decisions when we develop applications.

Similarly, we should be aware of the various vulnerabilities before we test the applications so that we can try and exploit it during the testing phase and ensure better quality and safer applications.

Slides: security-testing-operation-vijay

5. Automate your Mobile tests with Appium

• Introduction to Appium.
• Appium design for Android and iOS
• How to locate elements on android and iOS applications(Inspector).
• Hands-on code snippet for android and iOS(Wordpress as sample app)
• Generating reports (ExtentReports)
• Evolve code snippet written above into a Page Object Framework.
  

Slides: mobile-automation-using-appium

6. Increase Automation to REST

• What are web services and why we use them?
• How to test a web service in multiple ways?
• Increased familiarity with automation

Tools used : rest client, postman (mention alternatives), unirest JAVA and requests python

Slides: increase-automation-to-rest, api-webservice-setup-instructions

7. Let's cook Cucumber

In this workshop we will be covering:

• Advantages of BDD through cucumber example.
• Framework setup along with JAVA and Selenium.
• Writing one end to end test case in real world.
• If time permits - will be covering basic Refactoring
  

Slides: lets-cook-cucumber
 

8. Mobile Automation using Espresso

Imagine a situation where every commit spits out a build that can be deployed to production with confidence. In today's startup era, this can be a huge boost to business as it will reduce the time to market. UI Automation for mobile apps, be it native or hybrid, has been painful since long. But with mature frameworks coming up and Google/Apple realizing the importance of such tools, UI Automation is gaining traction in the mobile space. 

This talk is basically to understand what and why of espresso along with automating a simple scenario using espresso.

Slides: getting-high-on-espresso

WAAT 2.0 (BETA) available for use

Very excited to share that I, along with a few collaborators of WAAT did a workshop in vodQA, Pune on Sat, 27th Aug about Automating Web Analytics - Why? How?. 
We implemented the automation using WAAT.

Also, am very excited to announce that WAAT v2.0 BETA for Java is now available from the WAAT project.

The slides from the workshop are available here

Automating Web Analytics from Anand Bagmar

Here are some details of the workshop:

Abstract

Do you know –

• What is Web Analytics?

• How does Web Analytics work?

• Why is it important? How to test Web Analytics?

• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

 
We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

 
We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

 
Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Pre-Reqs

• JDK 1.8
• IDE (Eclipse / IntelliJ Idea / etc.)
• Git (and add it to path)
• Gradle 2.3 (and add it to path)
• Firefox browser compatible with Selenium v2.53.1 (v46)
• Also refer here for steps to run Appium tests using WAAT 

 

Sample Code

- Create a directory WAAT-Workshop and run
git clone git@github.com:anandbagmar/waat-sample-java.git
- If Using any existing framework: download WAAT-all-v2.0_BETA.jar
http://goo.gl/8bfyaJ
- To run sample code - from waat-sample-java directory - run the command -
gradle clean build 

Please reach out if you need help in using WAAT!