Using fakesms service in Functional Test Automation

How do you automate OTP related test scenarios? Do you use a fake SMS service? Does it have restapi to query the SMS messages? geography support? 

To clarify - this needs to be done as part of my functional test automation, where,

• the test could be running against a browser, where the browser does not have access to the phone, or,
• the test could be running against a real mobile device (without SIM), so no way to receive the SMS, or,
  
• the test could be running against an emulator (no SIM), so no way to receive the SMS

Scenarios include: login, payment, SMS content 

Hence I am thinking about using a fakesms service which has API access capabilities to retrieve the SMS. This will help when running automation on browser or devices / emulators without SIM.

Note:

• There is no access to DB or API to query the OTP. 
• I don't mind using a paid service

 

Thanks in advance for your help!

Quality & Release Strategy for Native Android & iOS Apps at AppiumConf 2019

What an amazing time speaking at the first AppiumConf 2019 in Bangalore, India. I spoke about my experiences in setting "Quality & Release Strategy for Native Android & iOS Apps"

Abstract:
Experimentation and quick feedback is the key to success of any product, while of course ensuring a good quality product with new and better features is being shipped out at a decent / regular frequency to the users.

In this session, we will discuss how to enable experimentation, get quick feedback and reduce risk for the product by using a case study of a media / entertainment domain product, used by millions of users across 10+ countries - i.e. - we will discuss Testing Strategy and the Release process an Android & iOS Native app - that will help enable CI & CD.

To understand these techniques, we will quickly recap the challenges and quirks of testing Native Apps and how that is different than Web / Mobile Web Apps.

The majority of the discussion will focus on different techniques / practices related to Testing & Releases that can be established to achieve our goals, some of which are listed below:

• Functional Automation approach - identify and automate user scenarios, across supported regions
• Testing approach - what to test, when to test, how to test!
• Manual Sanity before release - and why it was important!
• Staged roll-outs via Google’s Play Store and Apple’s App Store
• Extensive monitoring of the release as users come on board, and comparing the key metrics (ex: consumer engagement) with prior releases
• Understanding Consumer Sentiments (Google’s Play Store / Apple’s App Store review comments, Social Media scans, Issues reported to / by Support, etc.)

Slides:

Quality & Release Strategy for Native Android & iOS Apps from Anand Bagmar

Visual Validation - The Missing Tip of the Automation Pyramid at QuaNTA NXT at Globant

I spoke about Visual Validation - The Missing Tip of the Automation Pyramid at QuaNTA NXT event organised by Globant India Pvt. Ltd.

The event was very well organised and I had the opportunity to interact with a full house, and also later meet and talk with a lot of interesting people - curious about current state of testing, test automation and how AI can impact it in the future.

Agenda:

Below is the abstract of my talk:

The Test Automation Pyramid is not a new concept. While Automation helps validate functionality of your product, the look & feel / user-experience (UX) validation is still mostly manual.

With everyone wanting to be Agile, doing quick releases, this look & feel / UX validation becomes the bottleneck, and also is a very error-prone activity which causes brand, revenue and leads diluting your user-base.

In this session, we will explore why Automated Visual Validation is now essential in your Automation Strategy and also look at how an AI-powered tool - Applitools Eyes, can solve this problem.

Recording from the talk:

Some pictures:

.

Visual validation - The Missing Tip of the Automation Pyramid

At yet-another-vodQA at ThoughtWorks, this time in the Pune edition on 16th March 2019, I spoke about Visual validation - The Missing Tip of the Automation Pyramid

Abstract:

The Test Automation Pyramid is not a new concept. The top of the pyramid is our UI / end-2-end functional tests - which should cover the breadth of the product.

What the functional tests cannot capture though, is the aspects of UX validations that can only be seen and in some cases, captured by the human eye. This is where the new buzzwords of AI & ML can truly help.

In this session, we will explore why Visual Validation is an important cog in the wheel of Test Automation and also different tools and techniques that can help achieve this. We will also see a demo of Applitools Eyes - and how it can be a good option to close this gap in automation!

Slides are available from here

Visual Validation - The Missing Tip of the Automation Pyramid from Anand Bagmar

Video is available here:

Thanks to Priyank Shah for this pic!

I also received some awesome feedback for the same.

Thanks vodQA Team! Till next time, adios!

Talks and workshops in Agile India 2019

In the upcoming Agile India 2019 in Bangalore, I will be speaking about:

If you have not yet registered, you can use this code to get a discount on your registration - anand-10di$c-agile 

In addition, there are some great pre and post conference workshops as well. I will be participating in "Facilitating for Effective Collaboration...One Nudge at a Time" workshop - conducted by Deborah Hartmann Preuss and Ellen Grove. 

This is going to be one amazing conference to learn, network and share ideas and experiences. See you there!

.

Update: Upcoming webinar - The Missing Feedback Loop

Please note that the webinar will be on 21st November. 

Register here (https://hubs.ly/H0fp4by0)

Testing in the Agile World

Thanks to ThoughtWorks, I was introduced to many things - 

• Agile
• Blogging
• Speaking at conferences
• Contributing to Open-source
• ...
• ...

The list is actually quite long - but that is not the intention of this post.

The main takeaway in my learning at ThoughtWorks though, is how to Test better, and be more effective in that for the end-user. 

Even before my time at ThoughtWorks, I never agreed with the thought process that Functional Automation can / should be done only when the feature is stable. But at ThoughtWorks, I did learn many more tips and tricks and techniques and processes how to do this Functional Automation in a better way, as the product is evolving.

On 9th April 2011, I had written a detailed blog post / article regarding how can we test better in the Agile world. 

This post was titled - "Agile QA Process", and the document was uploaded to slideshare with the name - "Agile QA Process". I am very pleasantly surprised that till date, that document has had over 74K views and almost 2.7K downloads, and is still my topmost viewed post on slideshare.

When I look back at the document, it still seems very relevant and applicable, to me! 

What do you think?

Measuring Consumer Quality - The Missing Feedback Loop

I spoke in vodQA at ThoughtWorks, Pune on "Measuring Consumer Quality - the Missing Feedback Loop". 

This talk address the why and how from my earlier blog post on "Understanding, Measuring and Building Consumer Quality". I recommend you read that first, before going through the slides and video for this talk.

Abstract:

How to build a good quality product is not a new topic. Proper usage of methodologies, processes, practices, collaboration techniques can yield amazing results for the team, the organisation, and for the end-users of your product.

While there is a lot of emphasis on the processes and practices side, one aspect that is still spoken about "loosely" - is the feedback loop from your end-users to making better decisions.

SO, What is this feedback loop? Is it a myth? How do you measure it? Is there a "magic" formula to understand this data received? How to you add value to your product using this data?

In this interactive session, we will use a case study of a B2C entertainment-domain product (having millions of consumers) as an example to understand and also answer the following questions:

• The importance of knowing your Consumers 
• How do you know your product is working well? 
• How do you know your Consumers are engaged with your product? 
• Can you draw inferences and patterns from the data to reach of point of being able to make predictions on Consumer behaviour, before making any code change? 

Video:

Slides can be found here.

Pictures:

MAD-LAB - Capabilities & Features - Agile India 2018

I spoke about "Build your own MAD-LAB - for Mobile Test Automation for CD" at Agile India 2018.

Though I have spoken on this similar topic answering the question - "Why I needed to build my own MAD-LAB?" before at vodQA in July 2017 at Vuclip, quite a few things have changed since then.

Knowing the value of "being agile", a day before my scheduled talk in Agile India 2018, I decided to revamp the content substantially. To add to my challenges, (and thanks to "testing" my slides before the talk in the conference room), I also realised the slide size format I was using is incorrect, and also the projector was not "setup / configured" correctly, making all my slide colours go haywire.

So after last 10 minutes of scrambling before the talk time, I managed to get this done correctly (at least that is what I think now in hindsight.

Moral of the above story - do a test / dry-run of your slides before your audience comes in!

That said, here is the abstract of the talk.

Abstract

In this age of a variety of cloud-based-services for virtual Mobile Test Labs, building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it! 

The slides are part of the discussion on the Why, What and How I built my own MAD-LAB (Mobile Automation Devices LAB). The discussion also includes the Automation Strategy, Tech Stack, Capabilities & Features of MAD-LAB and the learnings from successful & failed experiments in the journey. 

Slides

Below are the slides from my talk. The link to the video will be shared once available.

MAD-LAB - Capabilities And Features from Anand Bagmar

Some pictures

Understanding, Measuring and Building Consumer Quality

It has been a long time since I posted anything on my blog. For those who don't know, I am working in Vuclip, a B2C company in the OTT space, where we have millions of consumers using our product via Android and iOS native apps, and the Browser too.

In the past few months, I have been in deep water taking on a new and very exciting initiative. Before I share what that is - here is a traditional approach to Quality.


Typically practices, processes, tools are chosen and implemented to help build a good Quality Product for the end-user. Evolving from Waterfall methodology to Agile methodology has been challenging for many (organizations and individuals), but has proven to be a huge step forward to achieving the goal of building a good and usable product.

In this course of time, we have (thankfully) changed the thought process of considering QA to be the "gate-keeper of Quality" to QA being a "Quality Advocate and Quality Enabler" for the team and the product. A very important change as a result has been changing the focus of QA from "finding defects" to "preventing defects".

And rightly so! After-all, why should the QA be the gate-keeper and -

• take the responsibility and blame of someone giving poor / incomplete requirements? or,
• someone writing bad code during development?

The QA is not a scavenger meant to clean up mess created by others. The QA instead is an enabler who -

• helps bring all stakeholders together through the life-cycle of the product - from conceptualization to end-delivery, 
• asks a lot of questions to find gaps, clarify assumptions, etc.
• helps find and radiate information including risks, and,
• is an active part of doing whatever it takes to prevent defects coming into the system

The Agile practices help do this in a collaborative way, getting features to completion in an incremental fashion, and iterating / pivoting based on the feedback received. This is also what practices related to Continuous Delivery enables us to do well.

But this is nothing new, at least for me. After all, during my fantastic journey at ThoughtWorks, I would say that these were basic tenets of why and how we worked.

That said, my eye-opener in the past few months has been to take this thought process many steps forward.

My agenda has been - how can I help influence and raise the bar of quality in such a way that we not only build a quality product, but also be in a position to predict how our millions of consumers will be able to use it.

This initiative we are calling as Consumer Quality -

• how do we understand Quality (= value) of the product as perceived by our Consumers, 
• what data can be relevant to understand this, how can we be proactive about looking at this data while building a quality product, and,
• the Nirvana stage - how can we predict what actions taken will have desired impact on Consumer Quality!

I hope to be able to share with you more of this in 2018!

Happy New Year everyone! Keep Learning, Keep Sharing!

Analytics - the forgotten child!

After a long time, I spoke about What, Why and How of Analytics Testing at Selenium Conference, Berlin 2017.

This talk was initially supposed to be focussed on Web Analytics only, with impact on / of IoT (Internet of Things) and Big Data, but my recent experiences made me realise, the learnings could easily be applied to Analytics from Mobile native apps as well.

So against better judgement, a full 30 minutes before I was supposed to go on stage, I started a revamp of the slides to include more content, which also meant a complete change of flow of the talk / slides. Talk about making stupid decisions, but thankfully, it turned out pretty ok!!

Abstract of the talk:

What is Web Analytics and why is it important? We'll walk through techniques for manually testing your data and automating the validation process.

Just knowing about Analytics is not sufficient for business now. There are new kids in town - IoT and Big Data - two of the most used and well-known buzz words in the software industry! With a creative mindset looking for opportunities to add value, the possibilities for IoT are infinite. With each such opportunity, there's a huge volume of data being generated which, if analysed and used correctly, can feed into creating more opportunities and increased value propositions.

There are 2 types of analysis that one needs to think about:

1. How is the end-user interacting with the product? - This will give some level of understanding into how to re-position and focus on the true value add features for the product.
2. What are the patterns in the data? - With the huge volume of data being generated by the end-user interactions, and the data being captured by all devices in the food-chain of the offering, it is important to identify patterns and find out new product and value opportunities based on these.

Video from the talk:

Slides from the talk:

The What, Why and How of Analytics Testing from Anand Bagmar

NullPointerException from RemoteWebElement in Selenium via Appium Java-Client 5.0.0-BETA9

As you may be aware from my previous posts about MAD-LAB, we are using Appium, with Java-Client 5.0.0-BETA9 to automate user journeys of the VIU app on Android & iOS devices.

Last week, suddenly, while in the middle of doing another round of significant changes to support more capability in the test framework for the Android app, the tests started failing. All infrastructure pieces were working fine, but when the App launched, I started getting this error:

ERROR AndroidLanguageScreen:16 - [5203bb1ae2771425] - ERROR in clicking on androidElement - 'By.id: tv_one' - exception - 'null'

java.lang.NullPointerException

The code in question was - driver.findByElement(myElementLocator).click()

On further investigation, it seemed that there was a problem in doing any interaction with the app, not just "click".

After lot of racking my head, asked a colleague to see if the problem reproduces on her machine. As she had not run the tests on her machine since a few days, as soon as she ran the test execution command, soon the same error happened on her machine as well. Interestingly though, we observed the following trace in her machine's console logs:

------------

Packages that were updated:

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-support/3.5.1/selenium-support-3.5.1.pom

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-api/3.5.1/selenium-api-3.5.1.pom

Download https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.pom

Download https://repo1.maven.org/maven2/com/google/guava/guava-parent/23.0/guava-parent-23.0.pom

Download https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.pom

Download https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.pom

Download https://repo1.maven.org/maven2/com/google/errorprone/error_prone_parent/2.0.18/error_prone_parent-2.0.18.pom

Download https://repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.pom

Download https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.pom

Download https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-parent/1.14/animal-sniffer-parent-1.14.pom

Download https://repo1.maven.org/maven2/org/codehaus/mojo/mojo-parent/34/mojo-parent-34.pom

Download https://repo1.maven.org/maven2/org/codehaus/codehaus-parent/4/codehaus-parent-4.pom

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/3.5.1/selenium-remote-driver-3.5.1.pom

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-support/3.5.1/selenium-support-3.5.1.jar

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-api/3.5.1/selenium-api-3.5.1.jar

Download https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.jar

Download https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar

Download https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.jar

Download https://repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar

Download https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/3.5.1/selenium-remote-driver-3.5.1.jar

:buildSrc:compileJava UP-TO-DATE

------------

This trace meant that something had changed in the dependencies (automatically), and gradle was fetching newer versions for the same.

This was a smoking gun we were looking for. On investigation for selenium 3.5.1 with appium java-client 5.0.0-BETA9, it quickly showed only 1 hit in search result - which was a bug reported on Java-Client 5.0.0-BETA9 - Warning: Selenium 3.5.1 breaks java client 5.0.0-BETA9

The solution / workaround was also already provided by QAutomatron

configurations.all {
    resolutionStrategy {
        force 'org.seleniumhq.selenium:selenium-support:3.4.0',
                'org.seleniumhq.selenium:selenium-api:3.4.0'
    }
}

This resolved our issue for now.

Why I needed to build my own MAD-LAB

I spoke about "Build your own MAD-LAB - for Mobile Test Automation" at vodQA - The Saga Continues! at Vuclip in collaboration with ThoughtWorks on Sat, 29th July 2017.

Join the vodQA group on facebook / LinkedIn to be part of the vodQA community.

Here are details of the talk:

Description

Building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!


Setting the stage - I am coordinating all Testing activities for VIU - an OTT (over-the-top entertainment) product available on Android, iOS and WAP platforms. This product delivers high quality, popular video content in various different languages for consumers in various different regions. One of the main items in my charter is to implement functional test automation for consumer / user functionalities, and to provide quick feedback to the team and stakeholders on the “true” state of the product on all supported platforms for VIU.


In this talk, using the above set context, I will be sharing the following:

• The automation strategy
• Chosen tech-stack
• How (and why) no cloud-based solution worked for me
• Implementation details - MAD-LAB - which arose from the learnings of the failed experiments - which resulted in setting up my own real-device in-house lab.
• The core implementation (code) of MAD-LAB (already open-sourced)

Takeaways for attendees

• Learning from my experiments (what worked, or didn’t)
• Approach to testing an OTT (entertainment domain) product
• How to build a Test Automation Framework using cucumber-jvm / Appium
• Implementation details to Manage Devices, Optimizing Test Execution via distribution, Appium server, Custom Reporting etc., enabling automatic test execution via CI on each new app build, and more.

Slides

Build your own MAD-LAB - for Mobile Test Automation from Anand Bagmar

Video (talk starts at 04m:45s)

https://www.youtube.com/watch?v=yiRbfhOqQuc#t=04m45s

vodQA - The Saga Continues in Pune!

After a long break, vodQA returns to Pune. This time, ThoughtWorks & Vuclip are jointly hosting vodQA.

At vodQA, we have always strived to focus on the art and practice of Testing. This edition of vodQA is no different. Hence the theme for this vodQA - "The Saga Continues". 

We welcome all roles interested in the (Software) Quality to participate in this conference.

The event will be held on Sat, 29th July 2017 at Vuclip office in Pune. 

Agenda is as below:

Address:

1st Floor, 

Nanasaheb Gaikwad Information Technology Park, 

Sarjaa Rd, Aundh, Pune, Maharashtra 411007

Above Croma

Link to Map - https://goo.gl/maps/4yWaX6atutw