NullPointerException from RemoteWebElement in Selenium via Appium Java-Client 5.0.0-BETA9

As you may be aware from my previous posts about MAD-LAB, we are using Appium, with Java-Client 5.0.0-BETA9 to automate user journeys of the VIU app on Android & iOS devices.

Last week, suddenly, while in the middle of doing another round of significant changes to support more capability in the test framework for the Android app, the tests started failing. All infrastructure pieces were working fine, but when the App launched, I started getting this error:

ERROR AndroidLanguageScreen:16 - [5203bb1ae2771425] - ERROR in clicking on androidElement - 'By.id: tv_one' - exception - 'null'

java.lang.NullPointerException

The code in question was - driver.findByElement(myElementLocator).click()

On further investigation, it seemed that there was a problem in doing any interaction with the app, not just "click".

After lot of racking my head, asked a colleague to see if the problem reproduces on her machine. As she had not run the tests on her machine since a few days, as soon as she ran the test execution command, soon the same error happened on her machine as well. Interestingly though, we observed the following trace in her machine's console logs:

------------

Packages that were updated:

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-support/3.5.1/selenium-support-3.5.1.pom

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-api/3.5.1/selenium-api-3.5.1.pom

Download https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.pom

Download https://repo1.maven.org/maven2/com/google/guava/guava-parent/23.0/guava-parent-23.0.pom

Download https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.pom

Download https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.pom

Download https://repo1.maven.org/maven2/com/google/errorprone/error_prone_parent/2.0.18/error_prone_parent-2.0.18.pom

Download https://repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.pom

Download https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.pom

Download https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-parent/1.14/animal-sniffer-parent-1.14.pom

Download https://repo1.maven.org/maven2/org/codehaus/mojo/mojo-parent/34/mojo-parent-34.pom

Download https://repo1.maven.org/maven2/org/codehaus/codehaus-parent/4/codehaus-parent-4.pom

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/3.5.1/selenium-remote-driver-3.5.1.pom

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-support/3.5.1/selenium-support-3.5.1.jar

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-api/3.5.1/selenium-api-3.5.1.jar

Download https://repo1.maven.org/maven2/com/google/guava/guava/23.0/guava-23.0.jar

Download https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar

Download https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.jar

Download https://repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar

Download https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar

Download https://repo1.maven.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/3.5.1/selenium-remote-driver-3.5.1.jar

:buildSrc:compileJava UP-TO-DATE

------------

This trace meant that something had changed in the dependencies (automatically), and gradle was fetching newer versions for the same.

This was a smoking gun we were looking for. On investigation for selenium 3.5.1 with appium java-client 5.0.0-BETA9, it quickly showed only 1 hit in search result - which was a bug reported on Java-Client 5.0.0-BETA9 - Warning: Selenium 3.5.1 breaks java client 5.0.0-BETA9

The solution / workaround was also already provided by QAutomatron

configurations.all {
    resolutionStrategy {
        force 'org.seleniumhq.selenium:selenium-support:3.4.0',
                'org.seleniumhq.selenium:selenium-api:3.4.0'
    }
}

This resolved our issue for now.

Why I needed to build my own MAD-LAB

I spoke about "Build your own MAD-LAB - for Mobile Test Automation" at vodQA - The Saga Continues! at Vuclip in collaboration with ThoughtWorks on Sat, 29th July 2017.

Join the vodQA group on facebook / LinkedIn to be part of the vodQA community.

Here are details of the talk:

Description

Building a real-(mobile)-device lab for Test Automation is NOT a common thing – it is difficult, high maintenance, expensive! Yet, I had to do it!


Setting the stage - I am coordinating all Testing activities for VIU - an OTT (over-the-top entertainment) product available on Android, iOS and WAP platforms. This product delivers high quality, popular video content in various different languages for consumers in various different regions. One of the main items in my charter is to implement functional test automation for consumer / user functionalities, and to provide quick feedback to the team and stakeholders on the “true” state of the product on all supported platforms for VIU.


In this talk, using the above set context, I will be sharing the following:

• The automation strategy
• Chosen tech-stack
• How (and why) no cloud-based solution worked for me
• Implementation details - MAD-LAB - which arose from the learnings of the failed experiments - which resulted in setting up my own real-device in-house lab.
• The core implementation (code) of MAD-LAB (already open-sourced)

Takeaways for attendees

• Learning from my experiments (what worked, or didn’t)
• Approach to testing an OTT (entertainment domain) product
• How to build a Test Automation Framework using cucumber-jvm / Appium
• Implementation details to Manage Devices, Optimizing Test Execution via distribution, Appium server, Custom Reporting etc., enabling automatic test execution via CI on each new app build, and more.

Slides

Build your own MAD-LAB - for Mobile Test Automation from Anand Bagmar

Video (talk starts at 04m:45s)

https://www.youtube.com/watch?v=yiRbfhOqQuc#t=04m45s

vodQA - The Saga Continues in Pune!

After a long break, vodQA returns to Pune. This time, ThoughtWorks & Vuclip are jointly hosting vodQA.

At vodQA, we have always strived to focus on the art and practice of Testing. This edition of vodQA is no different. Hence the theme for this vodQA - "The Saga Continues". 

We welcome all roles interested in the (Software) Quality to participate in this conference.

The event will be held on Sat, 29th July 2017 at Vuclip office in Pune. 

Agenda is as below:

Address:

1st Floor, 

Nanasaheb Gaikwad Information Technology Park, 

Sarjaa Rd, Aundh, Pune, Maharashtra 411007

Above Croma

Link to Map - https://goo.gl/maps/4yWaX6atutw

Changing logcat buffer size in Android devices ... almost works

My (debug-build of) app under test logs extra information about test execution to system logs which is accessible via logcat on Android devices. This is very powerful as now I can run my cucumber-jvm / Appium tests, copy the logcat file after the test execution completes, parse it for relevant information, and do appropriate assertions on the same.

The default buffer size on Android devices I have seen is 256kb. This is less for me - as I end up losing the earlier information, and hence my assertions fail.

Thankfully, there is a programmatic way to change the logcat buffer size in the device before running tests. The command is

adb logcat -G 3M

This adb command works in the Motorola devices in my MAD LAB, but does not work in Samsung devices. The error I see on running the above command is "failed to set the log size"

Any idea why this would not work in Samsung devices? or rather, what do I need to do to change the logcat buffer size?

[UPDATE] - Interestingly - this works on Samsung Galaxy S7, but NOT in Samsung J5 Prime OR Samsung J7 Prime

Criteria for setting up a Mobile Test Automation LAB

I recently got asked this question related to the MAD LAB (Mobile Automation Devices LAB) - "Would like to understand how can we setup something similar in our organisation?"

Since this question is applicable for all those thinking of, or have already set up their own lab, thought I would share my answer here.

To setup your own LAB for Mobile Test Automation, multiple things need to align:

Supportive management who -

• allows experiments (within reason of course) and encourages learning through failure, 
• willing to invest in infrastructure ($$)

Skilled and Passionate team members who -

• understand the domain well, 
• willing to learn, experiment, re-learn and fail fast, 
• keep looking for innovative solutions to solve problems on hand, 
• do not reinvent the wheel. 

Philosophy aside, our MAD LAB has the following: 

• Mac Minis (8-12 devices per Mac Mini), 
• Powered USB Hubs (I use the ones shown below - and they are working pretty well)

• High-quality USB cables (I use the ones shown below - and they are working pretty well)

• CI (Jenkins) setup correctly to keep running tests continuously, proper reporting  in place (else whats the use of running tests if you do not look at the results)

You could start with similar IF it fits your product-under-test context

After I answered this on LinkedIn, I realised, there are more parameters to think about, than just the above.

• Knowing which devices to use in your Lab
• Having good, reliable Internet connection
• Devices should be "seen" easily
• Should be easy to work on / with the devices as and when required
• Know how you the devices will be placed in the lab. We tried the following:
• 2-way tape - that didn't work. Devices used to stay up for a few days, then "drop" suddenly. Of course, that also depends on the back surface of the devices.
• We tried many mobile stands / hangers (shown below) - but each had their own limitations

• Finally I found an industrial-strength velcro (1" velcro tape that could take a couple of pounds of weight) - and my devices have not budged since. PS: Please be careful when putting on this velcro on the devices. IF it gets on your hand, you will have a velcro tattoo for a long long time.

What other parameters would you consider for setting up your own Lab? Looking forward to the comments below.

Introducing MAD LAB - for Mobile Automation

The past few months I have been heads-down in stabilising my Real-Device Mobile Test Lab - which we now call MAD LAB (Mobile Automation Devices LAB) .

For those who may not recollect, see my past posts for reference -

• A new beginning - entertainment on mobile

• How to enable seamless running of appium tests on developer machines?

• Features of my Android Test Automation Framework

• How to upgrade the appium-uiautomator2-driver version for appium 1.6.3?

• Finding my way out of bottomless pit with Appium & Android 7.0 for parallel test runs

• Sharing implementation of cucumber-jvm - Appium test framework

Along with my colleagues, we have put in lot of effort in setting up MAD LAB and have now added a lot of rich features to help running tests, seeing the results and making sense out of them easier. 

• All infrastructure management is implemented now in groovy (instead of gradle as shared earlier).

• All automatic test execution is managed via Jenkins file

• Actual test implementation is done in cucumber-jvm / java

List of features currently implemented:

• Device management (selection, cleanup, app install and uninstall)

• Parallel test execution (at Cucumber scenario level) - maximising device utilisation)

• Appium server management

• Adb utilities 

• Managing periodic ADB server disconnects

• Custom reporting using cucumber-reports

• Video recording of each scenario and embedding in the custom reports

Contents of MAD LAB:

• 1 Mac Minis - running various Jenkins Agents

• 2 Powered USB hubs

• 8 Android devices

Here are some pictures from the setup.

There are many more features, in various stages of implementation, being added to make MAD LAB more powerful.

Sneak peek into whats coming:

• Analytics Testing

• Trend and Failure Analysis 

• iOS

• Web

• A transformed MAD LAB

Finding MAD LAB interesting? Some very interesting changes are coming in soon. Watch out for my next blog post for that. 

Want to contribute and be part of this journey? Even better! Reach out to me!

Patterns in Test Automation Framework at STPCon

I spoke about Patterns of a "good" Test Automation Framework at STPCon 2017. Here are the details from the talk.

Abstract

Building a Test Automation Framework is easy – there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.

However, building a “good” Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.

Design Patterns play a big role in helping achieve this goal of building a good and robust framework.

In this talk, we will talk about, and see examples of various types of patterns you can use for:

• Build your Test Automation Framework
• Test Data Management

Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

Session Takeaways:

• Patterns for building Test Automation Framework.
• Patterns for Test Data Management, with pros and cons of each.

Slides

Patterns of a “good” test automation framework from Anand Bagmar

Pictures

Workshop - Client-Side Performance Testing at STPCon 2017

I conducted a 4-hour workshop on Client-Side Performance Testing at STPCon 2017 on 15th March 2017.

Workshop Abstract

In this workshop, we will see the different dimensions of Performance Testing and Performance Engineering, and focus on Client-side Performance Testing.

Before we get to doing some Client-side Performance Testing activities, we will first understand how to look at client-side performance, and putting that in the context of the product under test. We will see, using a case study, the impact of caching on performance, the good & the bad! We will then experiment with some tools like WebPageTest and Page Speed to understand how to measure client-side performance.

Lastly – just understanding the performance of the product is not sufficient. We will look at how to automate the testing for this activity – using WebPageTest (private instance setup), and experiment with yslow – as a low-cost, programmatic alternative to WebPageTest.

We will also look at the different dimensions of Client-side Performance Testing for native mobile applications.

PS: This workshop will be a combination of presentation and hands-on-activity with lots of discussion throughout. You should bring your laptop with you.

Workshop Takeaways:

• Understand difference between is Performance Testing and Performance Engineering.
• Hand’s on experience of some open-source tools to monitor, measure and automate Client-side Performance Testing.
• Examples / code walk-through of some ways to automate Client-side Performance Testing.

Slides

Client-Side Performance Testing from Anand Bagmar

Some pictures from the workshop 

(Thanks Mike Lyles, Curtis Stuehrenberg for the pictures)

Analytics Testing

I recently spoke in an Agile Testing Conference on - The What, Why and How of (Web) Analytics Testing.

I was also part of a panel discussion having the theme - "What's not changed since moving to Agile Testing - The Legacy Continues!" There were some very interesting perspectives in this discussion.

The great part was that the audience was very involved and vocal throughout the day. This made is very interactive and good sharing of information and experiences for all!

Below is some information about the talk. I will try to add the link to the video soon.
 

Abstract

Analytics is changing the way products and services are being created and consumed.

In this session, we will learn

• What is Analytics?
• Why is it important to use Analytics in your product?
• The impact of Analytics not working as expected

We will also see some techniques to test Analytics manually and also automate that validation. But just knowing about Analytics is not sufficient for business now.

There are new kids in town - IoT and Big Data - two of the most used and heard-off buzz words in the Software Industry!

With IoT, with a creative mindset looking for opportunities and ways to add value, the possibilities are infinite. With each such opportunity, there is a huge volume of data being generated - which if analyzed and used correctly, can feed into creating more opportunities and increased value propositions.

There are 2 types of analysis that one needs to think about.

1. How is the end-user interacting with the product? This will give some level of understanding into how to re-position and focus on the true value add features for the product.

2. With the huge volume of data being generated by the end-user interactions, and the data being captured by all devices in the food-chain of the offering, it is important to identify patterns from what has happened, and find out new product / value opportunities based on usage patterns.

Slides

The What, Why and How of (Web) Analytics Testing (Web, IoT, Big Data) from Anand Bagmar

Pictures

Sharing implementation of cucumber-jvm - Appium test framework

I recently shared the Features of my Android Test Automation Framework and also the challenges, and, how we overcame those, to make the parallel test execution work well with Android 7.0 devices as well.

In this blog post, I will be sharing the details (including code) of the implementation. If you have not read my post on - Features of my Android Test Automation Framework - I highly recommend you read that first.

Implementation Details

Tech Stack Summary

To recap - here is the tech stack that we currently have:

• cucumber-jvm - v1.2.5
• cucumber-reporting - v3.5.1
• appium - v1.6.3
• appium-java-client - v5.0.0-BETA2
• appium-uiautomator2-driver - v0.2.3

1. Configure Jenkins Node (in Jenkins Server)

We currently have 5 Jenkins Nodes setup as shown below.

Each node is configured like this:

2. Setup Jenkins Job (in Jenkins Server)

Once the Nodes are setup, we can now configure the Jenkins Jobs. 

We have setup the following jobs in Jenkins for our test executions.

Each job is configured as a Jenkins Pipeline Project and we use the the Jenkins file available here (https://github.com/anandbagmar/cucumber-jvm-appium-infra/blob/master/2%20-%20Setup%20Jenkins%20Jobs/2.2%20Jenkinsfile) as a sample from git to configure what the job is supposed to do.

3. Setup Jenkins Agent

Once the Jenkins Nodes and Jenkins Jobs are configured, we now need to get the Jenkins Agents itself setup and configured to be able to service the requests from the Jenkins server.

We use the JNLP way to connect the Jenkins Slave to the Jenkins server. For this, we have a template .sh script, which we need to copy and update 2 values in it. This is needed for each new Jenkins Node that we connect.

The template .sh script can be found here (https://github.com/anandbagmar/cucumber-jvm-appium-infra/blob/master/3%20-%20Setup%20Jenkins%20Agent/3.1%20start-e2e-moto.sh).

Now, our Jenkins setup is done. But a big piece is still missing. 

In order to run our tests on the Agent, we need some basic software to be installed. To do this, we created a shell script, that will help provision the machine. This is required to be done just once - but we do plan to have multiple Mac Mini host machines that will run various number of Jenkins agents - so the script will help keep same software (including version) on our machines - which means the same test execution environment.

This shell script can be found here - (https://github.com/anandbagmar/cucumber-jvm-appium-infra/blob/master/3%20-%20Setup%20Jenkins%20Agent/3.2%20JenkinsAgentMachineFirstTimeSetup.sh)

4. Manage Test Infrastructure & Test Execution

By this stage, our Jenkins Server, Jenkins Agent setup is done, including the software required to run the tests. Next thing is now at the Test Framework level.

Our build tool is gradle. All infrastructure related work is handled via this build.gradle file. 

Before we get into the details of the gradle file, it is important to understand what the code structure is.

Via groovy / gradle, we managed to solve the complexity of:

• Finding matching devices based on the CONNECTED_DEVICE_IDS
• Downloading the apk file from where ever it is available
• This is done just once per test run - regardless of how many devices the test is going to run on
• The URL to download is passed as an environment variable - APP_URL
• For local testing, you can give a local absolute path to the apk file via the APP_PATH environment variable instead of specifying APP_URL
• Finding the list of scenarios to be run (based on the cucumber tags specified via the environment variable - 'run'
• Managing start / stop of Appium Servers
• Cleaning up the device before test runs
• Executing Cucumber scenarios in parallel
• Building consolidated reports locally (cucumber-reports) - IF not using the Jenkins cucumber-reports plugin

The gradle file with all this configuration is available here (https://github.com/anandbagmar/cucumber-jvm-appium-infra/blob/master/4%20-%20Manage%20Test%20Infrastructure%20and%20Test%20Execution/4.2%20build.gradle).

5. Run Tests

Last, step in this process - is to manage the Android Driver. We use the Cucumber-jvm's @Before and @After hooks to set the right capabilities for instantiating the AndroidDriver, and also stopping the same after test execution is complete.

These helper files can be found here (https://github.com/anandbagmar/cucumber-jvm-appium-infra/tree/master/5%20-%20Run%20Tests).

Sample Code

All the sample code can be found from my github repository cucumber-jvm-appium-infra - https://github.com/anandbagmar/cucumber-jvm-appium-infra

Happy Testing!