Finding my way out of bottomless pit with Appium & Android 7.0 for parallel test runs

As mentioned in my earlier post - I designed and implemented a cucumber-jvm-Appium-based test framework to run automated tests against Android Mobile Devices.

We were using:

cucumber-jvm - v1.2.5

cucumber-reporting - v3.3.0

appium - v1.6.3

appium-java-client - v4.1.2

All was good, tests were running via CI, in parallel (based on scenarios) against devices having Android v5.x and v6.x.

Then the challenges started. We got some new Motorola G4 Plus devices for our Test Lab - which has Android 7.0 installed.

First the test refused to run. Figured out that we would probably need to upgrade the appium java-client library version to v5.0.0-BETA1. By the time we figured that out, appium-java-client v5.0.0-BETA2 was out. We also needed to change the instrumentation to UiAutomator2. This was all fine. Our tests started working (after some more changes in how locators were defined and used).

However, the tests refused to run in parallel on the Motorola devices with Android 7. The app used to launch correctly, but tests used to run as expected only on 1 of the devices - causing our test job to fail miserably, and without any clue.

These same tests continued to work correctly with all other devices having Android 5.x and 6.x. Very confusing indeed, not to mention highly frustrating too!

By this time, appium-java-client v5.0.0-BETA3 was out, but refused to upgrade to that - as the difference was iOS specific. Likewise, Appium v1.6.4 BETA is now available - but not feeling to upgrade so fast - and battle the new surprises, if any.

After digging through Appium's open issues, figured out that many people have faced, and got the similar issue resolved. The solution seemed to be to upgrade the appium-uiautomator2-driver to version > v0.2.6.

So - next question, which had an easier answer - how to upgrade this uiautomator2-driver. However - after the upgrade, my issue did not get fixed. In fact, now the Android Driver was unable to get instantiated at all. I was getting the errors shown below.

1. [MJSONWP] Encountered internal error running command: Error: Command '/Users/IT000559/Library/Android-SDK/build-tools/25.0.2/aapt dump badging /usr/local/lib/node_modules/appium/node_modules/appium-uiautomator2-driver/uiautomator2/appium-uiautomator2-server-v0.1.1.apk' exited with code 1 at ChildProcess. (../../lib/teen_process.js:70:19) at emitTwo (events.js:106:13) at ChildProcess.emit (events.js:192:7) at maybeClose (internal/child_process.js:890:16) at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)

2. org.openqa.selenium.SessionNotCreatedException: Unable to create new remote session. desired capabilities = Capabilities [{appPackage=com.vuclip.viu, noReset=false, appWaitActivity=com.vuclip.viu.ui.screens.IndianProgrammingPreferenceActivity, deviceName=motorola, fullReset=false, appWaitDuration=60000, appActivity=com.vuclip.viu.ui.screens.MainActivity, newCommandTimeout=600, platformVersion=7.0, automationName=UIAutomator2, platformName=Android, udid=ZY223V2H8R, systemPort=6658}], required capabilities = Capabilities [{}] Build info: version: '3.0.1', revision: '1969d75', time: '2016-10-18 09:49:13 -0700'

Eventually, found a workaround. I had to make the following 2 changes:

• When initialising the Android Driver, I had to pass an additional capability - "systemport" and set the value to the Appium port for the Appium server the test was connecting to.
• capabilities.setCapability("systemPort", Integer.parseInt(APPIUM_PORT)); 

• Before the test run started, I do a cleanup - which includes 
• kill any prior / orphan Appium server for that particular port, if remaining 
• Uninstall the app from the device. I had to add another step to also uninstall the following: 
• io.appium.uiautomator2.server, and, 
• io.appium.uiautomator2.server.test 

Post this, my tests are now working, as expected (from the beginning), sequentially or in parallel, against all supported Android versions.

Current stack::

cucumber-jvm - v1.2.5

cucumber-reporting - v3.5.1

appium - v1.6.3

appium-java-client - v5.0.0-BETA2

appium-uiautomator2-driver - v0.2.3

After the dust settled, my colleague - Priyank Shah and I were thinking why not many people have encountered this problem.

My thought is probably most people may be managing the Appium Server and the Android Driver from the test run, instead of from a build script. As a result, they would not have encountered the systemport related challenge as we did.

PS: Note that Appium Server is started / stopped via our build.gradle file and the AndroidDriver is instantiated (based on parameters passed via a combination of environment variables & properties file) from within each cucumber-jvm scenario (@Before hook).

Hope our learning helps others who may encounter similar issues.

Features of my Android Test Automation Framework

[UPDATED - Added link to implementation details at the end of the post]

As I have shared in my previous few blog posts (A new beginning - entertainment on mobile, How to enable seamless running of appium tests on developer machines?), a few months ago, I embarked on a new journey as "Director - Quality" for the Viu product at Vuclip.

Here are a few details about our Viu app:

Viu offers high quality, popular, regional video content in various different languages for consumers in various different regions - Indonesia, Malaysia, India, Middle-East, Egypt ....

The consumers on the move could be using Android devices or iOS devices to watch their favourite content - either stream it directly via their Mobile Data plan, or they could have downloaded their preferred video at home / office and watch the saved video later.

The very interesting, and, challenging bit from Testing perspective, is that the app would behave differently based on from which part of the world the user is using it. This logic is not based on Geolocation.

Objective

My objective from functional test automation is:

• Get quick feedback, from every build and know if the app is working well. If NOT, then be able to get to the root cause ASAP.
• Provide feedback and confidence to stakeholders (Product team, Business team, etc.) on the "true" state of the product. There should be no surprises, for anyone!
• Run tests seamlessly by simulating different regions

Approach

To achieve this - I choose to go with the following tech-stack:

• Test Intent specification - cucumber-jvm - to specify the business intent, in the business terminology (expected to be) understood by all.
• Reports - cucumber-reports provide good, rich, meaningful and easily understandable reports which provide meaningful information about the state of the product.
• Device interaction - Appium / Java - implement the intent specified by cucumber-jvm. This will also allow us the flexibility to test against Android, iOS and Web.
• Continuous Integration (CI) server - Jenkins - to be able to run tests automatically when a new build is ready. Also, we integrated the cucumber-reports via the cucumber-reports plugin directly in Jenkins - so now all stakeholders interested in the reports just need to go to one place and get all information, in real time. No more need for Test Reports to be generated manually and sent across for everyone.

Test Execution environment

I looked at various cloud-based service providers (SauceLabs, Test Object, pCloudy, AWS Device Farm, etc.) who could run our tests on real-devices in their data centers. These tests would need to be triggered automatically via Jenkins when a new build (apk) is available, or as we add more tests. However, none of these providers could satisfy my requirements (without having to compromise significantly on my objectives).

Hence, finally decided to setup my own private Test Lab for this. Oh fun!

Framework Architecture

Below is the architecture of the framework that I came up with. This is based on the architecture I posted in my earlier blog post on “Assertion & Validations in Page Object” (https://essenceoftesting.blogspot.com/2012/01/assertions-and-validations-in-page.html)

Approach, Features & Capabilities of the Test Lab

• For Jenkins configuration, we configure the job using Jenkins file - to ensure our Jenkins configuration is also under version control. Also, since this is groovy scripting language, we can have good logic and processing done as part of the job execution.
• This helped us keep the configuration consistent and common across any type of job we had to run.
• The Jenkins job will trigger a set of commands on the Jenkins Agent.
• Jenkins Agents are setup on Mac Mini machines. Each Agent has only 1 executor. This is essential to prevent using a device that is already in use by another job / executor / agent.
• The Mac Mini (more to be added on demand) has a powered USB hub which connects upto 8 devices
• Depending on types of devices connected, we have as many Jenkins Agents (with 1 executor only) allocated for them.

Example:

• Samsung devices (with API Level 22) allocated to Jenkins agent - viu-e2e-samsung
• Motorola devices allocated to Jenkins agent - viu-e2e-moto

Infrastructure

• To manage and use the devices allocated to each node and more important - prevent other nodes using other devices, each node in Jenkins has an environment variable in its configuration - CONNECTED_DEVICE_IDS - a comma separated list of devices allocated to this node.
• This approach makes it easy to change / add / remove devices on the fly. All we need to do in such cases is update the device ID in the appropriate node’s CONNECTED_DEVICE_IDS environment variable
• Our gradle file, reads the CONNECTED_DEVICE_IDS environment variable, and finds devices connected on the Mac Mini matching the provided device ids. This simple technique allows specific device allocation from the pool of devices to each node.
• PS: If anyone does an error and provides the same device id for multiple nodes, we all know what will happen. These are areas where we need to be very cautious in our execution and maintenance.
• The URL for the Android APK file is also passed to the gradle file as a an environment variable. We download the APK file once before test execution starts.
• Functional (end-2-end) Automation is painfully slow to run. To get the feedback quickly, we have to run this in parallel. All existing approaches to running cucumber-jvm tests in parallel failed to meet our requirement. I wanted to run each scenario in parallel, on whichever device is free (from the matching devices list). Eventually ended up writing a small script that allows me to run scenarios in parallel.

• Managing Appium Server (start / stop) is another important activity - which is required to be done just once per device. Gradle manages that as well.

Next steps

• Stabilize parallel test execution (problems with Android 7)
• Start with iOS app automation
• Start with Web automation (www.viu.com)

• Share gradle file with community, if anyone interested.

• [UPDATED] You can find the details of the implementation here

A new beginning - entertainment on mobile

After 7+ years, I finally took the heavy step and moved out of ThoughtWorks.

The past 7+ years have been awesome. I had loads of fun, learnt many new things, made a lot of friends and found inspiration and guidance from a lot of mentors.

Thank you ThoughtWorks and ThoughtWorkers! Wouldn’t have been who I am today without you and you all will always be a huge part of me!

Taking the decision was tougher than I thought it would be ... but new challenges were waiting for me, and the time had come.

On 1st December, 2016, I started my next stint as "Directory - Quality" at Vuclip, Inc for the Viu product. You can also find us via the PlayStore or AppStore.

Day 1 at vuclip, barring the first 2 hours of paperwork, was getting right into action. With the planning for 2017 in full swing, there was no time to settle - but instead had to hit the ground running.

The charter starting Day 1 for me was:

• Define & execute test strategy for Viu - for multiple platforms, for multiple regions & partners
• Build team to help execute the above (see section below on what I am looking out for)
• In scope - functional testing, automation, performance, analytics, benchmarking, infrastructure, tooling, etc.
• Out of scope - nothing

And so the fun has begun.

So, here is what I need to learn and execute immediately (looking forward to suggestions, links, feedback on how you have done it in the past)

• Is it worth setting up a mobile lab (real devices + simulators) in-house or use external services for running automated tests  exploratory tests? 
• If the latter to the above, what services have you used in the past? What have the results been?
• Is it possible (and worth) automating the checks for memory / processor / battery usage when running tests against the native app (on Android & iOS)?
• How to do native app performance testing (client-side) for Android & iOS?

Also, I am looking to build a strong testing team with team members having the primary skills & capabilities -

• Open-minded, quick learner
• A good Testing-mindset
• Mobile Testing experience (non-automated + test automation)
• Performance Testing (client-side & server-side)

Contact me if you are interested in being part of my team to work on this challenging product.