Wednesday, December 9, 2020

Getting started with implementing Automation

Getting started with implementing tests for automation (web or native apps) may seem daunting for those who are doing this for the first time. 

Assuming you are using open-source tooling like Selenium or Appium, there are multiple ways you can get started.

  1. DIY - Build your own framework by scripting based on the documentation

  2. Use Selenium-IDE for quick record and playback

  3. Use TestProject Recorder for quick record and playback

  4. Use TestProject SDK to build your own custom scripts for automating the tests

Each of the above approaches has its own pros-and-cons. Let's look at this in some detail:

Approach #1 - DIY - Build your own framework





You can build all features and capabilities as per your design & requirement

*You need to learn a programming language

*You have to build everyone on your own (though you can use supporting libraries)

* Depending on the context of the team, the above points can also be considered as an advantage

Approach #2 - Selenium-IDE



Easy to set up

Basic reports

Works in Chrome & Firefox

Works only for automating Web applications

Code can be exported in various formats

Recorded tests can be run from command line

Tests can be run in your own CI

Will always be in-sync with underlying WebDriver

Approach #3 - TestProject Recorder



Advanced recorder (lot of actions, validations, self-healing, customisations possible, and a lot of community Addons)

Recorder works only in Chrome, but tests can be executed on all browsers

Recorder works for Web applications as well as Native Apps (in real devices or emulators) for Android and iOS (even iOS on Windows machine)

Generated code is very simple - good as a reference to see how the underlying implementation / interaction is done

TestProject agent automatically determines all available browsers available and devices connected to the machine and execution can be customised accordingly

Each recorded test needs to be exported individually. No concept of reuse in this approach

Can schedule test runs as one-time, or repeated activity via build-in scheduler / CI/CD tool integrations or via their RESTful API

Reports are comprehensive with meaningful data, including screenshots and option to download to PDF format

Code can be generated from the recorded script

Can share tests easily using the "Share test" feature

Approach #4 - TestProject SDK



Probably the most powerful way of these 4 approaches as it uses WebDriver / Appium under the hood. Get the power of building your own framework, while reusing out-of-the-box features like driver management, automatic reporting, etc.

You need to learn a programming language

Driver management is TestProject responsibility. Test implementer can focus on automating tests

Sunday, December 6, 2020

Long time no see? Where have I been

It has been a few months since I published anything on my blog. Does not mean I have not been learning or experimenting with new ideas. In fact, in the past few months I have been privileged to have my articles published on Applitools and Test Project blogs.

Below is the link to all those articles, for which I have received very kind reviews and comments on LinkedIn and Twitter.

Apart from this, I have also been contributing to open source - namely - Selenium, AppiumTestDistribution and building an open-source kickstarter project for API testing using Karate and for end-2-end testing for Android, iOS, Windows, Mac & Web.

Lastly, I have also been speaking in virtual conferences, webinars and last week I also recorded a podcast, which will be available soon.

The end of Smoke, Sanity and Regression

Do we need Smoke, Sanity, Regression suites?
  • Do not blindly start with classifying your tests in different categories. Challenge yourself to do better!
  • Have a Test Automation strategy and know your test automation framework objective & criteria (“Test Automation in the World of AI & ML” highlights various criteria to be considered to build a good Test Automation Framework)
  • Choose the toolset wisely
  • After all the correct (subjective) approaches taken, if your test execution (in a single browser) is still taking more than say, 10 min for execution, then you can run your tests in parallel, and subsequently, split the test suite into smaller suites which can give you progressive indication of quality
  • Applitools with its AI-power algorithms can make your functional tests lean, simple, robust and includes UI / UX validation
  • Applitools Ultrafast Grid will remove the need for Cross-Browser testing, and instead with a single test execution run, validate functionality & UI / Visual rendering for all supported Browsers & Viewports

Design Patterns in Test Automation

Criteria for building a Test Automation Framework

Writing code is easy, but writing good code is not as easy. Here are the reasons why I say this:
  • “Good” is subjective.
  • “Good” depends on the context & overall objective.

Similarly, implementing automated test cases is easy (as seen from the getting started example shared earlier). However, scaling this up to be able to implement and run a huge number of tests quickly and efficiently, against an evolving product is not easy!

I refer to a few principles when building a Test Automation Framework. They are:
  • Based on the context & (current + upcoming) functionality of your product-under-test, define the overall objective of Automation Testing.
  • Based on the objective defined above, determine the criteria and requirements from your Test Automation Framework. Refer to my post on “Test Automation in the World of AI & ML” for details on various aspects you need to consider to build a robust Test Automation Framework. Also, you might find these articles interesting to learn how to select the best tool for your requirements:
    • Criteria for Selecting the Right Functional Testing Tools
    • How to Select the Best Tool – Research Process
    • How To Select The Right Test Automation Tool

Stop the Retries in Tests & Reruns of Failing Tests

  • Recognise reasons why tests could be flaky / intermittent
  • Critique band-aid approach to fixing flakiness in tests
  • Discuss techniques to identify reasons for test flakiness
  • Fix the root-cause, not the symptoms to make your tests stable, robust and scalable!

Measuring Code Coverage from API Workflow & Functional UI Tests

Why is the Functional Coverage important?
I choose an approach keeping the 80-20 rule in mind. The information the report provides should be sufficient to understand the current state, and take decisions on “what’s next”. For areas that need additional clarity, I can then talk with the team, explore the code to get to the next level of details. This makes it a very collaborative way of working, and joint-ownership of quality! 🚀

You can choose your own way to implement Functional Coverage – based on your context of team, skills, capability, tech-stack, etc.