Patterns in Test Automation Framework at STPCon

I spoke about Patterns of a "good" Test Automation Framework at STPCon 2017. Here are the details from the talk.

Abstract

Building a Test Automation Framework is easy – there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.

However, building a “good” Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.

Design Patterns play a big role in helping achieve this goal of building a good and robust framework.

In this talk, we will talk about, and see examples of various types of patterns you can use for:

• Build your Test Automation Framework
• Test Data Management

Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

Session Takeaways:

• Patterns for building Test Automation Framework.
• Patterns for Test Data Management, with pros and cons of each.

Slides

Patterns of a “good” test automation framework from Anand Bagmar

Pictures

WAAT out of hibernation ...

I am resuming work on WAAT. If interested in contributing, look at https://waffle.io/anandbagmar/WAAT or https://github.com/anandbagmar/waat/issues

Looking forward to pull requests :)

Shared (relatively less) pain of using Protractor in SeConf London 2016

On 15th Nov 2016, I spoke in Selenium Conference London 2016 on the topic - "Sharing the 'pain' of using Protractor & WebDriver". This time, the pain was less - as I had already shared this in Selenium Conference India 2016. From that experience, and lot of interactions, I worked on some of the challenges I was facing and implemented good solutions for the same.

So this time it was easier.

The interesting challenge though was when the A/V stopped working after the 1st slide - and had to do on-the-spot improv to keep the ball rolling and prevent delays in subsequent talks. To know more about the fun I had, watch the video linked below :)

You will find the abstract, slides and video of the talk:

Abstract

There is a saying ..."Sukh baatne se badhta hai, dukh baatne se kam hota hai", translated as - "happiness increases & sadness reduces on sharing with others".

We want to take this opportunity to share with our experiences - the good and the bad, in the journey of building a Test Automation framework for an AngularJS based application.

We will learn, by a case study, what thought process we applied on the given context (product, team, skills, capabilities, long term vision) to come up with an appropriate Test Automation Strategy. This Test Automation strategy covered all aspects of Test Automation - Unit, Integration, UI - i.e. End-2-End tests (E2E).

Next, we will share how we went about narrowing-down, and eventually selecting a specific Tech Stack + Tools (Javascript / Jasmine / Protractor / Selenium-WebDriver) to accomplish the Test Automation for the product.
 
Lastly, we will share the challenges that came up in the implementation of the Test Automation, and how we overcame them. This will also include how we managed to get the tests running in Jenkins - a Continuous Integration tool.

This discussion is applicable to all team members who are working on Test Automation!

P.S. We will be attempting to make a sample protractor-based automation framework available on github for anyone to use as a starting point for setting up protractor-based Test Automation framework.

Slides

Sharing (less) Pain of using Protractor & WebDriver from Anand Bagmar

Video

https://youtu.be/YpEC3rFkH5

 

Sharing the pain in Automating with Protractor & WebDriver

As mentioned in my earlier post titled "Taking Protractor to the next level", on 25th June 2016, Nikitha Iyer and I spoke in Selenium Conference 2016, Bangalore, India on "Sharing the pain in Automating with Protractor & WebDriver".

The video is finally available and I am now able to share (see below) all the details of the talk from one post.

Abstract

There is a saying ..."Sukh baatne se badhta hai, dukh baatne se kam hota hai", translated as - "happiness increases & sadness reduces on sharing with others".

We want to take this opportunity to share with our experiences - the good and the bad, in the journey of building a Test Automation framework for an AngularJS based application. 

We will learn, by a case study, what thought process we applied on the given context (product, team, skills, capabilities, long term vision) to come up with an appropriate Test Automation Strategy. This Test Automation strategy covered all aspects of Test Automation - Unit, Integration, UI - i.e. End-2-End tests (E2E).

Next, we will share how we went about narrowing-down, and eventually selecting a specific Tech Stack + Tools (Javascript / Jasmine / Protractor / Selenium-WebDriver) to accomplish the Test Automation for the product.

Lastly, we will share the challenges that came up in the implementation of the Test Automation, and how we overcame them. This will also include how we managed to get the tests running in Jenkins - a Continuous Integration tool. 

This discussion is applicable to all team members who are working on Test Automation!

P.S. We will be attempting to make a sample protractor-based automation framework available on github for anyone to use as a starting point for setting up protractor-based Test Automation framework.

Slides

Sharing the pain using Protractor from Anand Bagmar

Video

Any browsermob-proxy users facing issues with some requests not getting fired?

Is there anyone using browsermob-proxy who is having issues with some requests not getting fired?

I have integrated browsermob-proxy with my protractor tests. This works wonderfully when I run my tests from Mac (against local environment, or any other test environment).

However, when I run my tests from CI (agent is SUSE Enterprise 11.4) - my tests fail. 

I narrowed down the problem to the following scenario:

On some specific user action in the UI, there are a lot (>100) of requests fired from the browser in parallel (batches). There are a couple of scenarios like this in my application - and the test fails in all these cases.

Here is a screenshot of what the captured HAR file shows -

The same test works when I run this locally from Mac

Any idea how to fix this? Thank you in advance!

See this issue for more details - (https://github.com/lightbody/browsermob-proxy/issues/492)

Any WAAT (Web Analytics Automation Testing Framework) users out there?

It has been over 2 years since any update to WAAT - Java or Ruby. Over the years, I have realised, and also received a lot of thoughts / feedback from users of WAAT around where it helps, and what challenges exist. 

Also, given the widespread IoT & Big Data based work going on around the world, (Web) Analytics now plays a much bigger role in guiding business take better decisions. 

WAAT (again) fits in the grand scheme of things very nicely as a framework to automate the validation of correct reporting of tags to any Web Analytics solution provider.

Hence, its a no-brainer for me - it is high time I work on some of the feedback and limitations of WAAT to make it usable again!

At the recently concluded Selenium Conference 2016 held in Bangalore, India, I got an idea of how to overcome a lot of challenges (listed below) and pain in using WAAT. 

What's next?

To implement my new idea, this does mean a couple of things:

• Existing plugins have limited use - and needs to be deleted.
• A new plugin would need to be created - which may mean different set of APIs, and also different way to specify the test data.

Questions for you

Before I go ahead making these changes - I would like to get answers to the below questions (please add your answers directly in the comments):

• Is anyone currently using WAAT? If yes - 
• which version (Java / Ruby)?
• which plugin
• Using HTTP / HTTPS?
• Which Web Analytic solution are you using? (ex: Google Analytics, WebTrends, etc?)
• Would you be interested in using the new WAAT? If yes - 
• Which language? Java / Ruby / JavaScript / Python / etc?
• Would you like to contribute to implementing this new WAAT? If yes - contact me! :)

-----------------------------------------------------------------------------------------------------------------

Current plugins available in WAAT:

• Omniture Debugger (WAAT-Java)
• Pros:
• OS independent
• Run using regular-test-user 
• Cons:
• Browser dependent - need to implement ScriptRunner for the UI-driver in use
• Web-Analytic solution dependent - only for Adobe Marketing Clout / Omniture SiteCatalyst
• HTTPSniffer (WAAT-Java, WAAT-Ruby)
• Pros
• Web-Analytic solution independent
• Browser independent
• UI-Driver independent
• Cons
• 3rd party libraries are OS dependent
• HTTPs is not supported out-of-the-box
• Run tests as "root"
• JSSniffer (WAAT-Java, WAAT-Ruby)
• Pros
• Web-Analytic solution independent
• Browser independent
• HTTPs supported out-of-the-box
• No 3rd party library dependency
• Cons
• Need to write JavaScript to get the URL from the browser context
• UI-Driver dependent

-----------------------------------------------------------------------------------------------------------------

Learnings from Selenium Conference 2016, Bangalore

The value one gets by attending any conference / training / meetup / etc. is subjective to various aspects, some of which are mentioned below (in no particular order):

• Individual skills & capabilities
• Past experiences
• Existing knowledge / information / expertise on the subject 
• Open mindedness
• Willingness to learn
• Current work (tools & tech stack, challenges, risks, priorities, backlog, tech debt, team members, etc.)

The above aspects definitely played a part in what takeaways I had from the recently concluded Selenium Conference 2016 in Bangalore as well.

Here are my key takeaways, which I am going to work on learning more about, or implementing in the near future - special thanks to +Dave Haeffner , +Marcus Merrell , +Simon Stewart, +Bret Pettichord for helping me find these takeaways as part of various conversations during these few days.

• Related to Protractor
• Use Proxy Server in tests (Protractor framework) to capture HAR file on specific actions (AJAX calls) - and capture performance metrics from the same
• Read and experiment with the Marionette driver for Firefox - maybe it helps me overcome some of my challenges with Firefox & Maps in CI environment (headless using xvfb)
• Remove "phantomJS" as a supported browser from my framework by ensuring headless tests work with Chrome & Firefox using xvfb
• Highlight element when running tests before taking screenshots - will help in debugging
• Experiment with different loggers & reporters - Allure, Winston logger
• Better "promise" handling in framework to keep abstraction layers sane
• Revive WAAT - Web Analytics Automation Testing Framework - create new plugin using Proxy Server approach. Also remove Omniture Debugger and HttpSniffer plugin.
• Refocus energy on TTA - Test Trend Analyzer.
• Keep vodQA going strong - its a good community initiative

See you all in Selenium Conference UK in November 2016!

Taking Protractor to the next level

As I got onto a project early this year - my paths crossed again with Protractor / Angular / JavaScript. Despite my past experiences with this tech stack, I needed to remain positive when approaching this challenge.

In this case, the context was quite different. That said, the challenges, though reduced compared to my first attempt at using Protractor effectively, were still around. 

I had posted a set of questions on my blog, various LinkedIn and facebook groups - and got a lot of suggestions and advice how to proceed. Thanks to all of them, it did make my life easier.

Then, as part of Selenium Conference 2016 held in Bangalore on 24-25th June 2016, my friend and colleague - +Nikita Iyer and I spoke about "Sharing the Pain of Automating with Protractor & WebDriver". The intention was multi-fold:

• We wanted to share what challenges we had faced on the team when using Protractor, and also how we overcame certain challenges. This was to help others in similar situations to learn from our mistakes & solutions.
• We also wanted to learn from the attendees what other challenges & solutions they had come up with in their experiences with this Protractor tool-stack.

I am very happy to say that we were successful in achieving both these objectives quite successfully.

Below is the details from the talk.

Abstract

There is a saying ..."Sukh baatne se badhta hai, dukh baatne se kam hota hai", translated as - "happiness increases & sadness reduces on sharing with others".

We want to take this opportunity to share with our experiences - the good and the bad, in the journey of building a Test Automation framework for an AngularJS based application. 

We will learn, by a case study, what thought process we applied on the given context (product, team, skills, capabilities, long term vision) to come up with an appropriate Test Automation Strategy. This Test Automation strategy covered all aspects of Test Automation - Unit, Integration, UI - i.e. End-2-End tests (E2E).

Next, we will share how we went about narrowing-down, and eventually selecting a specific Tech Stack + Tools (Javascript / Jasmine / Protractor / Selenium-WebDriver) to accomplish the Test Automation for the product.

Lastly, we will share the challenges that came up in the implementation of the Test Automation, and how we overcame them. This will also include how we managed to get the tests running in Jenkins - a Continuous Integration tool. 

This discussion is applicable to all team members who are working on Test Automation!

P.S. We will be attempting to make a sample protractor-based automation framework available on github for anyone to use as a starting point for setting up protractor-based Test Automation framework.

Slides

Sharing the pain using Protractor from Anand Bagmar

Video

- will be updated once available

Pictures

Selenium Conference 2015 - it simply came, and went so fast

Its been a crazy summer - the 2nd week of September 2015 just amplified that…

A good few months ago we - the Selenium Conference Planning Committee started on the journey of planning this years Selenium Conference 2015. We started with debating where to have this years conference, till Portland magically came up on the radar, and became a reality. We met over Google Hangout every 2 weeks initially, and then as we got closer to the date, every week.

Can’t believe as I am writing this post, the conference is already over (a couple of weeks ago) …

The team put in a lot of hard work - me doing the least of that … and the turnout (approx 500 people), the interactions and the quality of talks proves the hard work paid dividends.

I traveled from Pune, India on 5th Sept at around 6pm headed to Portland, Oregon. The journey - from home to the hotel took approximately 35 hours.

After crazy 4 days, and a total of around 25-30 hours of sleep in 5 nights (thanks to the jet lag), and having delivered 3 talks as well, it was another 35 hour trip back home ... the only good thing after this hectic trip - I never got adjusted to the US time zone - which meant no jet-lag when I came back home :) This was a first for me :)

Slides & Videos from Selenium Conference 2015:

All the slides and videos for all the talks are available here.

Below is the list of my talks:

To Deploy or Not-to-Deploy - decide using TTA's Trend & Failure Analysis

I got a lot of very good feedback for this talk, and also quite a few people expressed interest in trying it out! Looking for feedback from their experiences!


Video of the talk is available on YouTube here:

Slides are available here:

To Deploy or Not-To-Deploy - decide using TTA's Trend & Failure Analysis from Anand Bagmar

Automate across Platform, OS, Technologies with TaaS

This topic is so relevant with anyone working in large enterprises, or when it is being "mandated" to work on a common test automation framework.

Video of the talk is available here:

Slides are available here:

Integration Testing in Enterprises using TaaS from Anand Bagmar

Say ‘No’ to (more) Selenium Tests

I paired with Bhumika on this talk. We were very agile in preparing for this talk - a day in advance to be precise. Also, it was very bold topic to have in a Selenium Conference - standing in front of 200+ Selenium enthusiasts, and telling them - do NOT write more Selenium tests. But went pretty well ... given that we were able to walk on our own feet out of the room, and that people were able to get the message we were trying to deliver :D

Video of the talk is available here:

Slides are available here:

Say NO To (More) Selenium Tests from Anand Bagmar

Patterns in Test Automation

I spoke in vodQA Hyderabad on Sat, 22nd August 2015 about Patterns in Test Automation - Frameworks, Data & Locators.

The slides are available on SlideShare:

Patterns in Test Automation from Anand Bagmar

The video is available on YouTube:

Abstract

Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey.

However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable.

Design Patterns play a big role in helping achieve this goal of building a good and robust framework.

In this talk, we will talk about, and see examples of various types of patterns you can use for:

• Build your Test Automation Framework
• Test Data Management
• Locators / IDs (for finding / interacting with elements in the browser / app)

Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

Learning outcome

• Patterns for building Test Automation Framework
• Patterns for Test Data Management, with pros and cons of each
• Patterns for managing locators / IDs for interaction with UI

Role of Automation in Testing

I am speaking in Discuss Agile 2015 conference on 13-14 June 2015 on the following topics - 

1. To Deploy or Not-to-Deploy - decide using TTA's Trend & Failure Analysis
2. Enabling Continuous Delivery (CD) in Enterprises with Testing

As part of this conference, I also did an interview with Saket Bansal and Atulya Mishra on - The Role of Automation in Testing.

This was an interesting, virtual interview - where interested people had asked questions during registration, and also a lot of questions came up during the interview.

Below is the video recording of the interview. 

I also referenced some slides when speaking about some specific topics. Those can be seen below, or directly from slideshare.

vodQA Cocktail - early in 2015

As we get ready for Celebrating Selenium's 10 year journey in vodQA Hyderabad, ThoughtWorks Chennai is ready to take vodQA to the next level on Saturday, 21st February, 2015 with an interesting Cocktail of topics related to Software Testing.

Register here as a speaker for vodQA Chennai, or here as an attendee.

Start 2015 by Celebrating Selenium in vodQA Hyderabad

ThoughtWorks, Hyderabad is proud to host its first vodQA, also the first vodQA of 2015 and start 10 Years of Selenium Celebration. This event will be held on Saturday, 31st Jan 2015.

Look at the agenda of this vodQA and register soon. Given that we have mostly workshops in this vodQA, seats are going to be limited!

Here is the address and direction to the ThoughtWorks office.

UPDATE:

Slides for my talk on the "Future of Testing, Test Automation and the Quality Analyst" are now available here:

Disruptive Testing with Julian Harty

As part of the Disruptive Testing series, the last interview of 2014, with Julian Harty is now available here (http://www.thoughtworks.com/insights/blog/disruptive-testing-part-8-julian-harty) as a video interview. The transcript of the same is also published.

Also look at ThoughtWorks Insights for other great articles on a variety of topics and themes.

The decade of Selenium

Selenium has been around for over a decade now. ThoughtWorks has published an eBook on the occasion - titled - "Perspectives on Agile Software Testing". This eBook is available for free download.

I have written a chapter in the eBook - "Is Selenium Finely Aged Wine?" 

An excerpt of this chapter is also published as a blog post on utest.com. You can find that here.

Perspectives on Agile Software Testing

Inspired by Selenium's 10th Birthday Celebration, a bunch of ThoughtWorkers have compiled an anthology of essays on testing approaches, tools and culture by testers for testers.  
This anthology of essays is available as an ebook, titled - "Perspectives on Agile Software Testing" which is now available for download from here on ThoughtWorks site. A simple registration, and you will be able to download the ebook. 

Here are the contents of the ebook:

Enjoy the read, and looking forward for the feedback.

Perils of Page-Object Pattern

I spoke at Selenium Conference (SeConf 2014) in Bangalore on 5th September, 2014 on "The Perils of Page-Object Pattern".

Page-Object pattern is very commonly used when implementing Automation frameworks. However, as the scale of the framework grows, there is a limitation on how much re-usability really happens. It inherently becomes very difficult to separate the test intent from the business domain.
 
I want to talk about this problem, and the solution I have been using - Business Layer - Page-Object pattern, which has helped me keep my code DRY.

The slides from the talk are available here. The video is available here. 

Video taken by professional:


Video taken from my laptop:


Slides:

Perils of Page-Object Pattern from Anand Bagmar

If you want to see other slides and videos from SeConf, see the SeConf schedule page.

What reporting / reporters you use with Selenium / WebDriver?

Test execution reports is usually an after-thought when doing automation. 

 

• What reporting techniques have you used on your automation projects (language / tools probably do not matter)? 
• What is the test log format used? 
• Do you use any special / different plugins, or rely on the CI tools with some plugins?
• What value do you get out of these reports?

 
 

Update from Webinar on "Build the 'right' regression suite using BDT" for NY Selenium Meetup

I had a challenging, yet good time speaking in a Webinar for the New York Selenium Meetup community on how to "Build the 'right' regression suite using Behavior Driven Testing (BDT)". This webinar was conducted on 6th May 2014 at 6.30pm and I am very thankful to Mona Soni to help organize the same.

Before I speak about the challenges, here are the slides and the audio + screen recording from the webinar. The video is not cleaned-up ... I had started recording the session and then we did wait for a few minutes before we started off, but you can forward to around the 01:15 min mark and audio starts from that point.

This was challenging because of 2 main reasons:
> With a webinar, I find it difficult to connect with the audience. I am not able to gauge if the content is something they already know about, so I can proceed faster. Or, if they are not following, I need to go slower. Or, the topic is just not interesting enough to them. There may be other reasons as well, but I just do not get that real-time feedback which is so important when explaining a concept and a technique.
Though there were some good interactions and great questions in form of chat, I miss that eye-to-eye connect. This webinar was conducted using GoToMeeting. Maybe next time I do this, I need to try to get webcams enabled for atleast a good few people attending to understand that body language.

> The 2nd challenge I had was purely my own body not being able to adjust well enough. I had flown in from India to Florida to speak in STAREAST 2014 conference just a couple of days ago, and was still adjusting to the jet-lag. Evening times turned out to be my lowest-energy points on the day and I felt myself struggling to keep focus, talk and respond effectively. I would like to apologize to the attendees if they felt my content delivery was not up to the mark for this reason.

I appreciate any feedback on the session, and looking forward to connect with you and talk about Testing, Test Automation, my open-source tools (TaaS, WAAT, TTA) and of course BDT!