Slides from vodQA Pune - Less Talk, Only Action! now available

vodQA-Pune - Less Talk, Only Action! was held on on Saturday, 27th Aug 2016, 8.30am - 5.30pm at ThoughtWorks, Pune.

Agenda

Abstracts with Slides

1. Automating Web Analytics - Why? How?

Do you know –

• What is Web Analytics?How does Web Analytics work?
• Why is it important? How to test Web Analytics?
• How can we ensure correct data is sent to the Web Analytics server, every time, for all the actions?

Attend this workshop to learn ‘What is Web Analytics?’ and why it is an extremely important aspect of Software Development & Testing for your product / service to succeed!

We will share some techniques for testing Web Analytics - in a non-automated way - and why that is very challenging and error-prone.

We will learn, via hands-on activity, about WAAT - Web Analytics Automation Testing Framework (https://essenceoftesting.blogspot.com/search/label/waat) - an open-source solution, to automate validation of correct information / tags being sent to the Web Analytic server for different user actions as part of your regular Selenium-WebDriver Test Automation Framework.

Lastly, we will see how the impact of Analytics has changed dramatically with more adoption and spread of IoT (Internet of Things) and Big Data, and what we need to do to be part of the change, if not influencers of change!

Slides: automating-web-analytics

2. Performance testing with Gatling for Beginners

Gatling is a server side performance testing tool. This workshop aims at giving introduction to Gatling and facilitating participants to write their first performance tests using Gatling.

• Brief intro to Gatling
• Using Postman to check stub server(created using mountebank for workshop)
• Write a sample test using gatling (pre set up machines are provided)

Slides: gatling-performance-workshop

3. Game of Test Automation

We are going to use a game to work out the WHY, WHAT and HOW of test automation within the context of consumers, application, skillset, mindset, etc.

4. Security Testing - Operation Vijay

It is now the days of the web. All businesses move their applications online for their customers to use.

Many of these applications contain critical data of the customers such as credit card details, their personal details and so on. These data are very valuable. If these data fall in the wrong hands, they can have disastrous consequences.

Attacks on these systems can destroy the trust that the customers have for the business, they can cause great losses to the customers as well as the business, and so on.

The motivation behind attacks could be different like earning money, earning popularity, destroying a competitor company, etc.

No matter what the intention of the attack is, we need to develop safe applications and we need to know the various vulnerabilities and the consequences of our decisions when we develop applications.

Similarly, we should be aware of the various vulnerabilities before we test the applications so that we can try and exploit it during the testing phase and ensure better quality and safer applications.

Slides: security-testing-operation-vijay

5. Automate your Mobile tests with Appium

• Introduction to Appium.
• Appium design for Android and iOS
• How to locate elements on android and iOS applications(Inspector).
• Hands-on code snippet for android and iOS(Wordpress as sample app)
• Generating reports (ExtentReports)
• Evolve code snippet written above into a Page Object Framework.
  

Slides: mobile-automation-using-appium

6. Increase Automation to REST

• What are web services and why we use them?
• How to test a web service in multiple ways?
• Increased familiarity with automation

Tools used : rest client, postman (mention alternatives), unirest JAVA and requests python

Slides: increase-automation-to-rest, api-webservice-setup-instructions

7. Let's cook Cucumber

In this workshop we will be covering:

• Advantages of BDD through cucumber example.
• Framework setup along with JAVA and Selenium.
• Writing one end to end test case in real world.
• If time permits - will be covering basic Refactoring
  

Slides: lets-cook-cucumber
 

8. Mobile Automation using Espresso

Imagine a situation where every commit spits out a build that can be deployed to production with confidence. In today's startup era, this can be a huge boost to business as it will reduce the time to market. UI Automation for mobile apps, be it native or hybrid, has been painful since long. But with mature frameworks coming up and Google/Apple realizing the importance of such tools, UI Automation is gaining traction in the mobile space. 

This talk is basically to understand what and why of espresso along with automating a simple scenario using espresso.

Slides: getting-high-on-espresso

Agile Testing in Singapore

As part of Next Generation Testing Leadership ASIA-PAC Summit, I will be doing a 1-day workshop on Agile Testing on Monday, 19th October 2015 in Singapore. Below are the details for the same. Also, I will be speaking on "What is Agile testing? And How Does Automation Help" on Tuesday, 20th October 2015. Do let me know if you are interested in attending the same.

Agile Testing Workshop

The Agile Manifesto was published in 2001. It took the software industry a good few years to truly understand what the manifesto means, and the principles behind it. However, choosing and implementing the right set of practices to get the true value from working the Agile way has been the biggest challenge for most!

While Agile has now gone mainstream, and as we get better at the development practices to being Agile, Testing has still been lagging behind in most cases. A lot of teams are still working in the staggered fashion - with testing following next after development completed.

In this workshop, we will learn and share various principles and practices which teams should adopt to be successful in testing on Agile projects. 

Agenda :

• What is Agile testing? - Learn what does it mean to Test on Agile Projects
• Effective strategies for Distributed Testing - Learn practices that help bridge the Distributed Testing gap!
• Test Automation in Agile Projects - Why? What? How? - Why is Test Automation important, and how do we implement a good, robust, scalable and maintainable Test Automation framework!
• Build the "right" regression suite using Behavior Driven Testing (BDT) - Behavior Driven Testing (BDT) is an evolved way of thinking about Testing. It helps in identifying the 'correct' scenarios, in form of user journeys, to build a good and effective (manual & automation) regression suite that validates the Business Goals.

Key Learnings for participants in this workshop :

• Understand the Agile Testing Manifesto
• Learn the essential Testing practices and activities essential for teams to adopt to work in Agile way of working
• Discover techniques to do effective testing in distributed teams
• Find out how Automation plays a crucial role in Agile projects
• Learn how to build a good, robust, scalable and maintainable Functional Automation framework
• Learn, by practice, how to identify the right types of tests to automate as UI functional tests - to get quick and effective feedback

Pre-requisites :

• Basic working knowledge and understanding of Agile

Questions about the Test Pyramid

After watching my presentation on "Enabling Continuous Delivery (CD) in Enterprises with Testing", I recently got asked a couple of questions about the Test Pyramid. Thought it would be good to reply publicly - that may help others who had similar doubts. 

If you have any other questions, please reach out, or add it as comments on this post.

• Why do you talk about a JavaScript Test? I mean, why you don't consider this type of testing inside another? So, what do you mean by JavaScript test.
• JavaScript testing requires different toolset, not the standard xUnit based ones. Hence I classify it separately. Also, there is potentially a lot of logic that can be built in the JavaScript layer - so it is essential to write tests for that too - say using Jasmine.

• What's the difference between View and UI?
• UI test should focus on business / user journey validations. However a view test is different. Consider a journey which has a 5 step / screen workflow. To validate some UI change on the 4th step / screen, you will need to go through, in sequence, from step 1 to 4 and then validate the changes. This is very slow and costly approach. Instead, if you build the right type of stubs / mocks, then you can setup the state in your product which simulates the step 1-3 are completed, and directly open the UI, go to step #4, and validate your changes. This is the difference in View and UI tests.

Experiences from "Agile Testing" workshop

As mentioned in this post, I conducted an "Agile Testing" workshop on 24th July 2015.

Here are the slides from the workshop:

• What is Agile testing? - Learn what does it mean to Test on Agile Projects

What is Agile Testing? from Anand Bagmar

• Effective strategies for Distributed Testing - Learn practices that help bridge the Distributed Testing gap!

Strategies for Distributed Agile Testing from Anand Bagmar

• Test Automation in Agile Projects - Why? What? How? - Why is Test Automation important, and how do we implement a good, robust, scalable and maintainable Test Automation framework!

Test automation - What? Why? How? from Anand Bagmar

• Build the "right" regression suite using Behavior Driven Testing (BDT) - Behavior Driven Testing (BDT) is an evolved way of thinking about Testing. It helps in identifying the 'correct' scenarios, in form of user journeys, to build a good and effective (manual & automation) regression suite that validates the Business Goals.

Build the "right" regression suite using Behavior Driven Testing (BDT) from Anand Bagmar

What is Agile Testing? How does Automation help?

I spoke in a conference recently on "What is Agile Testing? How does Automation help?"

Abstract

Agile Methodology is not new. Many organisations / teams have already adopted Agile way of Software Development or are in the enablement journey for the same. 

What does this mean for Testing? There is no doubt that the Testing approach and mindset also needs to change to be in tune with the Agile Development methodology. 

Learn what does it mean to Test on Agile Projects. Also, learn how Test Automation approach needs to change for the team to be successful!

Video

Slides

What is Agile Testing? How does Automation Help?

Here I am on the stage, in Main Hall, in front of 150+ people, delivering the talk - What is Agile Test? How does Automation Help?:

Enabling CD & BDT in March 2015

I have been very busy off late .... and am enjoying it too! I am learning and doing a lot of interesting things in the Performance Testing / Engineering domain. I had no idea there are so many types of caching, and that there would be a need to do various different types of Monitoring for availability, client-side performance testing, Real User Monitoring, Server-side load testing and more ... it is a lot of fun being part of this aspect of Testing.

That said, I am equally excited about 2 talks coming up in the end-of-March 2015:

Enabling CD (Continuous Delivery) in Enterprises with Testing 

- at Agile India 2015, on Friday, 27th March 2015 in Bangalore.

Abstract

The key objectives of Organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality!
In such a fast moving environment, CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury!

There are various practices that Organizations and Enterprises need to implement to enable CD. Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful.

Testing in Organizations on the CD journey is tricky and requires a lot of discipline, rigor and hard work. In Enterprises, the Testing complexity and challenges increase exponentially.

In this session, I am sharing my vision of the Test Strategy required to make successful the journey of an Enterprise on the path of implementing CD.

Build the 'right' regression suite using Behavior Driven Testing (BDT) - a Workshop

- at vodQA Gurgaon, on Saturday, 28th March 2015 at ThoughtWorks, Gurgaon.

Abstract

Behavior Driven Testing (BDT) is a way of thinking. It helps in identifying the 'correct' scenarios, in form of user journeys, to build a good and effective (manual & automation) regression suite that validates the Business Goals. We will learn about BDT, do some hands-on exercises in form of workshops to understand the concept better, and also touch upon some potential tools that can be used.

Learning outcomes

• Understand Behavior Driven Testing (BDT)
• Learn how to build a good and valuable regression suite for the product under test
• Learn different style of identifying / writing your scenarios that will validate the expected Business Functionality
• Automating tests identified using BDT approach will automate your Business Functionality
• Advantages of identifying Regression tests using BDT approach

Experiences from webinar on "Build the 'right' regression suite using Behavior Driven Testing (BDT)"

I did a webinar on how to "Build the 'right' regression suite using Behavior Driven Testing (BDT)" for uTest Community Testers on 18th Feb 2015 (2pm ET).

The recording of the webinar is available here on utest site (http://university.utest.com/recorded-webinar-build-the-right-regression-suite-using-behavior-driven-testing-bdt/).

The slides I used in the webinar can be seen below, or available from slideshare.

Build the "right" regression suite using Behavior Driven Testing (BDT) from Anand Bagmar

Here are some of my experiences from the webinar:

• It was very difficult to do this webinar - from a timing perspective. It was scheduled from 2-3pm ET (which meant it was 12.30-1.30am IST). I could feel the fatigue in my voice when I heard the recording. I just hope the attendees did not catch that, and that it did not affect the effective delivery of the content.
• There were over 50 attendees in the webinar. Though I finished my content in about 38-40 minutes, the remaining 20 minutes was not sufficient to go through the questions. The questions itself were very good, and thought provoking for me.
• A webinar is a great way to create content and deliver it without a break - as a study material / course content. The challenge and the pressure is on the speaker to ensure that the flow is proper, and the session is well planned and structured. Here, there are no opportunities to tweak the content on the fly based on attendee comments / questions / body language.
• That said, I always find it much more challenging to do a webinar compared to a talk. Reason - in a talk, I can see the audience. This is a HUGE advantage. I can understand from their facial expressions, body language if what I am saying makes sense or not. I can have many interactions with them to make them more involved in the content - and make the session about them, instead of me just talking. I can spend more time on certain content, while skipping over some - depending on their comfort levels. 

WAAT at StarEast2014

I was speaking about "Build the 'right' regression suite using Behavior Driven Testing (BDT)" at StarEast 2014 and met Marcus Merrell who was speaking about "Automated Analytics Testing with Open Source Tools". I figured out deep into our conversation that he has used WAAT, and a few others at the table also were aware of and had used WAAT before. Felt great!

Update from StarEast 2014 - "Build the 'right' regression suite using BDT"

I had a great time speaking about how to "Build the 'right' regression suite using Behavior Driven Testing (BDT)".

This time, I truly understood the value of practice, dry-runs. Before the webinar for NY Selenium Meetup group, I presented the content to my colleagues in the ThoughtWorks, Pune office and also shared the same with the hivemind @ ThoughtWorks and got great feedback. That helped me tremendously in making my content more rock-solid and well-tuned.

As a result, I was able to deliver the content, to a very enthusiastic and curious audience, who turned up in great numbers (>105) to hear about what the $@$#^$% is BDT, and how can it help in avoiding the nightmare of long, unfruitful, painful Regression Test cycles.

The slides from the talk are available here and the video is available here.

Update from Webinar on "Build the 'right' regression suite using BDT" for NY Selenium Meetup

I had a challenging, yet good time speaking in a Webinar for the New York Selenium Meetup community on how to "Build the 'right' regression suite using Behavior Driven Testing (BDT)". This webinar was conducted on 6th May 2014 at 6.30pm and I am very thankful to Mona Soni to help organize the same.

Before I speak about the challenges, here are the slides and the audio + screen recording from the webinar. The video is not cleaned-up ... I had started recording the session and then we did wait for a few minutes before we started off, but you can forward to around the 01:15 min mark and audio starts from that point.

This was challenging because of 2 main reasons:
> With a webinar, I find it difficult to connect with the audience. I am not able to gauge if the content is something they already know about, so I can proceed faster. Or, if they are not following, I need to go slower. Or, the topic is just not interesting enough to them. There may be other reasons as well, but I just do not get that real-time feedback which is so important when explaining a concept and a technique.
Though there were some good interactions and great questions in form of chat, I miss that eye-to-eye connect. This webinar was conducted using GoToMeeting. Maybe next time I do this, I need to try to get webcams enabled for atleast a good few people attending to understand that body language.

> The 2nd challenge I had was purely my own body not being able to adjust well enough. I had flown in from India to Florida to speak in STAREAST 2014 conference just a couple of days ago, and was still adjusting to the jet-lag. Evening times turned out to be my lowest-energy points on the day and I felt myself struggling to keep focus, talk and respond effectively. I would like to apologize to the attendees if they felt my content delivery was not up to the mark for this reason.

I appreciate any feedback on the session, and looking forward to connect with you and talk about Testing, Test Automation, my open-source tools (TaaS, WAAT, TTA) and of course BDT!

BDT in Colombo Selenium Meetup

[UPDATED again] Feedback and pictures from the virtual session on BDT

 









[UPDATED]
The slides and audio + slide recording have now been uploaded.


I will be talking virtually and remotely about "Building the 'right' regression suite using Behavior Driven Testing (BDT)" in Colombo's Selenium Meetup on Wednesday, 23rd October 2013 at 6pm IST.

If you are interested in joining virtually, let me know, and if possible, I will get you a virtual seat in the meetup.

Offshore Testing on Agile Projects

Offshore Testing on Agile Projects …

Anand Bagmar

Reality of organizations

Organizations are now spread across the world. With this spread, having distributed teams is a reality. Reasons could be a combination of various factors, including:

Globalization	Cost
24x7 availability	Team size
Mergers and Acquisitions	Talent

The Agile Software methodology talks about various principles to approach Software Development. There are various practices that can be applied to achieve these principles. 

The choice of practices is very significant and important in ensuring the success of the project. Some of the parameters to consider, in no significant order are:

Skillset on the team	Capability on the team
Delivery objectives	Distributed teams
Working with partners / vendors?	Organization Security / policy constrains
Tools for collaboration	Time overlap time between teams
Mindset of team members	Communication
Test Automation	Project Collaboration Tools
Testing Tools	Continuous Integration

** The above list is from a Software Testing perspective.

This post is about what practices we implemented as a team for an offshore testing project.

Case Study - A quick introduction

An enterprise had a B2B product providing an online version of a physically conducted auction for selling used-vehicles, in real-time and at high-speed. Typical participation in this auction is by an auctioneer, multiple sellers, and potentially hundreds of buyers. Each sale can have up to 500 vehicles. Each vehicle gets sold / skipped in under 30 seconds - with multiple buyers potentially bidding on it at the same time. Key business rules: only 1 bid per buyer, no consecutive bids by the same buyer.

Analysis and Development was happening across 3 locations – 2 teams in the US, and 1 team in Brazil. Only Testing was happening from Pune, India.

George Bernard Shaw said:

“Success does not consist in never making mistakes but in never making the same one a second time.”

We took that to heart and very sincerely. We applied all our learning and experiences in picking up the practices to make us succeed. We consciously sought to creative, innovative and applied out-of-the-box thinking on how we approached testing (in terms of strategy, process, tools, techniques) for this unique, interesting and extremely challenging application, ensuring we do not go down the same path again.

Challenges

We had to over come many challenges for this project.

• Challenge in creating a common DSL that will be understood by ALL parties - i.e. Clients / Business / BAs / PMs / Devs / QAs
• All examples / forums talk using trivial problems - whereas we had lot of data and complex business scenarios to take care of.
• Cucumber / capybara / WebDriver / ruby do not allow an easy way to do concurrency / parallel testing
• We needed to simulate in our manual + automation tests for "n" participants at a time, interacting with the sale / auction
• A typical sale / auction can contains 60-500 buyers, 1-x sellers, 1 auctioneer. The sale / auction can contain anywhere from 50-1000 vehicles to sell. There can be multiple sales going on in parallel. So how do we test these scenarios effectively?
• Data creating / usage is a huge problem (ex: production subset snapshot is > 10GB (compressed) in size, refresh takes long time too,
• Getting a local environment in Pune to continue working effectively - all pairing stations / environment machines use RHEL Server 6.0 and are auto-configured using puppet. These machines are registered to the Client account on the RedHat Satellite Server.
• Communication challenge - We are working from 10K miles away - with a time difference of 9.5 / 10.5 hours (depending on DST) - this means almost 0 overlap with the distributed team. To add to that complexity, our BA was in another city in the US - so another time difference to take care of.
• End-to-end Performance / Load testing is not even a part of this scope - but something we are very vary of in terms of what can go wrong at that scale.
• We need to be agile - i.e. testing stories and functionality in the same iteration.

All the above-mentioned problems meant we had to come up with our own unique way of tackling the testing.

Our principles - our North Star

We stuck to a few guiding principles as our North Star:

• Keep it simple
• We know the goal, so evolve the framework - don't start building everything from step 1
• Keep sharing the approach / strategy / issues faced on regular basis with all concerned parties and make this a TEAM challenge instead of a Test team problem!
• Don't try to automate everything
• Keep test code clean

The End Result

At the end of the journey, here are some interesting events from the off-shore testing project:

• Tests were specified in form of user journeys following the Behavior Driven Testing (BDT) philosophy – specified in Cucumber.
• Created a custom test framework (Cucumber, Capybara, WebDriver) that tests a real-time auction - in a very deterministic fashion.
• We had 65-70 tests in form of user journeys that covers the full automated regression for the product.
• Our regression completed in less than 30 minutes.
• We had no manual tests to be executed as part of regression.
• All tests (=user journeys) are documented directly in Cucumber scenarios and are automated
• Anything that is not part of the user journeys is pushed down to the dev team to automate (or we try to write automation at that lower level)
• Created a ‘special’ Long running test suite that simulates a real sale with 400 vehicles, >100 buyers, 2 sellers and an auctioneer.
• Created special concurrent (high speed parallel) tests that ensures even at highest possible load, the system is behaving correctly
• Since there was no separate performance and load test strategy, created special utilities in the automation framework, to benchmark “key” actions.
• No separate documentation or test cases ever written / maintained - never missed it too.
• A separate special sanity test that runs in production after deployment is done, to ensure all the integration points are setup properly
• Changed our work timings (for most team members) from 12pm - 9pm IST to get some more overlap, and remote pairing time with onsite team.
• Setup an ice-cream meter - for those that come late for standup.

Innovations and Customizations

Necessity breeds innovation! This was so true in this project.

Below is a table listing all the different areas and specifics of the customization we did in our framework.

Dartboard

Created a custom board “Dartboard” to quickly visualize the testing status in the Iteration. See this post for more details: “Dartboard – Are you on track?”

TaaS

To automate the last mile of Integration Testing between different applications, we created an open-source product – TaaS. This provides a platform / OS / Tool / Technology / Language agnostic way of Automating the Integrated Tests between applications.

Base premise for TaaS:

Enterprise-sized organizations have multiple products under their belt. The technology stack used for each of the product is usually different – for various reasons.

Most of such organizations like to have a common Test Automation solution across these products in an effort to standardize the test automation framework.

However, this is not a good idea! If products in the same organization can be built using different / varied technology stack, then why should you pose this restriction on the Test Automation environment?

Each product should be tested using the tools and technologies that are “right” for it.

“TaaS” is a product that allows you do achieve the “correct” way of doing Test Automation.

See my blog for all information related to TaaS.

WAAT - Web Analytics Automation Testing Framework

I had created the WAAT framework for Java and Ruby in 2010/2011. However this framework had a limitation - it did not work products what are configured to work only in https mode.

For one of the applications, we need to do testing for WebTrends reporting. Since this application worked only in https mode, I created a new plugin for WAAT  - JS Sniffer that can work with https-only applications. See my blog for more details about WAAT.