Dartboard - Are you on track?

In Agile Projects, we use the swim-lanes to track the status of the card life-cycle in an iteration. Unfortunately, the swim-lanes depict a sequential work-flow. Something has to come first, second, ... last. That puts the thought in a lot of minds that what is first is indeed first, and what is last, well, is last in the scheme of things.

That depicts testing being done towards the end - which is very anti-agile!

 

Testing starts off way before development is completed on any card. See the "Agile QA Process" for one way to do Testing on Agile projects.

On Agile projects, r

eality is that testing is going on in some fashion or the other right from the beginning. To help bring that visibility into the work-stream, I tried creating a grid (physically on a board, also in mingle) - with the rows representing the state of testing in each swim lane ... but that too was not as appealing as the image shown below. I chose to call it the Dartboard.

Dartboard - Are you on track?

Some explanation on how we used this:

Each triangle can represent your individual swim lanes. From testing perspective, we chose to club together "ready for Dev", "in Dev" and "in UI" in the same category.

There is a specific in Testing triangle - because there is some amount of work that definitely needs to be done from testing perspective AFTER development and BA signoff is complete.

The RED triangle means the card is blocked from ALL perspectives in the iteration.

The GREEN tringle means the card is completed from ALL perspectives (analysis, dev, testing - manual + automation)

In each of the triangle (except RED and GREEN), the different colors mean specific things:

- BLACK band = Testing not yet started

- RED band = Testing is blocked (maybe for additional info needed, etc)

- BLUE band = Testing is in progress. Could be identifying test cases, doing manual / exploratory testing, setting up test data, automation, etc.

As the card moves between triangles, the testing state of each of the card is very visible. 

Add to this a simple time line on top / bottom of the card to indicate where you are in the iteration, and you know if your "state-of-panic" is justified or not at a very quick glance at the dartboard.

Some swim lane states we have used:

> Backlog / Spillover?

> Ready for Dev / In Dev / In UI

> Ready for BA Signoff / In BA Signoff

> Ready for Test

> In Test / In Integrated Test

> Done

> Blocked

We also tracked each type of card separately:

Defects = RED cards / stickies

Story cards = Blue / Yellow cards / stickies

Tech cards = White cards / stickies

TaaS is available on rubygems.org

I have finally created and released the TaaS gem to rubygems.org. You can find it here.

To know more about TaaS - see all related blog posts here, or refer to the github project for the same.

Introducing Test Trend Analyzer (TTA)

The statement "I have a dream" is a very famous quote by American activist Martin Luther King Jr.

I resonate very closely with that. Here is why and how ...

Almost 2 years ago, I had a dream ... a vision about a product that can help those working in large organizations understand the health of their products / projects at a quick glance, instead of having a team of people manually scrambling frantically to collate and collect the information needed to get a sense of quality about the products they support. I chose to call this product Test Trend Analyzer - TTA

Given that Automation is a key factor in ensuring the success, quality and time-to-market for products, I took that as a baseline requirement and came up with a basic high-level diagram for TTA:

TTA - first diagram

Finally, a couple of months ago, I found a bunch of passionate people, who also had some time, to try and implement this tool.

We came up with this vision for TTA:

TTA Vision

Accordingly, we planned, prioritized, spiked, did some prototypes, did demos and showcases and got a base usable product completed.
This is a open-source project, available on github with more information about it on the github wiki.

The technology stack used is: RoR with mysql db. 

TTA falls in the Big Data + Visualization space - specific to Testing.




Watch this space for more information about TTA and the currently supported features. Email out to me if you need more specific information, or have questions on how can you use TTA, etc.

How can you contribute?

Given that we have implemented only a few basic features right now, and there are many more in the backlog, here is how you can help:

• Suggest new ideas / features that will help make TTA better
• Use TTA on your project and provide feedback
• More importantly, help in implementing these features

vodQA Geek Night - Test Automation Workshop

As a follow-up to our vodQA Geek Night - Behavior Driven Testing (BDT) workshop last year, we are conducting another workshop in ThoughtWorks, Pune focused on building a Test Automation Framework - the "right-way".

Here you will use your prior test automation experience to build a test automation framework from "scratch".

This will be a 2.5 / 3 hour workshop - starting at 5.30pm. Please plan to come in by 5.15pm latest to complete the registration formalities.

See more event and registration details on our vodQA Geek Night - Test Automation Workshop event page in vodQA group on facebook. 

Announcing vodQA Bangalore on May 18, 2013

ThoughtWorks is happy to announce yet another edition of the vodQA Bangalore happening on Saturday, 18th May 2013.
 
We take this opportunity to extend the invite to speakers and attendees for this event. Below is the mailer, with the event details. If you have a topic that you would like to present, please register with us. 

The vodQA event hosted by ThoughtWorks is being held the third time in Bangalore. We look forward to enlightening topics and proficient presenters, we look forward for your support and help on this. Please extend this invite to all your personal contacts whom you think will make a big impact at the event.

Note: Registration for Speakers close on 13th April' 2013.

Some important links:

Event page: get-set-test

Register as speaker

Register as attendee

thucydides for ATDD / BDD / BDT?

Anyone heard or used the tool "thucydides"-  for ATDD / BDD / BDT before?

Whats your review / thoughts on it? Any pros / cons? 

http://www.thucydides.info/
https://github.com/thucydides-webtests/thucydides

From their website: "Thucydides is a tool that lets you use WebDriver-based unit or BDD tests to write more flexible and more reusable WebDriver-based tests, and also to generate documentation about your acceptance tests, including a narrative description of test, along with the corresponding screen shots, and also high-level summaries and aggregations of the test results."

Security testing tools

Here is some information / links to potential tools / resources to use for security testing:

• KromCAT: http://firecat.toolswatch.org/kromcat/
• WebSecurify extension for Chrome: https://chrome.google.com/webstore/detail/websecurify
• safe3si: https://code.google.com/p/safe3si/ - seems interesting, for penetration testing
• Havij v1.16 Advanced SQL Injection: http://www.itsecteam.com/products/havij-v116-advanced-sql-injection/
• sqlninja: http://sqlninja.sourceforge.net/
• owasp (The Open Web Application Security Project): https://www.owasp.org/index.php/Testing_for_SQL_Injection_(OWASP-DV-005)
• sqlmap: http://sqlmap.org/

TaaS (Test-as-a-Service) coming to Sri Lanka and Delhi

Riding on the good response received on TaaS in 2012 and early 2013, I will be speaking more on the same in Colombo, Sri Lanka on 7th March 2013, and in Delhi on 14th March 2013. Looking forward to more feedback on TaaS!

What should be the new name for TaaS?

I have got a lot of interesting and valuable feedback for TaaS - Test-as-a-Service. One thing that stands out which people want is a different name for TaaS - since it does cause a lot of confusion with the business vertical of "Testing as a Service".

So, I am taking all this feedback seriously and considering changing the name of TaaS - before I publish the gem.

Since TaaS is essentially an orchestrator of your end-2-end Integration tests, some thoughts on potential names, with their meanings are:

• khansama - kindly suggested by Erik Petersen. A great contextual Indian name for a great tool that co-ordinates and organizes the testing.
• vahak - Sanskrit word - meaning Transporter, Conveyor, Carrier, Conduit
• dwar - Sanskrit word - meaning Door, Inlet, Outlet, Gate, Through, 
• honcho - leader, manager
• comptroller - controller
• relegate - refer to another for decision
• transmit - communicate

What do you think? What name do you like? Any other suggestions for better name?

rubygems.org & WAAT

With the recent problems the rubygems.org site is facing, you may not be able to get to the WAAT gem. In that case, you can also download the gem directly from the github repository. All information about WAAT can be found from my blog as before.

TaaS received pretty well in Pune conf

As mentioned in my earlier post, I spoke about TaaS - Test-as-a-Service in UNICOM's Next Generation Testing Conference in Pune on 23rd Jan. The talk went pretty well. Since this talk was for a slightly smaller duration, I had to use a scaled down version of the same slides that I used in the TaaS talk in Mumbai. The audio playback of the talk in Pune is available here.

There were a lot of interesting questions that came up this time:
1. Can I use TaaS to test in the cloud?
2. Can I use TaaS to do load / performance testing?
3. What are the security implications of using TaaS? 
4. Can I use TaaS to do mobile testing across different devices?

I will try to address these questions in subsequent posts.

Take a nap ... its good for you

... or so the research about "What Are the Facts About Naps?" says.

I particularly like #6.

So if someone finds me napping at work, I blame the article. :)

WAAT-Ruby crosses 1000 downloads on rubygems

Feeling very happy - the WAAT-ruby gem has crossed 1000 downloads from rubygems.org.

Taking video recordings of your test execution

I have always wanted to take video recordings of my test execution ... discard the video if the test passes, and save it if the test fails. 

Recently, I found that the headless gem does that for you. There is also some sample code to make the integration of this gem in your test framework very seamless.

While using the gem and its api's are very straightforward, this did not work very well for me.

Reasons:

1. I run my tests from various different types of machines / os, namely, mac, RHEL and Windows
2. The total number of machines I can run my tests from is huge (>20)  … mainly triggered via CI
3. The RHEL environment is controlled and configured by puppet.
4. There is a bunch of underlying library requirements for the video capture

So though it is easy to use the gem, it is not really easy to setup ALL test execution environments with these dependent libraries for this to work - at least for me.

Does anyone know of a better / easier solution?

Mac Air video problem? This may work for you

I have been using a Mac Air for some time now. One thing that has been a constant frustration for me on this machine is for some reason, the in-built camera stops working. For example, when I try to use Google Hangout or Skype, I get the error - "There is no camera connected". 

The System Profiler shows the camera is there.

Strangely, on restarting the machine, the camera starts working.

After lot of digging around, I found a post by "madams end" here which solved the problem for me.

The solution is: 

Go to Terminal.app and run the command:

sudo killall VDCAssistant

This fixed the problem for me.

The next thing I did was simply created a alias for this command in my shell - and then whenever I encounter the problem, just execute this alias.

TaaS (Test-as-a-Service) coming to Pune

2012 was a great year for me. I got the opportunity to share my learnings and open-source contributions in various conferences. 

2013 also seems to be starting off well. I will be talking in UNICOM's Next Generation Testing Conference, this time in Pune, on Wednesday, 23rd Jan. I will be repeating my talk from Mumbai on TaaS - Test-as-a-Service.

End-2-End automated Integration Testing using TaaS - Test as a Service

I spoke in Mumbai recently about an open-source framework created to assist in End-2-End automated Integration Testing using TaaS - Test as a Service. The slides for the talk are available here and audio here.

The session was planned for 1 hour. I managed to finish the talk in 50ish minutes, then then the Q&A went on for another 30ish minutes.

Plan is to create and publish TaaS as a gem on rubygems.org.

Looking forward to your comments / feedback for the same!!

Speaking in Bangalore about WAAT and Agile Testing

As mentioned in my earlier post, I spoke last week in UNICOM's Next Generation Testing Conference in Bangalore on 13th December. My talk was about the "The What, Why and How of Web Analytics Testing". The slides from the talk are available here on slideshare.net. The audio recording of the talk is available here. (You can download and use a player like VLC to listen to the recording.)

I had a great time speaking in this conference. My talk was probably the only very technical talk in the conference. Another thing I observed from the audience is that not many of them knew about Web Analytics. I managed to finish up my talk in 40 min, and surprisingly, for an audience who didn't know much about Web Analytics, there were no questions. BUT, in the lunch break and networking session, a lot of people came up to me and said they really enjoyed my talk, and will look forward to "seeing" more on how Web Analytics is used in their organisation. 

Also, there was a lot of interest and questions about Agile, and Agile Testing. This is a topic I can talk about for hours at length - and I controlled myself to a great extent to let other esteemed speakers also talk and answer questions about the same from the audience.

I ended up encouraging a lot of fellow speakers and attendees to think with an "open-mind" and be innovative and see how service organisations can provide "a good and true value" to customers, and why it is important to provide "good" solutions and ask the right and tough questions to the same! Also shared the concepts of TaaS with a few folks who potentially would be in similar soup with the "common test framework" concept.

All in all, a very good time in Bangalore. Now looking forward for the same in Mumbai on 18th December, where I will be talking about "Integration Testing in Enterprises - using TaaS (Test as a Service)".

December 2012 conferences - WAAT & TaaS it is!

December seems to be a busy time related to conferences for me.

First, I attended test-ed 2012 conference hosted by moolya where I had the opportunity to meet James Bach and a few other great speakers. 

I was a little let-down by James talk. It seemed like more of a marketing pitch for moolya - and somehow I felt I expected more from a person of his caliber! None-the-less, I am sure he inspired a lot of folks in the auditorium to become a free and innovative tester! 

I also got to talk with him 1-1 about BDD and what situations it works well, and more importantly, when it does not work well. Also spoke with him on how BDT (Behavior Driven Testing) helps in building the "right automated regression suite" and the challenges facing the Testers in India in order to become the "free-spirited, creative and innovative testers" he spoke about.

Next, on 13th December, I am speaking in UNICOM's Next Generation Testing Conference in Bangalore about "The What, Why, and How of Web Analytics Testing". I will be talking about my open-source framework - "WAAT - the Web Analytics Automation Testing", and how that can ease the manual drudge of web analytics testing.

To close the year, I will be speaking on 18th December in another UNICOM's Next Generation Testing Conference in Mumbai. Here I will be talking about "Integration testing in Enterprises using TaaS (Test-as-a-Service)- Via Case Study". This is about another open-source framework I have created - TaaS - Test as a Service.

Hope to see you all in Bangalore / Mumbai.

vodqa - Going Beyond the Usual updates

Our recently concluded vodQA Pune, on 13th October 2012(http://testing.thoughtworks.com/events/going-beyond-the-usual), was a huge success. Your participation, questions, comments and feedback helped us take this event to new heights. We thank you for that.

The first set of videos from this edition of vodQA are now available here: http://bit.ly/YtNwje, and some of thepictures are available here: http://bitly.com/TEWwjQ

Here are the links for the some of the talks that were held at vodQA:

- Opening note by Tarang Baxi 
YouTube: http://bitly.com/VN83v0

- Automated Infrastructure Testing - Ranjib Dey
YouTube: http://bitly.com/QhpYgz
Slides: http://bitly.com/UbSBgD

- The World Without Testing - Vikrant Chauhan and Supriya Pawar 
YouTube: http://bitly.com/TEXL2u

- The Lean Game- Sneha Kadam
YouTube: http://bitly.com/YtSzQO

- What's Accessibility - Vikrant Chauhan
YouTube: http://bitly.com/TtKBmC

Slides: http://bitly.com/Xd96JR
- Firefox Add-ons Use for Software Testers - Sumit Singhal
YouTube: http://bitly.com/RKi0HZ

Slides: http://bitly.com/VNaa1Y

- Redefining Bugs! - Sneha Kadam
YouTube: http://bit.ly/UtGgyR

Closing note: Chaitanya Nadkarny
YouTube: http://bit.ly/SHxZd9

You can join the following groups on Facebook and LinkedIn to stay up-to-date with what’s happening in vodQA,updates, meeting other testing enthusiasts.

LinkedIn group: vodQA
Facebook group: vodQA

WAAT v1.5.0 released to rubygems.org

The newest version of WAAT (Web Analytics Automation Testing framework) for Java and Ruby is now available with a new plugin - JsSniffer.

Following the announcement of the new JsSniffer plugin for WAAT, I have now completed the testing for the same, and released the gem - WAAT-1.5.0.gem to rubygems.org.

Please take a look at the FAQs and the major changes on Ruby / major changes on Java section for some known issues, potential solutions for the same - as some of these changes could break your tests.

WAAT (Java & Ruby) with JS_SNIFFER is out of the box

I have pushed in the latest changes to WAAT to get over the limitation of not working in a pure https environment (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html).

The solution is creating a new type of plugin - called JS_SNIFFER.

This plugin requires the user of WAAT to do a little more work than before. 

They need (to work with their development team) to figure out what JS script they need to invoke in the browser to get the URL of interest that is sent as a pure https request over the wire. WAAT then takes this request, and does the tag matching for you.

This generation of the JS script is a one-time effort - unless the way the tags are reported changes in the product. Then the test framework can work in a seamless fashion as before to test this is working consistently in an automated fashion. 

Another advantage of this is that with this approach, we do not need to install jpcap or run the tests as a "super-user" - a restriction posed by the network packet capture library.

The WAAT_v1.5.0.jar is available on here (https://github.com/anandbagmar/WAAT/tree/master/dist) on github.

Similarly, I have also updated the WAAT-ruby gem (WAAT-1.5.0.gem). This gem is not yet pushed out to rubygems.org - as I am still testing it out. However, if you are interested, you can download it from here.

As usual, feedback / comments / suggestions most welcome!

Error in building native extensions on mac / ruby?

If you encounter errors when a ruby gem on your mac (error in building native extensions), read more for a solution that worked for me.

I was trying to install ffi on my mac and got the following error:

sudo gem install ffi -v '1.0.7'


Building native extensions.  This could take a while...
ERROR:  Error installing ffi:
ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/ruby.h


Gem files will remain installed in /Library/Ruby/Gems/1.8/gems/ffi-1.0.7 for inspection.
Results logged to /Library/Ruby/Gems/1.8/gems/ffi-1.0.7/ext/ffi_c/gem_make.out


Here is what you need to do resolve the issue:

• Install xcode
• Install the "Command Line Tools" from xcode -> Preferences -> Downloads -> Components

Now you will be able to install the gem and build its native extensions on your mac.

An important UPDATE from my good friend Oscar Reiken:

Oscar Rieken5:52 PM (edited)

good tip ;) but that only works with newer versions of mac OSX where the latest version of Xcode is included, you might also want to add that if you dont have access to the latest xcode you can use brew(http://apple.stackexchange.com/questions/38222/how-do-i-install-gcc-via-homebrew) or install GCC directly(https://github.com/kennethreitz/osx-gcc-installer) 

New version of WAAT-ruby gem available

I finally got around to pushing out a new version of WAAT-ruby (1.4.1) on rubygems.org. The only change in this version is the removal of a dependency on a particular version of bundler. See the WAAT-ruby project on github for more information.

Watch this space for a new version of WAAT-ruby that overcomes the limitation of doing web analytics automation for https urls.

vodQA Pune - Going Beyond the Usual

The next edition of vodQA is coming up on Saturday, 13th October 2012 in Pune. The theme for this edition is "Going Beyond the Usual". There is going to be focus on functional and cross-functional areas like Security, Usability, Scalability and Performance, within industries such as Manufacturing and Banking, among others.

You can register as a speaker here, or as an attendee here. 

Whats next for WAAT?

It has been quite some time that I updated WAAT. The released version has been working well - but it does have its limitations as listed in the FAQs on github.

The biggest limitation I feel about the current release version of WAAT is that it does not work in a pure https kind of an environment. (http://essenceoftesting.blogspot.in/2011/06/waat-and-https.html)

Of late I have been spiking out different ways to overcome this limitation. I have experimented to create a HttpsSniffer, and hit various different road-blocks in that. That has forced me to look at another strategy. 

So I have changed direction in coming to a solution. I am looking at creating something like a JSInjector / JSSniffer plugin - which executes a javascript in the browser from where the action is invoked. This is not as straight forward to use as the earlier approaches. The user of this plugin will need to understand the DOM and some javascripts better, maybe take help from the development team, but then once the way to retrive the basic information is known, then we are in calm waters again :)

If you are facing this similar issue in a pure https environment for web analytics testing, look out for more information in this space.

My plan is to update WAAT, followed-by WAAT-Ruby and then lastly release a new version of the WAAT-Ruby gem following that.

vodQA - The ABCs of Testing

I am late in writing about this, but if I do not share this, I am sure I will be feeling very bad later on.

I am very happy to write the next vodQA is happening in Bangalore on Saturday, 8th Sept 2012. The theme for this event is "The ABCs of Testing (Automation, Big Data Analytics, Could Testing)". Also look at our vodQA group on facebook for more information.

There are great topics lined up by great speakers. There are going to be sessions across multiple tracks, fish bowl sessions and am sure a lot of thought-provoking interactions.

I will write an update with links to presentations, pictures and videos after the event. See you all there!

Google Has Open Sourced Octane, a New JavaScript Benchmark Suite

Posted from: Google Has Open Sourced Octane, a New JavaScript Benchmark Suite

Should now make it easier to test for performance more seamlessly and regularly.

----------------------------

Google has open sourced Octane, a JavaScript benchmarking suite consisting of 13 tests measuring browser performance.

Google has open sourced Octane, a JavaScript benchmarking suite consisting of 13 tests meant to measure the performance of browsers loading and executing complex and large JavaScript applications such as games, interactive and rich web pages and online tools. Octane consists of 8 tests found in the initial V8 Benchmark Suite plus the addition of 5 new ones – pdf.js, Mandreel, GB Emulator, Code Loading, Box2DWeb - that are meant to measure performance areas not covered yet by other tests:

• Richards - OS kernel simulation benchmark, originally written in BCPL by Martin Richards (539 lines).
• Deltablue - One-way constraint solver, originally written in Smalltalk by John Maloney and Mario Wolczko (880 lines).
• Raytrace - Ray tracer benchmark based on code by Adam Burmister (904 lines).
• Regexp - Regular expression benchmark generated by extracting regular expression operations from 50 of the most popular web pages (1761 lines).
• NavierStokes - 2D NavierStokes equations solver, heavily manipulates double precision arrays. Based on Oliver Hunt's code (387 lines).
• Crypto - Encryption and decryption benchmark based on code by Tom Wu (1698 lines).
• Splay - Data manipulation benchmark that deals with splay trees and exercises the automatic memory management subsystem (394 lines).
• EarleyBoyer - Classic Scheme benchmarks, translated to JavaScript by Florian Loitsch's Scheme2Js compiler (4684 lines).
• pdf.js - Mozilla's PDF Reader implemented in JavaScript. It measures decoding and interpretation time (33,056 lines).
• Mandreel - Runs the 3D Bullet Physics Engine ported from C++ to JavaScript via Mandreel (277,377 lines).
• GB Emulator - Emulate the portable console's architecture and runs a demanding 3D simulation, all in JavaScript (11,097 lines).
• Code loading - measures how quickly a JavaScript engine can start executing code after loading a large JavaScript program, social widget being a common example. The source for test is derived from open source libraries (Closure, jQuery) (1,530 lines).
• Box2DWeb - Based on Box2DWeb, the popular 2D physics engine originally written by Erin Catto, ported to JavaScript. (560 lines, 9000+ de-minified)

The benchmark runs in Chrome 14+, Firefox 13+, IE 10, Opera 12 and Safari 5.1.7+ on the desktop, and in the mobile versions of Chrome, Firefox and Opera. It does not run in IE 9 because Microsoft’s browser does not implement WebGL's Typed Arrays, and in several mobile browsers which fail to execute some of the tests: Android Browser, Chrome on iOS 4 (due to iOS restrictions), and Safari on iOS 4.

Octane is more comprehensive than other JavaScript benchmarking tests such as V8, SunSpider, Kraken or Dromaeo. Google mentioned their intent to keep improving the test suite, inviting users to fill in issues reporting performance areas or applications that can be used as a base for more comprehensive JavaScript testing.

The source code of the Octane benchmark is available under a New BSD License.