How a depressed robot ensures Adlib PDF is the best it can be
By Matt Jones | June 5, 2013
Marvin is a tool.
Essentially Marvin executes our test cases in an automated fashion. It helps us test that our software meets the necessary requirements.
Let me back up a bit…
Adlib PDF begins with developers building the software code, adding features, and addressing defects.
To ensure our code meets requirements and acceptance criteria, we need to create test cases, consisting of scripts of steps to execute and test particular functionality.
For example, say we add despeckle – the process of removing speckles from images and unwanted noise frequently associated with scanning – to OCR.
Developers code the requirements and create a build, while those in QA (Quality Assurance) compose tests to validate the acceptance criteria.
When the new code is finished, we need to perform regression testing to ensure all items we worked, on as well as previous functionality, aren’t broken.
Marvin represents a tremendous savings in time
It’s almost depressing to think about the number of test cases we have to execute. By giving Marvin the testing, we might be making him depressed.
(“Brain the size of a planet, and here I am running test cases.” – Marvin paraphrased)
To have a tester manually go through every step of every test is unsustainable, because we don’t have that many testers.
So we developed Marvin – a tool to launch those test cases and provide consistent, repeatable feedback.
A tester manually crafts test cases, and Marvin executes every step, returning a pass or fail response for those tests. And those tests can be re-executed whenever we like.
Marvin is a cool little tool that not only performs those steps, but also allows us to analyze the results. A typical scenario works like this…
• Copy files to a specific location
• Process those files using the application
• Analyze the results;
• Manually verify the results meet established acceptance criteria
• Verified results become the “baseline”
Subsequent testing is against that baseline. Marvin takes the latest test result and compares it to the baseline.
We were used to executing the steps manually, running the comparison, and “eyeballing” the results. Now Marvin executes the tests and reports passes and fails. In the case of failures, Marvin provides detailed feedback that assists Developers and QA in determining the root cause of the issue.
Marvin runs all of the tests
Before Marvin we had reached the stage at which we had to cherry pick which tests to run, because we didn’t have enough time for each one.
With Marvin we can execute exponentially more tests cases, whenever we want. For example, a nightly build can be compiled, all of our test cases can be executed, and the results can then be analyzed in the morning by the team.
Yes, we still perform spot checks to ensure Marvin is doing what it’s been told. The time savings however, gives us much more coverage, in terms of testing.
Marvin allows testers to do more free form exploratory testing, which is playing with the build in an unstructured way. Exploratory testing can find things that structured testing can’t. Marvin alleviates testers to do more fun stuff.
Some functions remain outside Marvin’s capability… for the moment. It has been designed so it is extensible – a very flexible tool that is constantly evolving.
The entire team can and has worked on Marvin. Certain individuals led to its creation… however the rest of the team has taken to adding new pieces and mechanisms to further expand the scope of our automation testing.
Marvin – the new Rock Star at Adlib!
About the Author