Automated Solutions

Automation Engineer to the Core!

"Test Automation is the key to dependable mobile and web applications. I make it my business to have computers work for us and not the other way around."

The Automated Selfie.....

During test automation things are going to go wrong. This fact doesn't have to rain on your parade though. While working on different projects at HotelTonight I've decided to overhaul the way that we do error handling within a few of our key test suites. This post will focus on a mobile app failure scenario.

On average the mobile suite I've put together takes about 15 minutes to complete a few key regression tasks. I expect the amount of time the tests take to reach completion to increase in the near future. With test runtime increasing, and more and more complexity being added to the code base, the need for smart error handling shows its head.

There have been a number of times that tests would be running and a failure would happen. Debugging these errors became an absolute pain, especially if there was no-one around to see exactly what app state was like during those failures. To tackle this problem on the mobile side I decided to do two things.

1.) Make the errors that were being reported way more realistic and less vague.(More on this on the next post)
2.) Snap an image of what the app state was like when the error was encountered.

Both of these solutions together make for an amazingly smart tool for debugging automation failures. Debugging errors for us now have gone from tedious to being absolutely painless. The amount of time it takes for me or anyone else to know what happened has gone from 20 minutes of investigation time to less than 1 minute in most cases.

Here's a brief example....

Lets say that we have some kind of an outage during test run time. These outages can be caused by anything from db migrations, bad api requests, all the way to just plain silly mistakes made. Lets also say that during this time I went out for a cup of coffee while the tests were happily doing their thing. When I return I see something like this in my test report report......


We can easily see what type of error it is, but we have no idea why this was thrown while trying to search for a random city on the app. Luckily, the automated selfie was in place. Lets checkout the screen shots directory....


BINGO! We've got a little bit more information. The time date and month as well as the exact test action it was performing. That tells me when, but I still need a little more info on what the app witnessed. If you open the screenshot .png we see this...



Now this is more like it. We clearly see exactly what the test saw while it was running. We had a failure during our same day booking test at the exact time of the screenshot, and we see exactly what the app was showing during the test run. This technique makes it much easier to approach a group of engineers about failures. This information gives us a great place to start investigating.

How does all this work using Appium? Just a few lines of code to do the trick in the rspec spec_helper.rb. Here is the code that makes this all work...


In short, after each scenario that has an exception take a screenshot and place it into my specified directory. I've also added some time formatters and test descriptions as well to make the reasons for the images much more clearer.

As you can imagine this folder will grow and grow with more and more failures happening. The last thing I want is a directory full of failed test images that I no longer need to reference. So I've written a task that clears them out called "rake clear_screenshots". Here is code example of that...



Once this is run you'll see the directory below is now empty again, ready for the next test run.