Automated Solutions

Data Science Specialist: Probability & Bayesian Statistics

"My ultimate goal is to help drive us toward a data driven automated future."

The Importance of Test Automation

During any release of a major product the testing cycle becomes increasingly important. Automation is not the end all be all for verifying software quality, but it sure does the trick for minimizing load on test teams. As I mentioned before at HotelTonight we utilize automated test strategies for a number of our behavior verifications. One question I've heard repeatedly in the past is "How does one go about determining what tests should be automated?" More importantly, "How do you implement automation into your test strategy once you've identified what to automate?" I'm going to use this blog post to answer both of those questions by taking a real world scenario, and illustrating how one could use automation to address it.

Within the mobile app for HotelTonight, our objective is to give our users the best options for booking last minute hotels. The thing is these hotels don't only encompass US options. We also will need to serve users hotel options in other countries. Serving users in other countries means that we need to deal with other time zones. If our market opens at say... 9am for our users, then thats a piece of cake to test manually from a San Francisco users perspective. Its not until you think about 9am local time for a specific user that is not under the pacific time zone scenario when testing this behavior gets a bit complex.

Lets also say that QA is given the task to check whether or not our prospective markets are showing our users hotels when the markets are open, as well as showing the correct message for our users during times when markets are closed. QA will ideally get a list of those cities opening and closing times and then go through the app and check if the app is showing the correct results at the appropriate time in relation to the specific time zone.

As you can imagine this could turn into a tedious process. After all, there are multiple cities on the planet, and there are multiple time zones. The US alone has multiple time zones that would need to be considered. Running through these time zone checks every time market opens to verify that inventory is available everyday for our users can be cumbersome for QA, especially if some of those markets open at 12am, or 1am pacific time!!! This sounds like a great candidate for automation.

How do we implement test automation to tackle this problem?  The answer is broken into these steps.

  1. Gather a list of Major Cities that represent different time zones (Paris, London, New York, San Francisco)
  2. Determine how to represent the 9am - 2am market open and close window (Relative to where the tests will be running. In this case in the pacific timezone).
  3. Most importantly, what should the automated tests expect to see during their execution of these behaviors? If the Market is closed, the user needs to see that appropriate message. If the market is open then they better see hotels, or we have a problem!
Using Appium to host the app, and the power of RSpec to drive the tests, lets look at what our automation logic looks like for the 4 cities listed above. Keep in mind the times are based on a 24 hour format from the pacific time zone perspective. This keeps the tests valid even during daylight savings time changes! 


As you can see we are searching for hotels in our 4 major cities I've chosen. The catch here is these automated tests will run at any time, so they need to be smart enough to handle these open/close time windows for each city. In addition to understanding the time window the tests also need to be smart enough to know that sometimes it will see a different message based on these time windows, especially if the tests run during hours that a specific market is closed for that city. These tests can now be added to a scheduler and run anytime of the day, seven days a week with no human interaction whatsoever! Check out how these automated tests look in action below!!