Building a testing ecosystem for high-quality games in the long run

At Tactile, we work in a fast-paced environment and strive to push the boundaries of what our teams are capable of delivering. This is why we have several teams in the business which are focused solely on providing customised tools to our game development and production teams. This enables those teams to work more quickly and efficiently, allowing them to devote more time to innovation, collaboration and creating high-quality user experiences.

One of such teams is our Quality Assurance (QA) Automation team. Their new mission is to become an even bigger part of this toolkit providing ecosystem within Tactile. Their day-to-day workflow currently revolves around supporting five game teams, with Lily’s Garden being their biggest focus. They are a strong team of six with diverse backgrounds, experiences and skill sets, which serves them greatly in trying to shape their team’s mission, which will take them from testers who code to developers who build tools for testers.

To learn more about their transition and the impact it will have on our business as a whole, we sat down with three of the team members: Levente, Alex and Matic.

Previous ways of working

Previously, the automation team had two main channels for carrying out work activities. One was participating in the release process as QA, while the other was providing custom solutions to requests from various teams (such as the Gameboard or the Content Tools Team – who are building tools for our Story & Art teams).

Their process for release testing was managed separately from manual tester teams. This meant that the automation team was in charge of creating, maintaining, executing, and analysing the test runs. If they discovered any issues, they notified the tester stakeholders, who were then in charge of tracking them. This allowed them to take the initiative in testing things they thought were valuable from their perspective, but it also meant that the tester teams’ coverage overlapped.

The team’s new mission

The new scope of the team clearly separates the responsibilities between the two teams, while also empowering both to excel in their own areas. The Automation team’s new mission is essentially to create the automated test cases based on existing test passes (meaning the steps of the test have already been created), and then handing over the execution/analysis part to the tester team. The important detail, however, is that these new tests replace the previously manually executed tests, and thereby eliminate the duplicated work.

This therefore reduces the tedious and repetitive work and frees up time for manual testers to tackle more impactful tasks. It also allows the Automation team to spend more time focusing on constructing a maintainable framework for creating and maintaining these tests. All of this enables us to better ensure that our games work as intended, meet the quality standards we have set for ourselves and fulfil the brand promises we have made to our players. As Alex explains: “It’s not about making the software more complex, but about catering to a really niche need in the current infrastructure that hasn’t been catered to before. We’re collecting a lot of building blocks in the entire Tactile ecosystem and putting them all together. Exciting stuff!”

It is important to emphasise that there are some tests that are not planned to be automated, allowing manual testers to keep doing their exploratory work. After all, they are the ones who spend their days interacting with the devices and testing our games on them. It is therefore crucial that they are the ones who are writing the test plans (rather than the Automation team doing so separately), and the Automation team’s duty is to make sure that these steps get executed in a consistent and predictable manner on any given build.

What this new line-up will enable

This new style of working will ultimately save a lot of man hours by avoiding repetitive and lengthy test cases that may require a lot of preparation to run.

An excellent example of such tasks are update tests, which are one of the most tedious tasks assigned to our manual testers. It requires them to first download an old build, do the work they need to do on it, and then upgrade to the new release to make sure that progress persists and everything functions as expected. There is nothing exploratory about their work here, it is simply the tiresome task of downloading 2 separate builds every time a release is being shipped. The Automation team’s new infrastructure will allow them to test this scenario in a fully automated way.

The team also keeps track of the automation codebase corresponding to each build that was released to the users, so if there is ever a need to test functionality on older builds, the manual testers are able to easily pick a version of the tests that was compatible with that version. Alex explains: “If the manual team wants to go back ‘in time’ and test specific features, they can now easily do that. We offer backwards compatibility with old builds, which also enables us to do update tests.”

Alleviating the manual team from having to work on such tasks will therefore enable them to do the work that truly brings value to the release process – investigative work on recent crashes and exploratory tests around new areas that are more prone to having issues.

New challenges and opportunities arising from the transition

Whilst this transition presents many positive opportunities for change, it doesn’t come without a few new obstacles for the team. Since their tools will now be consumed by a team other than themselves, including people with potentially no coding skills, they must ensure that the reports generated clearly show the relevant details about the test performed.

This type of experience engineering has a significant impact on their product’s overall usability, but it also requires them to learn a new set of skills. Levente explains: “The small things add up. We need to account for these nuances when we are creating user-experiences. Previously, we never had to think about someone else interacting with our product, but now this consideration is at the heart of how we work.”

In this mission of creating the best possible experiences for the manual testers, the Automation team requires support from other teams, such as our client and back-end developers. These teams are responsible for developing and maintaining our internal backend infrastructure and can create new features within this infrastructure to provide more functionalities for our testers. This offers an opportunity for a deepened collaboration between the different teams acting in a supporting function.

A great example of the two teams’ work interlinking is the Build Server, our internal game building pipeline. The Build Server integrates the build ordering, test ordering and test report collection into one easy to navigate website. After a build compiles, the testers simply have to select which set of tests they would like to order and then the Test Agent (one part of the Build pipeline) takes care of orchestrating the execution across a pool of mobile devices. The results then get collected and sent back to the Build Server. “The game testers just need to enter one website, one central place, where they can complete this new workflow. The only thing we have to add to it is fitting the reports to their needs,” explains Matic.

Exciting times ahead

After this collaborative workflow has matured, and the coverage for the games is sufficient for the testers, the automation team will have more time to expand upon other exciting areas. One of such areas is Unity runtime testing (we use Unity as our game engine). This will allow them to test our games on the integration level, which will require a deeper insight into the underlying game code architecture. Another big item on their agenda is fully supporting iOS devices for test execution.

“We want to transition from not only black-box testing, but also white-box testing,” says Matic, “Because what we ultimately care about is that the games are performing, independent of the device they’re being played on.”

This shows how the QA Automation team is leveraging a unique constellation of tools and technologies, and is supported in their work by close collaboration with multiple teams across the company. This style of working not only enhances our current workflows but also ensures that our commitment to delivering the highest quality experiences to our players remains paramount, both now and in the long run.

About the co-authors

Levente Buzga completed his bachelor in Computer Science in Budapest. He started his career in a software engineering role within a big data company. Afterwards he relocated to the beautiful Copenhagen and ended up in the QA Automation role at Tactile. In September, he’s starting a master’s course focusing on AI.

Matic Jovan studied electrical engineering in Ljubljana. After completing his studies, he started his career in software development, where he dabbled in Android development and learned more about the general software development practices. He then moved to Copenhagen to start his master’s degree in Computer Science at DTU (Technical University of Denmark), where he also worked as an Android developer and QA Automation Engineer. After graduating, he joined the Tactile Automation team and is extremely excited about the direction shifting away from QA and more towards development.

Alex Anemogiannis is the newest addition to our QA Automation team. He started his career over 5 years ago and has since then worked in both development and QA. He first worked within a hyper-casual gaming studio, then moved to work in an indie gaming start-up, and finally landed in Copenhagen and joined our team at Tactile.