All of our game projects (we have 14 live at this point in time!) have basic functionalities which integrate them into the Tactile ecosystem. These functionalities are interfaced with our back-end services, such as game web services (for example scheduled features, A/B testing, in-app purchases, configurations, authentications, etc.) and our internal data analytics collector. All of these services have a what we call ‘client’ component, which is their underlying framework. And since all of our game projects share these, we have a dedicated team of developers responsible solely for building and maintaining them. They’re our Core Client Team.

We sat down with two of our Client developers, Martin Traberg and Martin Gonzalez to learn more about what they do and how their work impacts the work our game teams do. So, grab yourself a cup of coffee or mate and let’s dive into the world of game programming.

‘Separate, but connected’

The Core team at Tactile provides services and tools, all built in-house, which help our game teams develop games faster and in a more efficient way. Having a dedicated team within our Core team looking after the lower-level systems and services that are shared between our games, allows our game programmers to focus on what is truly important to them – building all the fun features for our players to enjoy.

On top of looking after the shared code-base, our Client team provides custom-built tooling, which helps to speed up the game development process. It does so by allowing our game programmers to interact with the shared code base in a more effective way. The Client team is also responsible for the client component linked to our Build Server, which is our internal game building pipeline (see the ‘Build Agents: A look into Tactile’s internal build pipeline’ article for more info on this). This allows all game projects and those working on them to utilise our Build Server and create new game builds based on specific requirements.

‘Removing the walls in our codebase’

The Client Team serves many departments within Tactile. It takes ‘orders’ from a variety of stakeholders requiring information from the games – this can be people who are directly working on the projects (gameplay programmers, designers, product specialists), but also those who are not (producers, data analysts, marketeers and backend engineers).

Once an ‘order’ is raised, it is the Client Team’s responsibility to create a feature or build a system which then gets implemented and used in all of our games. Executing these orders requires a lot of communication with the rest of the teams, as the engineers need to understand what the teams are doing and how these new systems will fit into their daily workflows. What is more, our game teams are distributed across several geographical locations. So whenever the centralised Client Team, which is based in Copenhagen, prepares a new release, it needs to be integrated by all the other teams. This is where the good communication factor really comes into play.

As Martin G. elaborates: “We have a huge amount of shared code. For example, nearly 80% of Lily’s Garden code base is shared between different game projects*. The best thing is that there are no walls in our code and every developer can add to the shared code base. We have a kind of internal open source mindset. Even if our team is backed up, other developers can go into the shared code base, make a change and then submit a pull request for our team to review at a later time.” This prevents the Client Team from becoming a bottleneck in the development process. Ideally, they only need to step in when there’s a larger change which requires more attention and input from them.

Think of it like this – the Client team provides the underlying framework for the work that the game teams do. They are the most experienced developers when it comes to dealing with all the different projects and their needs. They really understand when something you are going to change in one place is going to positively (or negatively) impact something else in another place

Martin T., who is leading the Client Team, has been at Tactile for over 6 years (at the time of writing). “A lot has changed since I started,” he says, “We have moved a lot of things to the shared domain and things are a lot more structured than they used to be. We now have proper guidelines for how we develop and are using more modern technologies and coding practices in general. The quality of the code has really become a first-class citizen and everyone is very mindful of the maintainability of our products.” On top of this, the team spends a lot of time discussing how to balance the shared code base, how much should be shared and how much should be completely on the game side. It’s an ongoing balancing act.

‘Balancing compatibility whilst minimising disruption’

The real impact of the work that we do at Tactile is created when the players play our games and experience the new in-game features which our game teams worked on. Linked to that, the internal impact of the work that our Core team does comes from providing the game teams with the speed to develop. For example when we start working on a new game project, we have the infrastructure ready that integrates the game with all of our core services. This gives the game teams the time to focus on building new features which are unique to that project.

One of the biggest impacts that the Client team has on our game teams on a daily basis is producing new versions of the shared components – which the game teams need to keep up to date with. Sometimes the Client team needs to make these updates because they want to change something that is a real impediment for them. In this case, the game teams are affected by proxy. Updating to the new version of a shared component takes time and might temporarily disrupt the team’s workflow or introduce potential for making bugs. However, it contributes to the overall and long-term health of the shared codebase, which is the ultimate goal of the Client team. Martin G. explains: “There is a delicate balance between pushing the shared components into the direction we’ve settled on and not being too disruptive to the game teams.”

This is where the Client team’s work gets most challenging. Sharing as much code as possible provides speed of development. But, every time the team releases an update to the shared components, it becomes the responsibility of each individual game team (of which there are multiple in several geographical locations) to actually update them. It is not an automatic process because the game teams need to be in charge of what they’re putting into their projects.

And so the true challenge lies in convincing the game teams to actually do the update for something that they don’t need just yet. The Client team approaches this challenge by providing game teams with a model where the update can be done in the easiest or most convenient way possible. They also prepare release notes for every update, in which they document all the changes. Ultimately, it’s important to motivate the teams to update the shared components regularly, because this can later on cause big issues during critical releases. If the game teams are too far behind with their updates, it is not going to be easy for them to apply the fixes. They can find themselves in a painful spot and finding that right balance can be really difficult!

“If we want to share so much code, how do we solve these problems?” says Martin T.: “And if we share less, what implications does this have for the game teams? Maybe we shared too little before, maybe we share too much now and maybe the real sweet spot is somewhere in between? The fact is that we don’t know. Things change all the time and so we go with the flow and keep adapting.”

‘We care about code’

If you ask the Martins what their main motivator in their day-to-day is, they answer unanimously – they care about code and how we develop it. The entire Client team is aligned around this. Martin G. explains: “What motivates me is that we have an actual impact on the way we write code at Tactile. We can affect things in a positive way. We have the space to express and discuss our opinions and really move things forward.”

The Client team has the possibility to embrace change and try things out in order to find new ways of developing or experiment with new ways of interacting, which is a luxury that our game teams do not have. This is because they work on a completely different schedule, having to regularly deliver new features to our players. The Client team, on the other hand, has less strict deadlines and more time to fine tune things. They have the amazing opportunity and freedom to have conversations and (often heated 😂🔥) discussions about code architecture, code design and implementing something new. Because they service all the game at the same time, they are in the best position to do so. This brings a lot of dynamic to the team.

‘Thinking as one’

The team’s drive towards improving code quality led them to making big changes in how they work. One of the things that they introduced years ago was unit testing, which drove an effort to improve the code by actually verifying that it works as expected through automated testing. The natural continuation of that was moving towards pair programming. Martin T. elaborates: “We have a review procedure here. Every time you make a change to the shared codebase, you need someone to approve it. It’s a very asynchronous flow, which requires a lot of back and forth between two people. Pair programming gives us the option to go around that by having pre-reviewed code, because you always have two people working on it at the same time.”

Martin G. adds to that: “Basically, we create code that is going to be used by others. We need to create something that is readable and that other developers can understand well. This is why we read a lot of books on clean code and clean architecture. But the thing is, when you work in isolation, you can be biased by your own experience. Pair programming helps to create a connection with other team members, which helps to mitigate those biases, it prevents siloing and brings a bunch of different experiences to the table. Whether it’s from a junior or a senior developer, everyone can learn from each other and ultimately, this raises the ability of the entire team.”

---

* For the curious, out of 926.807 total lines of Lily’s Garden C# code, 723.518 lines are shared, which represents 79%! The shared code is also divided in two distinct domains; packages and modules (packages is mostly what the Core team is responsible for).

Lines of code in packages: 309.155
Lines of code in modules: 423.363
Lines of code in non-shared (the “Code” folder): 194.289

We all know how frustrating it can be to kick off your career as a junior, trying to enter the gaming industry. This is why we wanted to share some industry insights, and are happy to introduce two of our fantastic Level Designers, Vilma Lahtonen and Ecem Kıvılcım. They told us about the steps they took to successfully get into level design. Beyond that, we discussed the challenges they overcame (face your fears!), tips on how to increase your relevance as a candidate applying for level design jobs (make use of the endless amount of online knowledge!), and crucial skills to do great work (hubris kills team collaboration!).

Game development is a complex process. In 2023 alone we created more than 100.000 game builds, which is a huge number! Supporting our games team in the game development process are the brilliant minds of our Core team. Developers responsible for constructing and maintaining our internal infrastructure –  all the tools, platforms and systems, entirely built in-house and customized to our specific needs, which enable us to do live game operations.

One of these tools is the Build Server, our internal game building pipeline, which has been for the past 6 months under close inspection from Juan and Vlad (our Backend Engineers extraordinaire). We sat down with them for a chat about how we build our games, why we bother developing our own tools and to discover what improvements the team has been working on this year and why.

How do we build our games?

Our games are built using the Unity engine and every time there are updates to the game projects, such as asset bundles, new product features or in-game events, we create a new game build. For a game developer, this process takes a lot of time and resources from their machine. Despite the fact that today developers’ machines are more efficient and powerful, it can still take hours to compile all the assets into a new build. This is why, several years ago, we decided to build a collection of tools, which enable us to build games faster, in an automated and more efficient way.

Whilst there are companies that provide this service, having our own personalised, internal build pipeline offers us the flexibility to build in a way that suits our specific needs. It’s not uncommon that companies will have a dedicated Build Pipeline Engineer, or use an out-of-the-box solution. It is a necessity for making games. Ultimately, the product that the gaming company delivers is something that gets installed on some sort of a device (whether that’s a mobile phone, console or PC). Creating these game builds is the part that got automated with our Build Server tools.

Think of it like this – there’s two main components of the Build pipeline, the Controller and the Agents. The Controller allows game programmers, level designers and other users to order a build of whatever they are working on within a specific game. They can do so through our internal build dashboard. The build request then goes to a Build Agent, which starts going through a sequence of steps, resulting in an installable build. Besides these on-demand builds, we also have a lot of scheduled building, which normally runs overnight or on a specific schedule. All the ordered and scheduled builds, build agents and their respective statuses can be seen on the build dashboard.

Vlad elaborates: “The internal build pipeline gives our teams whatever flexibility and customisation they might need – whether someone wants to order a build for a game for a specific branch or a specific revision, if they want to run tests and validations, or not, whether they want to run it on Monday at 2pm, or every Tuesday overnight … You get the drill. The Build Server will notify them when the build is done and they can track everything relating to the building process on a user-friendly and consistently running interface.” These tools therefore give us a complete overview and context into each build, and save us a bunch of time that was previously spent looking into individual problems.

‘If you want it done right, do it yourself.’

This is where building our own tools gives us the advantage over using commercial tools. It enables us to make the game building process super flexible and to bend it to our specific needs. We make a lot of custom integrations in our builds and we have made it very easy for us to report on the things happening in the build pipeline.

The Build Agent is only one of the many moving parts of our build tools. However, it is THE part which actually executes ‘the final recipe’ composed for every game. We actually have several Build Agents (148 to be precise), which are all running on ca. 30 Mac Minis. Juan shares a fun fact about this: “We use Mac Minis because we compile new game builds using Xcode, which requires us to use the MacOS. Having our own internal build tools enables us to build things in Xcode for iOS, which is not something that many commercial tools offered nearly a decade ago when we started looking for an appropriate solution.”

Individual agents are set-up in the same way, so that we can reliably know that if the set-up worked for one project, it is also going to work for the others. Another good feature of our build pipeline is that it enables us to upload specific game builds to an app store, instead of our Producers having to do it manually. As Vlad explains: “It’s basically a big collection of tools that contribute to the overall ease of development, in particular by reducing the time that people spend doing repetitive tasks. There’s a lot of things that seem incremental in themselves, but end up adding to a lot.”

Changing things for the better

Juan and Vlad recently rewrote the Build Agents code from an older version of Python into TypeScript. The old code was difficult to maintain, so everything that needed to be added, fixed or modified to the codebase took a lot of time and effort from our development team. It was basically like constantly readjusting a big Jenga tower.

We therefore took the decision to re-do the Build Agents in a language that was comfortable for all our developers. We also took this opportunity to test our new coding patterns and practices. This was a small project to start with (compared to others), but complex enough that it was suitable for trying this out.

The new coding style is centred around dependency injection (DI). DI is a design pattern which promotes loose coupling between components by externalising the dependencies of a class. Instead of a class creating its own dependencies, they are provided, or injected, from the outside.

Applying DI therefore allows us to have total control over the business logic behind the code. To give an example, we have some business logic that says when the build is finished, we need to notify the user. In order to do that, we call a third party library. However, we do not have control over the third party library and have no way of testing that it actually does what we expect it to. But now, with DI, we are able to abstract that library into a third component which we have complete control over. Juan elaborates: “We now feel very comfortable making structural and behavioural changes in our code, because we are more confident that it will behave as it should. We have a big testing infrastructure, more control and a complete overview of its behaviour.”

Vlad goes on to illustrate this with another example: “There are many ways to boil an egg, right? You can boil it in hot water, you can boil it in a microwave, or you can set the house on fire and eventually, this will also make the egg boil.” 🤣

But if you’d rather avoid burning the house down, here’s another metaphor provided by the all-knowing ChatGPT: Imagine you’re building a house. In traditional programming, each room (or class) in your house knows how to make its own furniture (dependencies). With DI, it’s like hiring an interior decorator. You tell the decorator what furniture each room needs, and they bring it in. This way, if you want to change the furniture later, you don’t have to rebuild the whole room; the decorator can just swap it out.

So, applying DI is like having someone else handle the connections between different parts of your program, making it easier to change or upgrade things without messing up the entire system (and burning the house down).

The importance of continuously reiterating on our coding style

Within our team, we always strive towards making our code more maintainable and have an even better quality of life. It’s our long-term vision, and using DI is just one of the target conditions we want to reach on the way there. It is difficult to code consistently at scale in a big team, and to stay cohesive and coherent together. This is why it is extremely useful to have something that can be modified by all of our developers in a comfortable fashion. No matter your role, or your location, you can change something in the codebase and get a fast response into whether this is breaking something or not.

“This is why we need to keep reiterating on our coding style – to keep up to date with the standards and good practices,” says Juan, “As long as technology, languages, customer needs and the gaming market evolve, our style will have to follow. We need to be more agile and quicker to respond to things. If we keep doing things the same way, then we will not be able to stay competitive.”