InnoGames have redefined the role of QA within their organisation – with their new approach facilitating an evolution from Quality Assurance to Quality Assistance.

The video games industry is a faced paced sector, especially for free- to-play products which adopt a Games-as-a-Service approach to quench the desires of its content hungry users.

Trying to keep on top of the software quality aspect is a monumentous task utilising the QA approach of yonder.

Here at InnoGames we decided, after years of experimentation, to focus on the ‘quality assistance’ approach and empower the whole development team to own the quality topic.

Follow us on a journey from the past – to what the new approach means to us now – as well as taking a peek at how our approach impacted our largest product. InnoGames is a free-to-play browser and mobile games developer from Germany, which was founded in 2007 just outside of Hamburg.

We’re currently focused on developing mobile products utilising the Unity Game Engine. We have around 400 employees, of which 12 are dedicated to QA.

However, before we dive into where we are now and what we’re doing, we should first take a trip down memory lane to see how we got to this point.

An evolving strategy

Over the years, like many companies within our sector, the QA strategy and approach has slowly evolved with the times. The original methods relied heavily on utilising an external QA partner to provide the bulk of the QA bandwidth to our products, which had a purely black box testing focus.

The aim was to provide as many eyes (and hands) as possible to try and cover all of the various edge cases and features.

This approach worked for many years but was a process that couldn’t really be scaled, both from a business aspect and a purely organisational one. Back then we were releasing at a much slower rate and the complexity of our products was still manageable.

In 2012: the company slowly built up an in-house centralised QA team, moving away from the heavy reliance on an external partner, this team had the focus of black box testing across our product portfolio and would be able to move much more fluidly than the previous approach.

This presented a lot of varied challenges, one being how to scale the team when different product teams have requests on the same day!

We brought the knowledge in-house but we still had a scaling issue and were still approaching everything from a purely manual perspective.

In 2013: the company made a strategic decision to decentralise the quality assurance division and move the personnel to be embedded on the product teams to improve collaboration and ownership.

In 2014: it was decided that the QA staff embedded on the product teams would report into the technical product lead of the product that they were on.

This was in line with the whole organisation’s shift to a studio structure and moving away from a centralised matrix management.

The first foray into test automation also began in this year, with a proof of concept using the image recognition system called Sikuli.

In 2015: it was decided that the Sikuli approach was not a stable or maintainable approach and a research project was established to look into alternative solutions for test automation that would be compatible with the technology being utilised in the company at the time (Flash, Cocos2DX & Unity).

In 2016: the decision was made to approach QA with a whole new mind- set, this was inspired by Atlassian and their many talks and posts about quality assistance.

The short of it is that with quality assistance, everyone who contributes towards the development of the product is empowered and responsible for owning the quality topic.

This, in essence, means that QA’s job is to provide the team with the tools and knowledge to be able to handle the new responsibility, by becoming coaches and mentors to each and every team member – no longer being at the end of the pipe for development.

Coupled with a test automation framework which can utilise Java or C# for the test suites, this makes it easier for developers to contribute to the continued success and coverage of our test automation.

To make sure that the QAs do not become siloed on their respective products, we formed the ‘QA Community’, which meets on a weekly basis to share and exchange knowledge.

We believe this is a crucial element as we do not want to reinvent the wheel every time a specific topic appears.

Culture development & mission statement

Something else that we began to notice from taking this approach was that it inspired the QA members to organically help each other out, which in turn created a very strong culture that everyone has become very proud of.

We even created a mission statement for quality assistance to really embody the new approach and to help communicate QA’s new role within the organisation.

The mission statement reads:

• The WHOLE TEAM is responsible for the quality of the product
• Every team MEMBER who CONTRIBUTES to the product is responsible for the entire FEATURE development life cycle
• QA is responsible for ASSISTING the whole team in PROVIDING high quality.

So, where are we now?

We’re still slowly rolling the approach out team-by-team. All of our live products are following the strategy and we’re installing this success and learning into our new products.

You can imagine that such a shift in strategy takes time, as a lot of people are used to just throwing it over the fence to QA to handle and now this is no longer a valid approach!

But, we’re really happy with the progress and it has been a big surprise that the developers also now really enjoy the new empowerment and responsibility.

We now have the QA approach and test automation being added to the company’s tech strategy, which all products have to conform to, which is a big achievement and demonstrates that our management see the benefits of the approach and believe in it.

With test automation, we’re still driving the technology internally and want to eventually open-source our approach as we feel a lot of other companies would benefit from seeing our method.

There is still no timeline for this, but it’s a topic which is always on the table.

We also see a lot of additional development opportunities to improve our test automation tool as well as research things such as machine learning to enhance its functionality.

It is still early days down this new path and, as Atlassian say, they have not finished with their migration to this new approach some 10 years on.

We still have some way to go before it’s truly embedded into the company culture.

We didn’t stop there though, and are very active in sharing our knowledge and experiences with a number of companies in the free-to-play space.

We’re very open with the ins-and-outs of our approach and we believe this is the next evolution in Games QA and want to help shape its vision.

The new QA in action

Now, let’s have a closer look at how this new approach had an impact on our most successful product. Forge of Empires is a cross-platform city builder. It is one of the most played and successful titles in the strategy genre worldwide.

Starting with a small city in the Stone Age, you lead it throughout different eras, across the Middle Ages and Post Modern Era, all the way to the Virtual Future and beyond.

Battle friends and foes, in a single player campaign or with your guild mates in the Guild Expeditions, anywhere at any time.

Originally developed as a browser game based on Flash, the game has grown in complexity over the years – not only by expanding the platforms with a move to mobile, but also new eras and features have been steadily added since the game was first released.

While trying to keep up with the developments, and with the mobile platform and the browsers on their way to abandoning Flash, we were faced with further technical challenges, which led to QA increasingly becoming the project bottleneck.

Although QA was already part of the development process, reviewing design concepts and contributing to groomings and plannings, the QA process still was more like a mini-waterfall with stories going through a designated testing phase.

This led to defects being found too late, resulting in delayed mobile submissions and even re-submissions which a few times put events and feature activations on risk.

To give some insights, at its peak we had an overall bug count of up to 350 known issues. Although most of them were of a minor nature, the sheer amount could easily tarnish the perception of the quality of the game.

With this status quo in mind, we have started to implement quality assistance on Forge of Empires, which moves the quality more into the focus of the team as a whole.

The new approach to QA is not just a new QA process, it is a mindset involving the whole team which eventually enables the team to own the quality topic.

A change in the mindset and culture of a team isn’t easy, of course, and it takes time.

Within quality assistance everyone who contributes to a feature is responsible for the entire feature development lifecycle, which starts with the design of a feature till the feature sees the light of day.

Typically, QA team members know the ins and outs of the whole system better than anyone else on the team! Thus their input in grooming and planning helps the team to understand the features and to estimate them.

These discussions can identify functionality some team members may not have considered, but can also provide additional input on the overall system knowledge, particularly around inter-dependencies and the potential impact on other parts of the system.

The estimations do not include the development effort only, but also the testing effort, which usually gets estimated higher by QA as developers often focus on the happy path!

For example, a small change in the e-source handling may just involve two to three story points from a developer’s perspective, however when looking at it from a QA perspective this might be a risky change and can have an effect throughout the whole game as resources are involved with almost every feature and may bump up the story points to 13.

While the whole team is invited to give feedback on features at any time, QA is responsible for assisting the whole team in providing high quality. The best defects are faults which don’t end up in the code.

To prevent them we use testing notes which are a set of hints, written before the development of a user story starts, about the type of bugs and risks which might be expected to be found in the story once completed.

They are a guide and by no means a checklist, meant to help the developers during implementation to avoid introducing bugs in the first place.

Moving from demo to done

The testing notes also serve as the base for the QA demo which happens once a developer is happy with the implemented user story.

A QA demo is a discussion between equals, and not a test that the developer or story can ‘pass’ or ‘fail’. This allows the QA engineer to understand the implementation details of the story, down to the code level.

They receive information regarding decisions taken surrounding risks, and what testing they still intend to carry out before the changes are to be merged to master.

At the end of the demo, the QA engineer and developer will make a joint decision on what should happen to the story next. If both parties are confident about its quality, the story is moved directly to ‘done’.

A story being marked as ‘done’ is a big deal – it means that the feature will be deployed in a release to production with no further manual testing or review. Note that anyone can slip in to the role of the QA engineer for the demo and take part in the discussion (this can be another developer, game designer, or artist).

This can, of course, depend on the story and experience in the team – as other team members may be better suited for different topics, like an art-heavy story would certainly benefit from an artist joining the demo.

Not all issues can be prevented after all, especially with features being split into multiple user stories, and allowing for integration issues to evolve. Hence, we organise ‘testing dojos’ with the team, particularly around bigger feature sets.

These serve two purposes; the team finds bugs and gathers feedback about potential improvements, but also gains confidence in the quality of the feature, as well as the experience it provides the user, ensuring it will be something fun to play once released.

Importance of test automation

Automation plays a crucial part in the process as it serves as a safety net for the development team to be able to make mistakes at times, because after all we are only human and are not infallible!

We’re constantly increasing the stability, coverage, and general efficiency of our UI automation and we currently have over 500 tests per platform executed on a nightly basis.

With the strength of our UI automation, we were not only able to reduce the number of produced bugs, but also to completely eradicate our existing bug backlog within a year – now we are staying consistently below 20 active bugs overall.

Along with other process improvements and automations, such as our deployment flow, it enabled the team to confidently release the latest version in development to our beta server on a daily basis.

Conclusion

That’s our brief overview of where we’ve come from and what we’re doing in terms of QA here at InnoGames.

We’re always learning from our challenges and everyday we’re striving for the global ownership of quality within our organisation.

We truly believe the next evolution of QA is that everyone involved is upheld to a high-quality standard and is empowered from all levels to own it.

Jana Gierloff, senior QA engineer, InnoGames & Joseph Hill, program manager, InnoGames