Algorithms, apes and improv: the new world of reactive game music

If you were born after 1985, you’ll know Rock Band, the 2007 game that challenged you to play along to your favourite songs with plastic guitars and drum controllers. One of the minds behind that play along rhythm game was developer and musician Matt Boch. With a background playing in bands and making electronic music, Boch is now an NYU professor of game design.

For Ape Out, a top-down action game for PC and Nintendo Switch, Matt has used his skills in music, audio production, programming and machine learning to create a game soundtrack that is both integral to the game and remarkably responsive to the player’s actions and playing style. That’s to say, the soundtrack is never the same twice.

Before working on Ape Out, Matt Boch spent over a decade working in a variety of roles at Harmonix Music Systems, the makers of Rock Band. In his time there, he had a hand in developing other play along rhythm games Dance Central and the critically acclaimed Fantasia: Music Evolved, which puts you in the role of an orchestral conductor. Both relied on the detection of the game player’s body movement using Microsoft’s Xbox Kinect technology.

Ape escape

From a gameplay perspective, Ape Out is remarkably simple. The aim is for your character – an enraged ape – to escape a labyrinth without getting killed. As you unceremoniously dispatch scores of faceless, gun-toting enemies, volleys of rapid drumbeats are generated. The more violent and frequent your altercations (and they get fairly bloody) the more frantic the music becomes.

“I think the role of music in Ape Out is fundamentally different from the role of music in [play along] rhythm games. Rhythm games ask you to pay attention to the music, whereas with Ape Out, the music is paying attention to you, the player.”

While many video-game soundtracks respond to the actions of the player – triggering new phrases or layers as the player succeeds in their mission, Ape Out takes this idea to a whole new level. The outcome is masterful and the gaming community agrees. Ape Out has already earned an impressive 9/10 from user reviews on gaming distribution platform Steam and a score of 83 on Metacritic, with much feedback praising the responsive soundtrack.

This reactivity was all entirely by design. “Ape Out is an attempt at making an interactive experience on a sonic level, allowing players to collaborate with a simulated drummer,” says Boch. “We’ve jokingly described it as rhythmic violence,” he continues, “but it’s all about supplying an engrossing, dynamic experience that can keep surprising you with every playthrough.”

Missing link

Indie game developer, Gabe Cuzzillo, who conceived the game, knew from the outset that to conjure this idea into reality, the music – and the tech driving it – needed to be something special.

He approached Boch early on in the game’s development process: “At that time, the score consisted of three drum solos ripped from YouTube and a cymbal sound that would trigger whenever you killed an enemy,” recalls Boch. Cuzillo’s brief was to come up with a musical score that kept the same percussive feel, but would be less repetitive, perhaps even procedural – that is, randomly generated, based on a set of rules.

Intrigued by the challenge of creating a soundtrack that puts the power of musical creation into the hands of the player through their gameplay, Boch began prototyping.

Opposable drums

“The first version of the soundtrack I developed was for a playable trailer for the game, which was released about two years ago,” Boch explains. “In that version, the soundtrack consisted of composed drum loops that repeated ad infinitum. We tried out some variations on that system where we had many individual drum loops and a system for picking them, but it became quickly obvious that it would be untenable to have enough drum loops to sound reactive without ballooning the game’s file size and load times.”

He then moved over to a sampler-like approach, trying to make a virtual drum instrument. As any producer will know, there’s no greater giveaway of a computer-generated (or poorly programmed) drum part than the machine-gunning sound of the same sample being rapidly repeated. That’s why Boch went to great lengths to make sure that natural variation was built in.

“Each of the kits in the game has seven different velocity levels and 12 round robins per velocity level, because we were really concerned about making the drums feel as live as possible. Eventually, we had MIDI hooked up to the [game’s] sampler, and we could play back linear MIDI, and that allowed us to fit a lot more content in,” he explains.

He began by composing and sourcing parts, but to get the amount of musical variation they wanted, Boch needed many more MIDI performances to work with. That’s when he turned to machine learning to generate MIDI clips from his existing performance data.

Machine music

Using a machine-learning technology called a variational autoencoder neural network – very similar to those that have been used to generate fake human faces – Boch was able to create dozens of hours of drum playing. He claims players only hear about one-hundredth of the total patterns in any given playthrough of the game.

Once all the MIDI patterns had been created and cleaned up, the final piece of the puzzle fell into place when Boch found a way of organising and moving between the MIDI clips in a way that was tied to gameplay.

“Initially, we were moving between these clips randomly, but I developed an algorithm to allow macro control of pattern selection based on parameters like intensity, density, and similarity. Now, every MIDI pattern that makes up the score knows what other patterns are most or least like it, and this allows the drums to change, directly in response to player action.

“We are also able to change the velocity of hits based on player action, so even if we can’t cue up the new MIDI pattern for a second or so, we can make the remaining hits on the current pattern a lot harder, so it still feels responsive.”

The result? A game-playing experience that feels like you’re being accompanied by a live drummer who is improvising their performance to share in your victories and your losses. The exact ways in which the music reacts to your game-playing style are many.

“When you get shot, the drummer stops for a couple of beats and will switch to playing something different from what they were playing before. If you stay in high-intensity situations, the drummer will keep going on a similar motif until either the level ends, you get shot, or the intensity level of your gameplay decreases,” explains Boch.

But it’s not just the intensity of gameplay that affects the soundtrack. “The cymbal that crashes when you kill an enemy is dependent on their position on screen, spatially mapped on to a drum kit from the drummer’s perspective – so for example, killing an enemy on the bottom left side of the screen triggers an open hi-hat,” says the professor.

Immersion therapy

To the casual game player and the production expert alike, it’s clear that every element of the game’s sound has been thought through. “With the sound design, I was aiming for a very realistic take on combat and player action sounds to lend some more weight to the highly abstract visual presentation,” says Boch.

The game features realistic-sounding automatic rifles, wielded by enemies painted into scenes that have all the visual boldness of a 1950s Saul Bass title sequence. Visually, the cymbal crashes are accompanied by a screen shake, too, making every attack more involving for the player. Players also report that the music sounds ‘better’ when you’re playing well. This, too, is by design.

“We endeavored to make the player sound stylish anytime they were doing well in the game. It’s totally intentional that the most bombastic version of the soundtrack is the one that a skilled player would hear as they ape their way out.”

It’s not just the intensity or density of the MIDI performances that change when you’re skillfully smashing your way through the game. Boch researched music psychology and computer-music theory while designing the soundtrack, which fed into sound-design decisions such as humanisation bias changing based on gameplay intensity. Producers will know the term ‘humanisation’ from their recording software to mean adding random variation to quantised MIDI in order that it doesn’t sound too robotic. In the context of the game, it means something slightly different.

“The drummer anticipates the beat in periods of high intensity and will lean back into the pocket at low intensity. This unsettles the player and adds to the intensity of combat,” say Boch. Anticipating or ‘pushing’ the beat imbues a performance with tension. This tallies with players’ reports of feeling energised by the music when they’re succeeding, to the point where some are filled with such conviction of their own skill that they take greater risks within the game.

Next generation

Creating a soundtrack that’s totally different each time you play is a tall order, even if you know the layout of every level and where the enemies are going to be coming from. However, to keep players coming back time and again, the levels are also algorithmically generated. This means the labyrinth and, consequently, the spawning locations of enemies are different every time the game is played.

“The level design was left entirely up to [game designer] Gabe; my goal was to build a system that would aesthetically match any kind of combat encounter the game could procedurally throw at you.”

Apart from the level introductions, there are no scripted moments in the game and therefore no fixed musical score. “The drummer is always just reacting to whatever gameplay scenario is happening on screen. As such, everyone’s Ape Out soundtrack is unique to their playthrough.”

Code monkey

While some of the tools that Matt used to make the score would be entirely at home in any project studio – he did a lot of his prototyping and composition in Ableton using the Push 2 – he also had to get under the hood with coding.

“I used a lot of Python for prepping and cleaning up MIDI data, and TensorFlow [an open-source machine-learning library] for the neural network that generated drum loops. I used Reaper extensively to make a lot of the more repetitive sound-design tasks more automated, allowing for things like faster creation of new drum kits: it’s really powerful to have a tool you can script,” Boch says.

The sampler that handles music playback in the game is custom-built in the C# programming language, drawing some basic functionality from assets including Audio Helm, a live audio synthesizer, sequencer and sampler for the Unity game engine and a toolkit called SECTR.

Sound choices

When it came to creating drum sounds, Boch had no intention of reinventing the wheel: “Given the ambition of everything else involved with the sound design, I opted not to record any new sounds for this game. Instead, I drew from a large collection of sound-effects libraries, recordings I did in the past, and VSTs to create the dozen-or-so drum kits in the game,” he explains.

“In developing the drum sounds, the most important thing at the start is to give people the feeling that this was a real drummer and a real drumset. I opted to use mostly a highly compressed room mic of a drum kit for all of the first four levels of the game. The garage-y, live feel of the recording helps to sell the idea that it was a real drummer.”

The music references in Ape Out are baked into the game; each level is presented as a track on an album. There are four ‘albums’ each with an A and B side, and united by a general location. These locations comprise an office block, a container ship and a research laboratory and even a zoo. Another part of the challenge was creating different sonic and musical worlds to accompany these different spaces.

As Boch explains: “Many of the stylistic choices in the first album are based on drum solo choices that Gabe had made. But the sonic palettes of the subsequent albums were more based on my interpretation of the theme of the level.”

He continues: “Gabe and I discussed the album concepts and what kinds of drum sounds and patterns we might put along with them. For each album, we also identified exemplar sound samples and expanded the sonic palette to get a sense of how they would feel. There were a tonne of different iterations of the music system that were kind of happening in parallel to the rest of the game’s development.

“So, for example, with the office [level], we went with bucket drumming to give it a metropolitan feel, and then as you get to the ‘under construction’ part of the building, we transition to stomps and claps with sounds of power tools and hammers banging mixed in. The beats in the office [level] are also more rock and funk influenced – they’re still improvisational, but they’re not jazz.”

MB tips

It remains to be seen whether Ape Out will open up a new way of thinking about generative game soundtracks. However, it certainly achieves its aims with flair and makes for a riveting, adrenaline-fuelled game-playing experience where your desire to play well is driven just as much by making the drums sound great as completing the levels.

Does Matt have any pearls of wisdom for those wanting to follow in his footsteps? His advice is simple and it’s at the heart of Ape Out: “Adopt interesting constraints.”

Learn more about Ape Out here.