vid or screenshot here

Dynamic Music Composition In Video Games

Adaptive audio in video games impacts the player's experience significantly. This is notably seen when the player encounters a tense and suspenseful part of the game. The player's experience is significantly improved when using adaptive audio compared to using linear audio.

Our research addresses the question: is it possible to dynamically compose music for a video game in real-time? A critical area of video game development which is often neglected is sound design and music, although it is one of the most important elements when it comes to user experience in games.

By algorithmically composing music in real time, we hope to enhance the player’s experience and the replayability of our game. We were inspired by aleatoric music, where chance events and the performer's (in our case, the player's) decisions determine some elements of the composition.

Makings of The Project

Skills

  • Coding
    • GDScript
    • Pure Data visual scripting language
  • Ability to look up information, research, and create solutions to unknown problems.
  • Performing QA during development, including regression testing
  • Working in a Team environment
  • Communicating and collaborating on different aspects of the projects

Technologies

  • Godot Game Engine
  • Pure Data
  • Git for version control
  • Github Projects for issue tracking
  • Google Drive & Sheets for documentation
  • Discord for Informal and Formal Communication across Team Members

Schematics

1 / 4
What the Ballbot enemy looks like in Node and animation view.
What the Ballbot enemy looks like in Node and animation view.
2 / 4
All the folders involved in getting enemies up and running
                  with the game
All the folders involved in getting enemies up and running with the game
3 / 4
The Schematic for the Game Engine
The Schematic for the Game Engine.
4 / 4
The Schematic for the Music Engine
The Schematic for the Music Engine.

Meet the Game Engine

"The Brains"

Keeps track of every little thing and constantly communicates with the Music Engine

Major Features

  • Contains a map built from hand using sprites
  • Enemies
    • Ability to track player and attack
    • All Have own sets of animations
    • Featuring four different enemies with different behaviors
  • Boss
    • Massive and powerful enemy
    • Capable of shooting several fireballs at the player
    • Does lots of AOE damage
  • Demo Video

Say Hello to the Music Engine

"The Composer"

Dynamically reacts to the Game Engine's input and generates music using different components.

Areas 1 and 3
Areas 1 and 3 highlighted.

Components

Finite State Machines

Main Events FSM Minor Events FSM

In Godot, we created state machines to represent the player and the game's state. The player's health, number of enemies in the player's vicinity, the player's distance to enemies, and level progression are kept track of by state machines and signals. If a signal is received and certain parameters are met then a given machine will change state. The music controller updates the sequencer's values when a state changes.

Sequencer

Sequencer Part 1 Sequencer Part 2

The sequencer is built using Pure Data, a visual scripting language and real-time audio generation software. We modified Martin Brinkmann's "random percussion" patch to accept 23 parameters which control different parts of the sequencer: bpm set flag, bpm (beats per minute), master volume, instrument volumes, instrument note lengths, scale selection, pitch, swing percent, rhythm sequence and probabilities, loop density, loop length, and note probabilities. The are five "instruments," or percussive voices which are interconnected and outputted to one channel.

Connecting to Pure Data

Pure Data Log

We used a local TCP socket in order to transmit data from Godot to Pure Data. The FUDI protocol implemented in Pure Data allows us to receive packets from Godot in the patch. All 23 music parameters are sent over TCP connection in one packet, rather than in separate packets. This was done to avoid latency, ensuring that sudden changes in the game would affect the music immediately.

Music Controller

TCP Ready Area 1 Entered

The Music Controller is a script in Godot which drives the music composition. It establishes a connection on local port 4242 upon starting the game. It takes signals sent by the state machines and game events, changes the global music parameters in response, and sends the new "state" of the sequencer to the Pure Data patch using the networking protocol every time the state is changed.

Sound Effects

  • Tie the Sequencer to overall theming of the game.
  • Allows for a more integrated player experience.

Demo

Downloads

Windows
Mac

Project Details PDF

PDF project link