Get Info: #audio

Enhanced sfxr for Love2D

by Matt Sephton — Tue, 11 Mar 2025 18:23:00 +0000

I’ve made a bunch of usability and quality of life changes to the sfxr.lua demo app.

fix: getDirectoryItems (from a PR)
fix: off by one sample length error (from a PR)
fix: space bar was not playing the sound
fix: selected wave form was being ignored (sounds were always square wave)
fix: loaded wave form not updating the interface (but sound was playing correctly)
add: clone method added to sfxr.lua
add: history form with list of previous sounds, save current, and undo/redo
add: “play on changes” so that any time you adjust a sound it plays automatically
add: keyboard navigation in file picker (A–Z: jump to files, Enter: choose, Esc: close picker)
add: window title reflects the most recent loaded/saved file
add: window title and filename reflects last operation
add: file selector shows data directory path
add: unified save directory across platforms

You can get it at: gingerbeardman.itch.io/enhanced-sfxr-for-love2d

At some point I’ll fork the project on GitHub and file some PRs with my changes, but the repo for it seems very dead so I’ll wait until I hear back from the owner. Source code is at the above link.

Dynamic music and sound techniques for video games

by Matt Sephton — Sat, 09 Dec 2023 01:22:00 +0000

The only aspect of game development I’ve not attempted myself is the music. I mostly use royalty free music of Japanese origin (just because I dig their vibe, man) as in the case of Sparrow Solitaire or Fore! Track or in rare cases I pay friends (like the amazing Jamie Hamshere) to write music specifically for a game as in the case of YOYOZO. Maybe one day that will change, but until then I’m enjoying gaining more understanding and control of the music in my games. Whilst I develop games for Playdate these techniques are general enough to apply anywhere.

The main way I make the music into more than a static track is to apply a dynamic, reactive, or adaptive effect in one way or another. In this blog post I’ll go into how I’ve achieved this. Please note this is by no means an exhaustive list, rather it’s just the ones I have personally used.

Dynamic BPM

I use this method in YOYOZO because it uses “chip tune” music data representing songs composed by my friend Jamie Hamshere using Playdate Pulp. A playback engine for this data, written by Pulp creator Shaun Inman, works beautifully when integrated into games written using Lua and the Playdate SDK. I added hook to allow me to set the BPM at any point to any value. The end result is that the BPM of the music scales from 130 to 135 as your score increases. As you improve at the game you’ll notice the music speed up ever so slightly along with an increase in tension and anxiety.

Of course, it’s possible to do this in a pre-recorded song stored as a digital music file, but it’s much more difficult for that to respond to the what the player does in the game. An example that takes an interesting approach to this is the track “Sunny Day” from the game Vib Ribbon, and indeed the rest of its soundtrack, where tempo changes over the duration of each song.

Infinite Variations

Another technique I use in YOYOZO, again made possible because I can modify the music data and playback parameters in real time. With this one I cycle the values for the instrument voices pseudo-randomly so that the track plays once as it was programmed and then morphs slightly for each subsequent playback. The track is quite minimal and repetitive in Steve Reich, Philip Glass or John Adams sort of way, so there are automated variations wandering around the original arrangement work really well. Perhaps the ultimate implementation of this approach is Wii Play’s Tanks game.

For sound effects, I vary the playback sample rate to change the pitch of sound effects. This prevents the same sound effect becoming monotonous. Two examples might be Lara Croft in the first Tomb Raider game, groaning the same way every time she climbs up a platform compared with the rich variety of sounds when Mario walks on different surfaces.

Blending/Fading/Balance

Another idea I had was to fade or blend two tracks as the player makes progress in the game. But how to find two tracks that can be cross-faded in a way that always makes sense? Of course you can have them composed, but what about in music that already exists? If only there was an easy way to find such tracks!

There is: stereo pairs! You’d be surprised at how different the left and right channels can sound whilst obviously being the same tune. Of course this means that are output audio will be mono but for me on Playdate that’s just fine. I use this method in Fore! Track.

The idea is to adjust the balance of the two parts of the audio, at once point you’re playing just the left audio across both outputs, then you adjust the balance to play a mix of both, at the other end of the scale you’d be playing just the right audio. Unfortunately the Playdate SDK currently has no API to easily adjust balance, so I had to program a method myself. First, I convert to the destination format which is for me ADPCM using adpcm-xq. Once the files are in this format I split the stereo pair into two files, one for the left channel and one for the right channel. Converting to the destination format before splitting ensures that the two are exactly the same length in terms of samples/bytes.

In the game I fade between the two as the score/chain increases, which has the effect of subtly changing the instrumentation of the tune. It’s one of those things that most people wouldn’t notice, but that once you know about it you can’t miss it. Sadly, I don’t have an easy way to demo this in a video or sound file.

For sound effects, you might consider panning to increase immersion and guide the players visual focus through use of audio. In YOYOZO I pan certain sounds relative to the location of the ball, certain other sounds relative to the location of the player, and there are global sound effects that are not panned at all.

Progressive Loops

Digital audio is a different beast. I always try to find a time that fits the game, going so far as to audition many hundreds of tracks and creating playlist of songs far ahead of ever making a game or even having and idea for a game. I try to find tracks that will loop well and not get annoying, which is easier said than done. If I can’t find a track that loops well, there’s another way.

You can use PyMusicLooper to analyse a digital audio track and spit out information about ranges that loop nicely, along with a percentage indicating how good it considers the loop. In other words you can identify and extract a loop from digital audio files that sound like they could loop. Of course, can’t identify loops in tracks that aren’t repetitive or consistent in their structure. You might get PyMusicLooper to split the file into into three sections (intro, loop, outro) or just export the loop information as a text file to use in your game. Which I choose depends on how much of the file I want to use.

For an example, in my game Super ICARUS I’m using a file that gives the vibe I wanted in the game and sounded like it contained some loops even though it was not provided as a looping song. PyMusicLooper reported that it contains tens of possible loops of varying quality.

loop	start	end	duration	match
0	3.437	30.755	27.318	94.87%
1	0.023	27.341	27.318	94.79%
2	7.279	34.598	27.319	94.63%
3	6.850	34.168	27.318	93.95%
4	8.127	35.445	27.318	93.92%
5	22.221	52.953	30.732	93.80%
6	25.635	56.366	30.731	93.23%
7	11.970	39.288	27.318	93.17%
8	6.850	35.875	29.025	92.13%
9	6.850	54.660	47.810	91.98%
10	20.515	52.953	32.438	91.54%
11	10.263	37.581	27.318	91.26%
12	22.221	54.660	32.439	91.11%
13	23.928	68.325	44.397	90.92%
14	39.300	70.031	30.731	90.82%
15	12.399	70.449	58.050	90.78%

My goal was to find three loops of increasing length and with a high percentage loop quality. After some experimentation and listening, I decided on loops 0, 9, and 15 (table only shows the top 15 loops from this track, even though their percentage loop match are not 100% they still sound like good loops, so selecting these loops was a case of finding three of suitable length and content. PyMusicLooper will let you audition the loops directly, so there’s no need to use an audio editor.

Using the Playdate SDK I can do setRange() on the audio track to change the playback range and the music will loop between those new points when the playhead reaches the end of the range. For this reason, this method does not provide immediate results so is better used to signify a large change in progress as the delay until the change is noticed will be an unknown amount of time. But when the change does kick in it’s a really nice surprise!

The final result sees the game start by playing loop 1 (synth and drums) and then as the player gets makes some good progress I switch to loop 2 (synth, drums, guitar licks), and finally as they pass a certain threshold I switch to loop 3 (synth, drums, guitar licks into guitar solo). This provides music that sounds very dynamic with little effort. You could even drop back to the shorter loops if the player lost a life, missed a target, and so on. Again, there’s no real way of me demoing this as it’s something that will become apparent through play, and the final result is just one long dynamic song!

For sound effects I use the same approach as above. As an example, in Fore! Track there is a clapping sound effect after the player gets the ball in a hole. This is a long sound effect but I play three increasingly long sections of it as the player’s chain increases (number of successive holes-in-one). It starts off as a short clap, increases to a longer more enthusiastic clap, and finally it begins with a whoop and continues to enthusiastic clap. I have a separate sound effect for the end game cheer that plays over the top of the full clap, resulting in a raucous end of game celebration.

Quantised Sounds

In Super ICARUS created certain game event sounds from small sections of the music track. I then adjust playback rate/speed/pitch. The result is that the sounds appear to be quantised or matched to the music. I’m not sure how to describe this phenomenon accurately in musical/technical terms.

I’d love to hear about other methods of achieving dynamic music and sound in video games. Feel free to reach out to me on social media!

Elsewhere

2023-11-22—Hacker News
2023-11-22—Tildes

Automatically classifying the content of sound files using ML

by Matt Sephton — Sun, 13 Aug 2023 17:01:00 +0000

Following on from yesterday’s extraction of old sound effects, I quickly realised I needed an easier way to search them as they came out of Director as unlabelled, numbered files. I can use QuickLook or a media player to quickly audition them, but how could I easily find the sample that contains the sound of running water or a horse trotting?

I wondered if there was a way of using Machine Learning (ML) to automatically categorise sounds. It seemed like something that should be possible, especially given the recent explosion in “AI” (really: ML) tools. I quickly found Google’s AudioSet, which sounded like the perfect dataset:

AudioSet consists of an expanding ontology of 632 audio event classes and a collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos. The ontology is specified as a hierarchical graph of event categories, covering a wide range of human and animal sounds, musical instruments and genres, and common everyday environmental sounds.

But the data set is only one half of the solution. You need to use the dataset to create a model and then run that model against your own data to get the required results. Thankfully, I found YAMNet:

YAMNet is a deep net that predicts ~521 audio event classes from the AudioSet-YouTube corpus it was trained on.

I guess YAMNet is tracking behind AudioSet in terms of total categories, but it is good enough for me. Here is a list of all the classes of sounds it can recognise.

Let’s go

I used the script described in this tutorial as a starting point. I’m not a regular python user but using pip to install tensorflow, along with any other missing imports, and after that it …just worked.

Getting your files in order

According to the documentation all sound files need to be at a sample rate of 16000Hz. After getting some calssification results of “Silence”, I realised they also need to be 16-bit resolution. So I ran a quick sox command to create compliant copies of all my sounds. I’ll delete these when I’m done. Notice how I decided to trim sounds to a maximum length of 3 seconds. This helps speed things up and most sounds can still be recognised with such a short starting section.

find . -iname "*.wav" -exec sox {} -c 1 -r 16000 -b 16 {}_16k.wav trim 0 00:03 \;

Optimisation

Running the classifier works at about real-time, a few seconds per sound, but I noticed that it was leaving a lot of my CPU unused. This struck me as a prime candidate for parallelisation, which is pretty easy on the command line. I used the parallel command to scale up the classification to use all 10-cores of the M1 Pro CPU in my 2021 MacBook Pro.

find . -iname "*.wav" -exec parallel python3 classify.py {} ::: {} \+

As I type my computer is making short order of the task, whilst remaining perfectly responsive, if a little warm. Final speed for me is one sound every ~0.85 seconds.

Python Script

Creating my SFX Library

Soundly is a sort of iTunes for sound effects. It’s an app that enables easy, automatic organisation of files, quick searching of metadata, painless playback/auditioning, non-destructive edits, and simple exporting of the final sounds. The free version allows a local library of 10,000 files which is more than enough for my usage. I’m not affiliated with them in any way, but they offer a free version and a 1-month free trial of their paid version.

As you add your local folder of files it allows you to import a (semicolon-separated) .csv file containing additional metadata. It’s here that I point it to the file that was generated by the classifier. The categories are imported as the description of the sound, and are able to be searched. Perfect!

Extracting sounds from Macromedia Director files

by Matt Sephton — Sat, 12 Aug 2023 19:42:00 +0000

For my latest “quick” Playdate project—a remaster of a ~1997 web game by Thoru Yamamoto—I decided to add sound effects. In order to keep it as authentic as possible I decided to use only sound effects created by Thoru Yamamoto that were used in his other productions.

The largest collection of sounds I could think of were those included in his Macromedia Director web experiments which include everything from short animations, through games and interactive toys, to abstract slideshows. The problem is, these Director files are tricky to deal with some 25 years later.

Halt and Catch Fire

A bit of reading and some help from Misty De Méo led me to ProjectorRays which can convert a protected .dcr file into an editable .dir file, and also allows saving of all the individual chunks that comprise each file. Think of it as one chunk for each piece of graphics, sound, and so on.

Once you’ve converted the .dcr to .dir you can open the file in, say, Macromedia Director 2004 and use CastRipperTool to export sounds and graphics and more. But it’s a very manual process and the whole setup is prone to crashing. Crucially, it won’t open some older Director movies so this wasn’t a good enough solution for me.

Chunks

I looked to the chunks that had been dumped by ProjectorRays and some quick experimentation showed that the ‘snd ’ chunks contained raw PCM audio date with a bespoke header. Loading these into something like ocenaudio wave editor was proof enough, but I would need to figure out some of the header to see if I could get the correct sample rate of each file.

With a bit of help bouncing ideas off hikari_no_yume in a bug report on one of their tools I figured out the location of the sample rate. Interestingly, the values were unexpected and not the usual values 11025, 22050, etc. It turns out that classic Apple Macintosh used some slightly different sample rate values: 11127, 22254, etc.

Using the xxd tool I dumped the relevant section of the headers of 1163 sounds from 105 .dcr files. The breakdown in sample rates found was as follows:

Rate (Hz)	Hex	Quantity Found
7,418	0x1cfa	1
11,127	0x2b77	616
22,050	0x5622	31
22,254	0x56ee	515

The lack of any unexpected sample rate values was further proof that this was the real deal.

Shell script

Armed with this information I decided to write a short shell script that would do the following:

extract the sample rate from the ‘snd ’ chunk
create a trimmed raw pcm file excluding the 78-byte header
use sox to add a new WAV header using the correct sample rate

This script allows me to batch process all the audio and that’s much more to my liking: it captures everything compared to CastRipperTool and it’s a lot quicker.

So, dump your chunks using ProjectorRays and then call my script like this:

find . -iname "*.bin" -exec ./bin2wav.sh {} \;

Notes:

I’m assuming 8-bit, mono, unsigned PCM data in the ‘snd ’ chunks and have not found anything else in them.
SoX (Sound eXchange, the Swiss Army knife of audio manipulation) requires the .raw extension for raw PCM audio data, it will refuse to process the files without it.

The biggest crime in pop music?

by Matt Sephton — Fri, 25 Nov 2022 00:00:00 +0000

The re-recorded version of “Ride on Time” (featuring an uncredited Heather Small on vocals) released in the UK on Deconstruction Records.

It was (re)made as the original samples couldn’t be cleared. The track’s success is what gave us M People, for better or worse. 😅 Sadly this version is the one you hear today on UK radio.

And here’s the original version of “Ride on Time”, released in Italy by the real BLACK BOX in 1989.

The original Italian version of “Ride on Time” can be found today because the samples of the Salsoul Records classic, Loleatta Holloway’s “Love Sensation”, were eventually (finally!) cleared in 2018!