nsynth

NSynth for Strudel

A part of the NSynth dataset formatted for use with the live coding language Strudel.

The data itself is hosted on Hugging Face here, because it'd be too big for GitHub.

Usage

samples('github:vvolhejn/nsynth')

// each n from 0 to 4 maps to a velocity, changing the timbre
$: n("0 1 2 3 4@4") 
  .note("<g1 f2 e3 d4>")
  .sound("bass_acoustic_000")
  .release(0.1) // otherwise, each sample plays for 3s

NSynth instruments are recorded at multiple MIDI velocities. On piano/keys, this corresponds to how hard the key was pressed, and gives us more expressive instruments. Notice how in the sample above, it's not just a volume change, but rather a different timbre.

Strudel also has .velocity() but it's normally just a gain multiplier, so above we have to use .n() to choose between the samples. We can register .nsynth() function to make things easier:

• it loads the sound
• sets .release() so that the sound doesn't ring out for the whole 4 seconds
• repurposes .velocity (if used) to select the MIDI velocity for the sample (see above) as well as compressing the velocity value itself so that the changes in gain aren't as strong

setCpm(160/4)
samples('github:vvolhejn/nsynth')

// Load an nsynth instrument with the proper sample based on velocity.
// Must be called after .velocity() to work properly
register('nsynth', (val, pat) => {
  return pat
    .sound(val)
    .release(0.1)
    .withValue(x=>Object.assign(x,
      {
        n: Math.floor((x.velocity || 0.7) * 4),
        velocity: 0.5 + (x.velocity || 0.7) * 0.5
      }
    ))
})

$: note("d4 a3 b3 g3 f#3")
  .velocity(saw.slow(4)) // call *before* .nsynth()
  .nsynth("string_acoustic_004")

List of available instruments

The instruments in italics are only sampled at one velocity in the dataset, so .n() has no effect.

bass_acoustic:

• bass_acoustic_000 - double bass

bass_electronic:

• bass_electronic_000 - picked, bassy
• bass_electronic_001 - picked, mids
• bass_electronic_002 - picked, bright
• bass_electronic_003 - picked, slightly muted
• bass_electronic_004 - picked, muted

bass_synthetic:

• bass_synthetic_000
• bass_synthetic_001
• bass_synthetic_002
• bass_synthetic_003
• bass_synthetic_004

brass_acoustic:

• brass_acoustic_000 - trumpet?
• brass_acoustic_001 - bright mid-range brass, trombone?
• brass_acoustic_002 - tuba
• brass_acoustic_003 - ?
• brass_acoustic_004 - trumpet?

flute_acoustic:

• flute_acoustic_000
• flute_acoustic_001
• flute_acoustic_003
• flute_acoustic_004
• flute_acoustic_005

flute_synthetic:

• flute_synthetic_001
• flute_synthetic_002
• flute_synthetic_004
• flute_synthetic_005
• flute_synthetic_006

guitar_acoustic:

• guitar_acoustic_000
• guitar_acoustic_001
• guitar_acoustic_002
• guitar_acoustic_003
• guitar_acoustic_004

guitar_electronic:

• guitar_electronic_000
• guitar_electronic_001
• guitar_electronic_002
• guitar_electronic_003
• guitar_electronic_004

guitar_synthetic:

• guitar_synthetic_000
• guitar_synthetic_001
• guitar_synthetic_002
• guitar_synthetic_003
• guitar_synthetic_004

keyboard_acoustic:

• keyboard_acoustic_000
• keyboard_acoustic_001
• keyboard_acoustic_002
• keyboard_acoustic_003
• keyboard_acoustic_005

keyboard_electronic:

• keyboard_electronic_000
• keyboard_electronic_004
• keyboard_electronic_005
• keyboard_electronic_006
• keyboard_electronic_007

keyboard_synthetic:

• keyboard_synthetic_001
• keyboard_synthetic_002
• keyboard_synthetic_003
• keyboard_synthetic_004
• keyboard_synthetic_005

mallet_acoustic:

• mallet_acoustic_000
• mallet_acoustic_001
• mallet_acoustic_002
• mallet_acoustic_003
• mallet_acoustic_004

mallet_electronic:

• mallet_electronic_000
• mallet_electronic_001
• mallet_electronic_002
• mallet_electronic_003
• mallet_electronic_004

mallet_synthetic:

• mallet_synthetic_000
• mallet_synthetic_001
• mallet_synthetic_002
• mallet_synthetic_003
• mallet_synthetic_004

organ_electronic:

• organ_electronic_000
• organ_electronic_002
• organ_electronic_003
• organ_electronic_004
• organ_electronic_005

reed_acoustic:

• reed_acoustic_000 - clarinet
• reed_acoustic_001 - horn?
• reed_acoustic_002 - horn?
• reed_acoustic_003 - saxophone
• reed_acoustic_004 - flute?

reed_synthetic:

• reed_synthetic_001

string_acoustic:

• string_acoustic_000 - string orchestra, good for low notes
• string_acoustic_001 - string orchestra tremolo
• string_acoustic_002 - string orchestra muted
• string_acoustic_003 - single violin
• string_acoustic_004 - string orchestra pizzicato

synth_lead_synthetic:

• synth_lead_synthetic_000
• synth_lead_synthetic_001
• synth_lead_synthetic_002
• synth_lead_synthetic_003
• synth_lead_synthetic_004

vocal_acoustic:

• vocal_acoustic_001 - men's choir "eeh"
• vocal_acoustic_002 - women's choir "aah"
• vocal_acoustic_003 - breathy women's choir
• vocal_acoustic_004 - breathy women's choir
• vocal_acoustic_005 - women's choir "eeh"

vocal_synthetic:

• vocal_synthetic_000
• vocal_synthetic_001
• vocal_synthetic_002
• vocal_synthetic_004
• vocal_synthetic_005

Development

First, download the train split of NSynth in the "json/wav" format from here. Watch out, it's 22 GB when compressed and 35 GB uncompressed! Valid and test sets can be useful for debugging.

Some of the instruments have different reported velocities even though the audio is exactly or very nearly the same. For these, we want to only list a single velocity for simplicity and clarity. Use:

uv run compute_velocity_matters.py nsynth-train/examples.json > velocity_matters_train.json

Afterwards, we can generate the strudel.json file:

uv run convert_to_strudel.py nsynth-train/examples.json \
    --comparison-json velocity_matters_train.json \
    --max-per-category 5 > strudel.json

It's also possible to test without uploading by using --base-url, hosting the files using a local server and running Strudel locally as well.

Reorganize the dataset so that the directories aren't too big (Hugging Face doesn't like that), converting to mp3 along the way:

uv run reorganize_dataset.py nsynth-train/ --strudel strudel.json
mv strudel_reorganized.json strudel.json # updated with the new paths

Then upload to Hugging Face:

uv run upload_to_hf.py

Frequently asked questions (probably by future me)

• Why not use the test/valid set? They don't cover the notes and velocities nearly as densely for the given instruments, for some reason. The train set seems to have maybe 99% velocity/pitch combinations for a given instrument, whereas the two have maybe 20%
• Why reorganize the directory structure to one directory per instrument? Hugging Face doesn't allow you to have more than 10k files in one directory
• Why do we only use some pitches and not all of them? This matches the behavior of Strudel's piano, plus it does decrease the number of network requests needed

License

The dataset is made available by Google Inc. under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

The code itself is CC0.