Need help fixing my own soundfont 2.04-based MIDI synthesizer's attenuation issues

[superfury]

I've been working on implementing my own Soundfont 2.04-based synthesizer for a while now.
I've managed to get basic MIDI parsing and basic playback of the samples with effects (ADSRs, volume, pitch and normal dynamic low-pass filters) and most basic effects done.
The chorus is simply made using the official method (playing a sample once raw and once with modulation).
The reverb is very simple: a small delay line (two taps) with loopback that has volume reduction and a low-pass filter based on the frequency of the sound at the loopback of samples onto itself, using a simple circular FIFO buffer with taps.

The code can be found at:
https://bitbucket.org/superfury/unipcemu/src/default/UniPCemu/hardware/midi/
for the synth implementation (and MIDI parsing).
Then the soundfont reading is at
https://bitbucket.org/superfury/unipcemu/src/default/UniPCemu/support/sf2.c
It's headers are at:
https://bitbucket.org/superfury/unipcemu/src/default/UniPCemu/headers/hardware/midi/
https://bitbucket.org/superfury/unipcemu/src/default/UniPCemu/headers/support/sf2.h

It uses my custom SDL (1.0/2.0/3.x)-based output system (that mixes all channels using their own sample rates).
All mixing happens at a very low level (groups of just a few samples at a time). Most of the synth itself runs at a very high latency (rendering it all live as commands are received). All blocks of rendered samples only render 42 samples at 44.1kHz speed for each buffer call.

One issue I currently have with it is that the volume for various instruments sounds messed up. Some instruments are way too loud, some are way too soft, some are roughly correct.

I'm using the basic soundfont 2.04 and look at the Fluidsynth source code to try and figure out what's going wrong with my formulas.

The ADSR is mostly linear in nature (runs using simple addition of a divided number based on the amount of samples to render a phase at), although I've customized it's convex curve based on the Fluidsynth formula combined with the SF2-DLS paper (**https://basicsynth.com/uploads/SF2-DLS.pdf) and official SF2 spec (2.04).

Can anyone see what's going wrong with my synth? Are the concave/convex formulas incorrect? Is the attenuation performed incorrectly (it uses a basic power of 10 with an exponent of x/-200 for that)? I can't seem to figure out why the sounds aren't attenuated correctly fully.

One odd thing I did notice in the official specs is that it uses sqrt (square root) on the square of the modulator's input. But no other synthesizer I can find (both the SF2-DLS documentation and Fluidsynth) seems to perform that step? I've used it in the ADSR attack phase, where it seems to do no harm, but when applying it to the modulator concave/convex curves it causes all sound to be (as good as) muted, which isn't supposed to happen?

The attenuation is simply applied using a 1440-entry lookup table, which has the 10^(x/-200) outputs loaded into it, that it then multipled onto the low-pass filtered sample that's retrieved from the soundfont to obtain the resulting sample to render in later steps (panning first, then applying reverb effects).

For people trying to use the app themselves, the simplest way test run it to play midi files is:

• Open the app (after compiling it) and close it to make it generate a settings.ini file.
• Open the settings.ini file, the section XTCMOS has two setting you want to change:
  -- Set cpuspeed to 1 (to give the MIDI synth maximum speed to run at). Otherwise, the emulated system will slow down the MIDI synth's rendering to be in sync with it, which is too slow for live rendering.
  -- Set soundfont to the filename of the soundfont (including .sf2 extension) in the soundfonts subdirectory.
  -- Set MIDIsynthdefaultmode to the default mode of the MIDI synth (MMA, GM, GS or XG). This is applied when the synth is reset (or initialized) or when the GM/GS/XG mode is exited by a MIDI command.
• Place the soundfont in the soundfonts subdirectory.
• Place any songs to play in the music subdirectory (mid or midi extension).
  Then, to play a MIDI file, start the app, backspace or click the Set button at the bottom right while the yellow text displays at the upper left corner (this enters the pre-boot menu, which is required to play music files using the built-in audio player). Navigate (using Q/W/left/right arrow keys) to the Advanced menu, click on Sound settings (or navigate there using cursor keys and press enter), next choose Music Player, navigate to the song to play using up/down keys (left/right keys fast scroll by the first letter of the filename) and press enter to play the song. During playback, the escape key stops playback. (alternatively numpad 6(escape) and numpad 2(enter) can be used, as they're the PSP-compatible key mappings for the same functionality of those keys).

[superfury]

Anybody knows how soundfont synthesis is supposed to properly handle the attenuation and what formulas are used? The soundfont documentation doesn't talk about it very much and the SF2-DLS spec also leaves a lot of holes (it doesn't even mention the SF2 specs and all it's connections, like it's modulators and generators).

[spessasus]

I've created my own softsynth which passes all the SF2 compatibility tests and mostly matches fluidsynth in the produced audio.

There are two things about attenuation you need to know:

1. The initialAttenuation generator has to be multiplied by 0.04 to get decibels instead of 0.1. for example 250 is 10 dB. All soundfont synths do this because the original Sound blasters did that for some reason and it stayed.
2. If your synth implements modulators, you need to apply the generator limits after calculating modulators. For example some soundfonts put negative values in the initialAttenuation generator at the instrument level, which should get clamped to 0. However, this has a quirk of still being at full volume when cc7 is at values like 100-127 since the modulator gets cancelled out by the generator in that range. This is undefined behavior but other synths apply it too I believe, and unfortunately a lot of soundfonts do things like these.

I hope this helps.

[superfury]

I've created my own softsynth which passes all the SF2 compatibility tests and mostly matches fluidsynth in the produced audio.

There are two things about attenuation you need to know:

1. The initialAttenuation generator has to be multiplied by 0.04 to get decibels instead of 0.1. for example 250 is 10 dB. All soundfont synths do this because the original Sound blasters did that for some reason and it stayed.
2. If your synth implements modulators, you need to apply the generator limits after calculating modulators. For example some soundfonts put negative values in the initialAttenuation generator at the instrument level, which should get clamped to 0. However, this has a quirk of still being at full volume when cc7 is at values like 100-127 since the modulator gets cancelled out by the generator in that range. This is undefined behavior but other synths apply it too I believe, and unfortunately a lot of soundfonts do things like these.

I hope this helps.

So initialattenuation is applied using 10^(x/-500) instead? -500 is built as an optimization from: -10 for official dB attenuation (itself an optimization for -dB/10, where -dB is pre-applied to the bottom instead), times 25 (1/0.04), since the original was times 10(1/0.1) in the SF2-DLS documentation.

Do you mean clipping the attenuation before adding to the initialAttenuation generators or after? Right now, clipping occurs after adding to the initial attenuation (so if for example, the instrument has 10, preset has 10, modulators have -20, then before applying the modulators it's at 20. Then after the modulators are applied, it's at 0. So it's basically CLIP(instrumentgen+presetgen+modulators,0,1440) right now.
Or is 'modulators' supposed to be clipped too, so modulators can't substract from the generators? Thus CLIP(instrumentgen+presetgen+CLIP(modulators,0,1440),0,1440) Is that true for other modulators too? Or do you mean that instrumentgen, presetgen and modulators are all clipped, then clipped again after adding them together? What about modulators modulating other modulators (destination top bit set)? How is amount applied in that case (or normalization)?

Also, why is it called "initial Attenuation" and not just "attenuation"? What's 'initial' about it?

Also also, is 'sqrt' (square root, same in c/c++ sqrt(f) function?) for concave/convex really to be applied to the (value^2) or not? Fluidsynth doesn't seem to, nor any others i've found?
This is what I've gotten implemented for concave right now

float MIDIconcave(float val, float maxvalue)
{
	float result;
	if (val <= 0.0f) //Invalid?
	{
		return 0.0f; //Nothing!
	}
	if (val >= maxvalue) //Invalid?
	{
		return 1.0f; //Full!
	}
	//result = (-200.0f * 2 / 1440.f) * log((maxvalue-val) / (256.0f - 1.0f)) / the_ln10; //Concave curve
	result = (maxvalue-val); //Value. This is reversed for convex
	//maxvalue is max range. range=maxvalue-minvalue, so since minvalue=0, range=maxvalue
	maxvalue *= maxvalue; //Range is squared
	result *= result; //Value is squared
	result = (-20.0f/96.0f)*log10f(result/maxvalue); //-20/96 * log(sqrt((value^2)/(range^2)))
	result = LIMITRANGE(result,0.0f,1.0f); //Limit the range!
	return result; //Give the result!
}

Convex simply inverts the x (input) and y (output) using maxvalue-x and 1.0-y respectively.

[superfury]

Just compared it to Fluidsynth. Apparently fluidsynth divides the log output by ln10.
And it doesn't square the maximum value and value inputs? And instead of -20/96 it uses -200/960(which is the same) and divides the log output by ln10 (2.3025850929940456840179914546843642076011L when I look up Fluidsynth's used library's source code).

Thus, I ended up with:

define the_ln10 2.3025850929940456840179914546843642076011L
//val needs to be a normalized input! Performs a concave from 1 to 0!
float MIDIconcave(float val, float maxvalue)
{
	float result;
	if (val <= 0.0f) //Invalid?
	{
		return 0.0f; //Nothing!
	}
	if (val >= maxvalue) //Invalid?
	{
		return 1.0f; //Full!
	}
	//result = (-200.0f * 2 / 960.f(peak attenuation)) * log((maxvalue-val) / (256.0f - 1.0f)) / the_ln10; //Concave curve
	result = ((maxvalue-1)-val); //Value. This is reversed for convex
	//maxvalue is max range. range=maxvalue-minvalue, so since minvalue=0, range=maxvalue
	//maxvalue *= maxvalue; //Range is squared
	//result *= result; //Value is squared
	result = (-200.0*2.0/960.0)*log10f(result/(maxvalue-1))/the_ln10; //-20/96 * log(sqrt((value^2)/(range^2)))
	result = LIMITRANGE(result,0.0f,1.0f); //Limit the range!
	return result; //Give the result!
}

Though some (short lasting) notes still sound too soft? Could it be because of the volume envelope malfunctioning (for short notes). Right now, release velocity is ignored and when a note is released (even when it's still attacking), it switches to release phase, calculates a linear slope from the current (attack/hold/decay/sustain) level and starts releasing the note. Is that correct behaviour? Or does it depend on if a note has a sustain or not?

[superfury]

What happens when a MIDI note off is received while the ADSR isn't at sustain yet? Will it enter release starting from the current attack/hold/decay level for the specified interval? Does release velocity affect it?

Edit: The ADSR also uses roughly the same formula, except adjusted for 1440cB of attenuation:

//val needs to be a normalized input! Performs a convex from 0 to 1!
#define the_ln10 2.3025850929940456840179914546843642076011L
float attackconvex(float val)
{
	float result;
	if (val <= 0.0f) //Invalid?
	{
		return 0.0f; //Nothing!
	}
	if (val >= 1.0f) //Invalid?
	{
		return 1.0f; //Full!
	}
	result = val; //Value.
	//The range and input is both the same, so ignore it in our calculations.
	result = ((-200.0*2.0/1440.0)*(1.0/the_ln10))*log10f(result); //-20/96 * log((value^2)/(range^2))
	result = LIMITRANGE(result,0.0f,1.0f); //Limit the range!
	result = 1.0f - result; //Inverted!
	return result; //Give the result!
}

Normal concave/convex (for modulators) is simply this now:

#define the_ln10 2.3025850929940456840179914546843642076011L
//val needs to be a normalized input! Performs a concave from 1 to 0!
float MIDIconcave(float val, float maxvalue)
{
	float result;
	if (val <= 0.0f) //Invalid?
	{
		return 0.0f; //Nothing!
	}
	if (val >= maxvalue) //Invalid?
	{
		return 1.0f; //Full!
	}
	//result = (-200.0f * 2 / 960.f(peak attenuation)) * log((maxvalue-val) / (256.0f - 1.0f)) / the_ln10; //Concave curve
	result = ((maxvalue-1)-val); //Value. This is reversed for convex
	//maxvalue is max range. range=maxvalue-minvalue, so since minvalue=0, range=maxvalue
	//maxvalue *= maxvalue; //Range is squared
	//result *= result; //Value is squared
	result = ((-200.0*2.0/960.0)*(1.0/the_ln10))*log10f(result/(maxvalue-1)); //-20/96 * log(sqrt((value^2)/(range^2)))
	result = LIMITRANGE(result,0.0f,1.0f); //Limit the range!
	return result; //Give the result!
}

//val needs to be a normalized input! Performs a convex from 0 to 1!
float MIDIconvex(float val, float maxvalue)
{
	float result;
	if (val <= 0.0f) //Invalid?
	{
		return 0.0f; //Nothing!
	}
	if (val >= maxvalue) //Invalid?
	{
		return 1.0f; //Full!
	}
	val = ((maxvalue-1)-val); //Inverted!
	result = MIDIconcave(val,maxvalue); //Same as the concave curve!
	result = 1.0f-result; //Inverted!
	return result; //Give the result!
}

Maxvalue is usually 128. val is ranging from 0.0-127.0

The 960 is simply made 1440 in this formula, that's the entire difference (and maxvalue-1 division and inversion optimized out to make it more optimized).

[superfury]

Looked at the FluidSynth volume curve. It seems to use 10^(x/-200) too. So for now I've changed it back from -500 to -200. That at least fixes most of the volumes to not be too loud.

Some instruments still oddly seem very loud though? Perhaps a generator/modulator handling issue somehow?

The Soundfont documentation does document those, but various things are undocumented apparently, like:

• How modulators actually modulate other modulators (how are they scaled when applied to a modulator source from another modulator's output)? Right now I've implemented the scale to be scaled back by dividing by a scale set when the bottom operator (which is applied to the generator) gives all it's parent modulators the scale to use (mostly taken from Fluidsynth for now, although none of the Soundfonts I can find seem to even use it).
• How does overriding of modulators and local/global even work? It's kind of confusing how it's documented. I at least know that later modulators in the list override earlier modulators (or default for instrument modulators from what I can tell). Then how do global modulators combine with this? Do they override it, add to it? What about generators?

[superfury]

The Soundfont 2.04 spec for the PMOD chunk gives the following:

The latter is indicated by the top bit of
the sfModDestOper field being set, the other 15 bits designates the index value of the modulator whose source should be the
output of the current modulator RELATIVE TO the first modulator in the instrument zone.

Doesn't it mean 'preset zone' instead of 'instrument zone'? It's pointing to a PMOD entry, not a IMOD entry? In my implementation, it points to a PMOD entry (within the preset zone's list).

[spessasus]

Linked modulators (modulators that alter other modulators) are not implemented in fluidsynth nor my synthesizer. I don't know of any SF2 file that actually makes use of them, so I recommend skipping them.

[spessasus]

Details of fluidsynth's SF2 implementation

[superfury]

How do instrument/preset zones and modulators and generators (and default modulators) combine/override? Soundfont 2.04 section 9.4 is kind of confusing in how it describes priorities and how they combine (it even says 'supercedes or replaces' which sound like unclear?

From what I gather local > global > default for identical modulators or generators?
So local negates the global/default, replacing them if identical. And global replaces default if identical and not local. So far, it's simple.
But then it says:

- A generator in a local preset zone that is identical to a generator in a global preset zone supersedes or replaces that generator in the global preset zone. That generator then has its effects added to the destination-summing node of all zones in the given instrument.

- A generator in a local preset zone which is not identical to a default generator or a generator in a global preset zone has its effects added to the destination summing node of all zones in the given instrument.

How so, "destination summing node of all zones in the given instrument" so all zones receive any local preset settings from all other zones that apply? So say one zone of note 60-65 and one zone of note 66-70 for example get their values of their non-index generators summed for both key ranges? Or if both are played? That doesn't make much sense to do? Or only if multiple parts of the same key/velocity range play for a single note being played (notes with multiple zones matching)?

Edit: Improved the modulators and generators to properly apply local > global > default(modulators only). The default modulators are actually applied at the local step, but they check if any global or local modulators exist to determine if they're to be ignored or not.
Global modulators check for local modulators to determine if they're to be ignored or not (just like the local modulators do).
The order they're applied in are basically:

1. Default (if not blocked by any of the below if detected, skipped otherwise). Actually applied before local modulators, but detecting if any matching on local/global exists to be skipped/ignored.
2. Local modulators (overruling default/global modulators and generators)
3. Global modulators (if not blocked by local, skipped otherwise, in the case of generators, overruled by local generators)

Edit: OK. Loaded up the Soundfont inside Viena. The first 3 instruments play at 150(->169 modulated), 67(->86 modulated), 172(->191)cB respectively. But the notes in the test song all seem to use the same volume settings? So somehow it's not dampening the sound enough?
Double checking the formula (10^(x/-200)) seems to match the basic law for cB being applied? Is there an issue there?
Edit: Trying a soundfont without attenuation generators sounds fine. When attenuation generators are specified, volume becomes messy somehow (too soft or loud for many instruments)? The default modulators are doing their work in this case?

[superfury]

Thinking about it... Modulators modulating modulators... Perhaps their 0-1 range is mapped to amount units as usual, then the result being part of the destination modulator's source operand's range normalizarion (either 128 or 0x4000 or maybe even 32767 for none/linked), being added to the source that's modulating? That does match the pictures in the 2.04 documentation, where the -->(+) is placed before the normalization step on the target modulator.

So for example, you could put a modulator modulating amount 127(/128) from controller 20h+ modulating a modulator that drives for example initialAttenuation from modulator 0+ to archieve a 14-bit volume controller.

And for the case of the 'linked' modulator or 'no modulator', the range is simply 0x4000 or maybe even 32767 for maximum control.

[superfury]

Managed to get some 'missing' instruments to work. For example, preset 9 (glockenspiel) had multiple preset and instrument zones, with two zones needing playback to be working properly. The first playing zone had an issue that makes it end abruptly (giving a pop because of playing a single sine wave and then ending due to the loop setting not being specified (neither globally nor locally). Then the other preset zone points to two instrument zones, one which contains the proper instrument to play (with proper looping on the sample being enabled).
One issue that was happening was a misdetection and misparsing of the preset and instrument bags, with the second and onwards detected zone not being played because of a malfunctioning playing zone counter (it would reset for each ibag searched, whereas it should clear only before starting to scan all preset and instrument zones. And still monotonically increasing for each instrument zone that can be played according to some basic playback criteria (the usual preset/instrument filters (key/velocity) and sample validity checks).
Now some effects sound proper when multiple zones need to be played back simultaneously (including the weird one pop (single sine) plus normal looping zones of instrument # 9 (glockenspiel)).
That one was found by accident, but no clue if it's a soundfont conversion error(since it's a converted soundfont (AweROMGM.sf2)) or a human oversight (since the instrument sounds fine with just the second zone playing properly).

I also added some bag range checks and some more Soundfont documented ignore cases (like ignoring all generators after instrument generator, ending operator checks (sometimes omitted from the soundfont itself) etc.).

Despite the volume issues still (all over the place) it sounds way better without missing notes now.

[spessasus]

If you're planning on making a fully compliant SF2 synth (with modulators), then I suggest you test it with these two:

• SoundFont Spec Test - a tester for soundfont compatibility, good for various undocumented behaviors that are expected of fully-compliant synths
• GeneralUserGS - one of the most complex soundfonts out there, format-wise. It uses a ton of modulators, generators and it's a great way to test your synth. And a great GM bank in general :-)

[superfury]

If you're planning on making a fully compliant SF2 synth (with modulators), then I suggest you test it with these two:

• SoundFont Spec Test - a tester for soundfont compatibility, good for various undocumented behaviors that are expected of fully-compliant synths
• GeneralUserGS - one of the most complex soundfonts out there, format-wise. It uses a ton of modulators, generators and it's a great way to test your synth. And a great GM bank in general :-)

Tried that GeneralUserGS soundfont with some music. Like with the Awe32ROMGM soundfont, it sounds fine. Just volume being all over the place. Some instruments sound like they're in the volume ballpark, some way too quiet, some way too loud.
Familiar with the Runescape 2 songs? Stratosphere sounds fine, but the small bell is way too loud, as well as one other instrument producing some vibrato effect somehow.
https://www.dropbox.com/scl/fi/ceq3ywe6z41hd8v1rfpvm/recording_16-stratosphere.zip?rlkey=6qaoc78md0blyviomzz9jhgx6&st=wuuacqus&dl=0

[superfury]

https://www.polyphone.io/fr/forum/support-bug-reports/426-attenuation-modulator-destination-calculated-wrong
Is it really true that generators only set attenuation to 40% of what's configured?

[superfury]

Oh, I seem to have misread your comment about the initialAttenuation generator. I applied it to all resulting values, including the modulators, which it shouldn't. Also shouldn't it be multiplied by 0.4 (40%) instead of 0.04? Although that depends on how you look at the used formulas (as the x0.1 multiplication is done in my generic attenuation precalcs (a lookup table) instead of at the generator/modulator processing).

[spessasus]

Initial attenuation generator uses centibels which are 1/10th of a decibel. So to get decibels you do 0.1 * 0.4 which gives you 0.04.

[spessasus]

Also it only applies to the generator, not the modulators!

[superfury]

Implemented it like that now. Modulators being unchanged, generators times 0.4.
The modulators are added to the resulting generators (in cB). Then the volume envelope is added (0-1 scaled to 0-1440). After that, the lookup table is applied to the 0-1440 range number (it contains the formula "10^(x/-200)"). That is applied to the sample by multiplication to obtain the resulting sample to play. That's after the low-pass filter but before applying chorus, reverb and panning.

The ADSR envelope is effectively calculated with a peak of 1.0. The sustain level is simply inverted an normalized (y=1440-x or y=1000-x, then y/1440 or y/1000). The attack is performed using the normal MIDI concave formula (on the 0 to 1 curve). It's just optimized to use less redundant operations that the normal modulator concave/convex (inversion of inversion etc.). Release is started from the level the attack/hold/decay/sustain calculated at the sample it starts (the current ADSR is calculated before starting the release phase handling, so if for time point x the attack would be 0.5f (on the 1.0 scale), then the release starts ramping down from 0.5f to 0.0f linearly.
Said volume envelope is inverted again before passing to the attenuation combinator (which multiplies it by 1440 again to convert to cB of attenuation) for the attenuation lookup table.
Is that correct behaviour for the sustain?

Edit: Also, how does the Master Volume combine into all this?
https://www.recordingblogs.com/wiki/midi-master-volume-message
In my case, it's simply taken and converted to 1440cB scale of attenuation (by inverting the number) linearly.