Thanks @odysseywestra, it looks good and I rebased without any merge conflicts or anything.
I added yet another setting, mostly just to help test different modes of mixing the paints. I wanted to try HCY blending, too, so I copied the functions from @AnTi's branch and made the mode selectable with the Smudge RYB Mode setting. Ranges 0- <1 use just RYB for the mix without messing with saturation or luma. 1-<2 uses HCY, 2-<3 uses HSL, and 3-4 uses HSV. These latter three modes use RYB for just the hue, and then mixes the saturation (unless sat=0) and luma with the selected mode. I rather like the plain RYB mix, but really there are things I like about all of them so it's hard to pick a favorite.
This image shows some mixes with the 4 modes across 3 ryb smudge sat settings (0, .5, 2) and finally just rgb for comparison. The ryb columns in each row should be identical since that mode doesn't use the sat setting. Its just pure RYB mixing.
There are interesting differences between each kind of mix. Notice how HSV goes towards white when mixing luma and HSL goes towards dark?
I'm changed the whole RYB Smudge setting to a generic Smudge Mix Mode setting and abandon the crossfade in the process (blending RYB / RGB is generally pretty ugly). So basically you'd have:
Smudge Mix Mode: Range 0-1: RGB, 1-2: RYB, 2-3: CMYK, 3-4: (new fancy Spectral mode by Scott Burns), etc
Smudge Adjustment Mode: 0-1: Native (no adjustment), 1-2:HCY, 2-3: HSL, 3-4: HSV, 4-5: LAB, etc
Smudge Desaturation: -10 to +10. Assuming you picked an adjustment mode, the saturation is decreased or increased based on hue angle difference
Only RGB and RYB are implemented at the moment, for mix modes (I want to add CMYK soon and Spectral... maybe never). For Adjustment modes HCY, HSL, and HSV are working.