"Real" color blending [WIP]


#1

tl;dr: myPaint smudge mixes color in RGB, which makes “blue + yellow = green” impossible to mix.
I modified the smudging to use HCY instead, so it blends along the Hue circle:

Disclaimer: I suck at maths and programming, so there might be errors. Feel free to correct me.

The issue
We all learn in school that the primaries are red, yellow and blue (the color circle by Johannes Itten).
And while this is grossly inaccurate in theory, it still holds some practical thruth:
Blue and Yellow pigments really blend to green (not neutral grey). As far as I know ArtRage is the only
application that tries to emulate this “real” color blending. So, when looking at the HCY picker I suddenly came up with an idea that shouldn’t be hard to implement.

The plan
My first thought was to interpolate on straight lines (Like MixerHCYDesaturate, line 18) through HCY space. The benefit are mixed colors that are dull and not so saturated (= more realistic). The downsides however are that B+Y != G but neutral grey again and that it’s quite complicated. So instead I opted for h, c, y separately.
Step number one was to pull over the hcy-to-rgb-to-hcy functions from mypaint/lib/color.py to brushlib/helpers.c.
The next big, buggy hurdle was to jump over 0 to 360 degrees (red). In the end I came up with my own formula, but there are some in the web and even in color.py. And last I had to deal with the greys and the transparent BG, which has hcy 0-0-0 by default. We don’t want to mix red in those…

Edit: See post 12 & 13 for newer code with fixed errors.
replaces line 748 to 776 in brushlib\mypaint-brush.c

// color part

float color_h = mapping_get_base_value(self->settings[MYPAINT_BRUSH_SETTING_COLOR_H]);
float color_s = mapping_get_base_value(self->settings[MYPAINT_BRUSH_SETTING_COLOR_S]);
float color_v = mapping_get_base_value(self->settings[MYPAINT_BRUSH_SETTING_COLOR_V]);
float eraser_target_alpha = 1.0;

if (self->settings_value[MYPAINT_BRUSH_SETTING_SMUDGE] > 0.0) {
	
	float a_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_A ]; // alpha of canvas
	
	// mix (in HCY) the smudge color with the brush color
	float fac = self->settings_value[MYPAINT_BRUSH_SETTING_SMUDGE];
	if (fac > 1.0) fac = 1.0;
	// If the smudge color somewhat transparent, then the resulting
	// dab will do erasing towards that transparency level.
	// see also ../doc/smudge_math.png
	eraser_target_alpha = (1-fac)*1.0 + fac*a_Src;
	// fix rounding errors (they really seem to happen in the previous line)
	eraser_target_alpha = CLAMP(eraser_target_alpha, 0.0, 1.0);
	
	if (eraser_target_alpha > 0.0 && a_Src > 0.0) {
    	 
    	// get and convert canvas
    	float r_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_RA] / a_Src;
		float g_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_GA] / a_Src;
		float b_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_BA] / a_Src;
		rgb_to_hcy_float (&r_Src, &g_Src, &b_Src);
    	 
    	// convert brush
		hsv_to_rgb_float (&color_h, &color_s, &color_v);
		rgb_to_hcy_float (&color_h, &color_s, &color_v);
		
		float h_Brush = color_h;  // brush
		float h_Src; //canvas
		float q_fac = fac; // 0.0 = 100% brush, 1.0 = 100% canvas
    	        float dist; // delta between hues
		float h_Tmp;
		
		if (g_Src <= 0.0 && color_s <= 0.0) {	// both greyscale
			
			color_v = (fac*b_Src + (1-fac)*color_v); // luminance only
			
		} else { // one or none greyscale
			
			h_Src = r_Src;
			
			// one grey, replace default 0.0 red hue
			if (g_Src <= 0.0) {
				h_Src = h_Brush;
			} else if (color_s <= 0.0) {
				h_Brush = h_Src;
			}
												 
			dist = h_Src - h_Brush;
			
			// CW/CCW turn at over 180deg
			if (fabs(dist) > 0.5) {
				if (h_Brush < h_Src) {
					h_Brush += 1.0;
				} else {
					h_Src += 1.0;
				}
				dist = h_Src - h_Brush;
			}

			h_Tmp = h_Brush + (q_fac * dist); // interpolate hue
			
			// Wrap around 360deg
			if (h_Tmp >= 1.0){
				h_Tmp -= 1.0;
			} else if (h_Tmp < 0.0) {
				h_Tmp = 1.0 - h_Tmp;
			}
				
			color_h = h_Tmp; // Hue		
			color_s = (fac*g_Src + (1-fac)*color_s); // Chroma;
			color_v = (fac*b_Src + (1-fac)*color_v); // Luminance;
		
		}

		hcy_to_rgb_float (&color_h, &color_s, &color_v);
		rgb_to_hsv_float (&color_h, &color_s, &color_v);
    
	} // else: We erase with the original hsv brush colors
}`

Issues & Improvements
The performance is probably not as good as rgb, because I do some more conversions.
It bothers me a bit that the saturation in the mix is so high. Real pigments dull a bit. However, I can’t really change this. As far as I understand the smudge does not know if it is “halfway there”. Every dab is a new “start”.
The biggest problem however is the 180 degree turn CW/CCW at 0.5 distance (shortest angular dist). You just need to be 1 degree off from blue to violet and you wont get green when mixing with yellow. If you look carefully at the image above you even notice how cyan-red-cyan and green-magenta-green switch direction around the circle…
I already have an idea how to prevent this a bit manually. This will also fix some of the “odd” mixes like red+cyan to green - I’d say this should go through magenta…
Also: In my local code I just replace the rgb smudge. Implementing it in mypaint for real needs a bit more consideration. e.g. ArtRage enables/disables it’s real color blending in the options, however it’s off by default and especially beginners may never see it.
It could be added as a second smudge brush setting, this ensures that existing brushes behave the same as before (may or may not be a good thing, though). Downside: Overkill.
Another thought just hit me: maybe hcy is not even necessary and hsv works as well o_O I choose hcy in the beginning because of beautiful straight triangulation lines through hcy space…

mixing from “primaries”, pure red, blue, yellow plus black&white…


#2

Cool!

It’s interesting that when you paint cyan on red it turns green, but when you paint red on cyan it turns pink/purple.


#3

Any cylindrical space ought to work for getting the hue term right, but HCY is probably the nicest simple one if you want luma (approximate visual brightness) to be interpolated nicely. You can play with this in the MyPaint palette editor on master right now, if you’re experimenting at home:

The mix methods MyPaint uses for this menu are defined in lib.color, e.g. lib/color.py#L539. Here’s what the three methods (“RGB”, “HCY” and “HSV”) look like in turn for this palette:

I’m disappointed with the way the Python HCY interpolation code handles red-cyan, but I’m happy with the other interpolations between very saturated colours. HSV space interpolations seem to me to contain the most surprises for those.

For the less saturated ones, I’m less sure. HCY tends to make some funky ramps with unduly high absolute saturations here.

Map folks, data geeks, and many others have thought a lot about making nice gradients. They tend to go with cylindrical versions of CIE Lab such as HuSL or LCHUV. I guess that HCY is a poor but serviceable workalike for those.

Links



#4

@bleke

It’s interesting that when you paint cyan on red it turns green, but when you paint red on cyan it turns pink/purple.

Yes, that’s exactly the problem with picking the shortest angle for colors that are complimentary 180 degrees apart. My idea is to split the circle in two overlapping sides orange-violet and orange-teal and instead hardcode the ccw/cw direction if both hues are in the same side, so the angle can be over 180 degrees.

@achadwick
Thanks for the links, I’ve seen some before - there are not many resources about hcy. I suspect that the Lab color spaces need even more conversions… Besides, I like hcy, it’s logical for painting :] . I’m still working on a local branch from the latest version with brushlib as submodul, to make easy windows builds. This palette editor isn’t in there, but I understand. red-cyan there is the same problem as mentioned above. In that case Teal (aka cyan-green aka #00FF80) would be a more natural point to switch direction instead of the 180 degree shortest angle. Red to Cyan looks a lot nicer through magenta.


My theory so far…


#5

Blue and yellow pigments, yes (as opposed to blue and yellow light). I would call this ‘traditional paint blending’ rather than ‘real’ blending (which IMO the blending of light has most claim to, tied as it is to how our eyes work)

This interests me a lot as an option. I would probably want it on most of the time, but not all; I’ve experimented a lot with different colorspaces for blending and find that colorspaces with polar axes always have some really weird wrong-looking cases, which I would only want if I was being intentionally ‘artsy’.

(and you’re right, implementing correct LAB <-> RGB conversions is complicated and fairly slow. LAB blends do look nicer than RGB though, without the polar-axes issues you encountered:
)


#6

Quick update (full view! it’s tall…):


Take note of the complementaries! No more switching around… This is now basically as I described above: If both hues in sector I or both are in sector II we will not check for the shortest angular distance, so e.g. a reddish orange will mix the long way through magenta to get to a greenish cyan. This also prevents a shift of the direction if you are one degree off at complementaries (aka limits this to orange). Now I just need to test this in a real painting. And clean up that code…

@tilkau
If I remember correctly ArtRage called it “Real Color Blending” There it is: http://artrageteam.deviantart.com/art/Realistic-Color-Blending-in-ArtRage-419141057 Yeah, I know, sadly the majority of people still thinks that this is the only “real and true” way color works. I even had teachers in design school talking about red-yellow-blue “primaries” :tired_face: I agree that calling it “traditional …” or “pigment blending mode” or similar would be better.

Also, I noticed in ArtRage that colors tend to add up saturation during “real” blending. I need to test if this happens in my case, too.


#7

There seems to be some odd value shifts as the hue changes sometimes. It gets darker, then brighter, then darker again. It’s probably an optical illusion because the eye doesn’t perceive the colours equally. Still looks a bit strange… maybe one could add adjustment curves for various transitions?


#8

You are right, and actually HCY is supposed to account for exactly those perceived values. Y is the luminance (or was it “luma”?). I made a quick test with lowering the chroma (= something like saturation) and apparently the values are “right”. But I agree, especially cyan looks darker / brighter in both samples (full view! tall again):

I’m not sure if this is a problem caused by PC screen calibration, maybe? My screen is not color profiled (but it’s a good screen nevertheless).

Anyway, if the luminance values were completely off, I’d say we could fix this. But like this… not sure.


#9

Perhaps it’s the Linux implementation of HCY? I can imagine that Apple and Microsoft have invested more in getting the colours right than the Linux folks.


#10

It’s a problem caused by the imperfections of colorspaces (particularly that our response to light of different colors is surprisingly complex). Calibrating your monitor correctly won’t fix it.
Even LAB, which was specifically designed to address problems of perceptual non-uniformity and revised over years, isn’t perceptually uniform. So you’re unlikely to get better than ‘good enough for government work’ on this problem.

@bleke:
There is no Linux implementation of HCY. There’s just MyPaint’s implementation of HCY :wink:


#11

Ok. I thought it was imported from a library.


#12

At least HCY tries to diminish the perception-value-problem… I guess it would be more obvious in other color spaces. Also: In a real painting users will likely mix more muted tones…

I got an update:


Some scribbles with the deevad brushes.

And some code (with fixed errors, too):
`
// color part (replaces rgb mixing block)

float color_h = mapping_get_base_value(self->settings[MYPAINT_BRUSH_SETTING_COLOR_H]);
float color_s = mapping_get_base_value(self->settings[MYPAINT_BRUSH_SETTING_COLOR_S]);
float color_v = mapping_get_base_value(self->settings[MYPAINT_BRUSH_SETTING_COLOR_V]);
float eraser_target_alpha = 1.0;

if (self->settings_value[MYPAINT_BRUSH_SETTING_SMUDGE] > 0.0) {
	
	float a_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_A ]; // alpha of canvas
	
	// mix (in HCY) the smudge color with the brush color
	float fac = self->settings_value[MYPAINT_BRUSH_SETTING_SMUDGE];
	if (fac > 1.0) fac = 1.0;
	// If the smudge color somewhat transparent, then the resulting
	// dab will do erasing towards that transparency level.
	// see also ../doc/smudge_math.png
	eraser_target_alpha = (1-fac)*1.0 + fac*a_Src;
	// fix rounding errors (they really seem to happen in the previous line)
	eraser_target_alpha = CLAMP(eraser_target_alpha, 0.0, 1.0);
	
	if (eraser_target_alpha > 0.0 && a_Src > 0.0) {
    	 
    	// get and convert canvas
    	float r_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_RA] / a_Src;
		float g_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_GA] / a_Src;
		float b_Src = self->states[MYPAINT_BRUSH_STATE_SMUDGE_BA] / a_Src;
		rgb_to_hcy_float (&r_Src, &g_Src, &b_Src);
    	 
    	// convert brush
		hsv_to_rgb_float (&color_h, &color_s, &color_v);
		rgb_to_hcy_float (&color_h, &color_s, &color_v);
		
		float h_Brush = color_h;  // brush
		float h_Src; //canvas
		float q_fac = fac; // 0.0 = 100% brush, 1.0 = 100% canvas
    	float dist; // delta between hues
		float h_Tmp;
		
		// both hues are greyscale
		if (g_Src <= 0.0 && color_s <= 0.0) {		
					
			color_v = (fac*b_Src + (1.0-fac)*color_v); // luminance only		
					
		} else { // one or none greyscale
			
			h_Src = r_Src;
			
			// one hue is grey, replace default 0.0 red hue
			if (g_Src <= 0.0) {
				h_Src = h_Brush;
			} else if (color_s <= 0.0) {
				h_Brush = h_Src;
			}
			
			const float degree30 = 30.0/360.0;
			const float degree120 = 1.0/3.0;
			const float degree240 = 2.0/3.0;
			h_Src -= degree30; // orange to origin offset
			h_Brush -= degree30;
			h_Src = circular_wrap(h_Src, 1.0);
			h_Brush = circular_wrap(h_Brush, 1.0);
			
			dist = h_Src - h_Brush;
			
			// See if both hues are in one sector, else...
			if (!(h_Brush <= degree240 && h_Src <= degree240) 
			&& !(h_Brush >= degree120 && h_Src >= degree120)) {
				// CW/CCW turn at over 210deg
				if (fabs(dist) > (0.5-degree30)) {
					if (h_Brush < h_Src) {
						h_Brush += 1.0;
					} else {
						h_Src += 1.0;
					}
					dist = h_Src - h_Brush;
				}
			}

			// interpolate hue
			h_Tmp = h_Brush + (q_fac * dist); 
			h_Tmp += degree30; //remove orange origin offset
			
			// Wrap around 360deg
			h_Tmp = circular_wrap(h_Tmp, 1.0);	
			
			color_h = h_Tmp; // Hue		
			color_s = (fac*g_Src + (1.0-fac)*color_s); // Chroma;
			color_v = (fac*b_Src + (1.0-fac)*color_v); // Luminance;			
		}

		hcy_to_rgb_float (&color_h, &color_s, &color_v);
		rgb_to_hsv_float (&color_h, &color_s, &color_v);
    
	} // else: We erase with the original hsv brush colors
}`

I also figured out that new stuff in helpers.c needs to be in helpers.h, too (noob) so I finally outsourced the wrap-in-a-circle-function and fixed the errors in it [edit: even more fixes]:

float
circular_wrap(float num, float max) {
    if (num >= max){
        num = fmod(num, max);
    } else if (num < 0.0) {
        num = max - fabs(fmod(num,max));
    }
    if (num == max) {
        num = 0.0;
    }
	return num;
}

Also: I found that awesome palette editor interpolation thing. It’s not in the “palette editor”, but in the palette tab… :zipper_mouth:


Seamless, repeating pattern mode [WIP]
#13

I tried to add this stuff to github:

1.) I can’t test it with the latest mypaint state, because of windows, so no guarantee that everything works
2.) it’s only the brushlib and the default RGB mixing is simply commented out. Anyway, should be all you need to try it for yourselves.

I had to delete my messy, old forks, but will add all the old code (e.g. the offset) into tidy new branches.
Edit: I have now a fresh brushlib fork with three flowering branches: offset-test, direction360-test and pigmentblending-test. This should be all my old code from brushlib.



#14

Looks good to me! Thanks for making it easily testable.

I reviewed your various branches but haven’t had time to test yet.

There is one notable mistake: “stoke” is consistently used instead of “stroke” in offset-test branch .


#15

Typo is fixed now, thanks.


#16

Not sure where to start with this, but I would encourage the most dramatic shift for painting will come from a linearized reference space. Try painting fuzzy red on cyan at full intensity. See that dark fringing? That is from a broken, nonlinear reference space.

All mixing of colour is broken in nonlinear references, no matter how much muddling is attempted. Long before attempting to tackle an alternate set of primaries as a reference, it would be most prudent to start with a strictly linearized reference space.

(The screenshots all show the earmarks of broken nonlinear reference if you look at the subpixel fringes on the green strokes against red, the red strokes against green, etc.)


#17

Andrew Chadwick already tried linear RGB [in a branch available on github] and concluded the blending has different problems (eg. try painting white on black. The white comes out ‘too strong’; I’ve tried this myself.). Though I think it might be an acceptable tradeoff, myself.

Something like LAB/LCH might be better (but as noted, is tricky and definitely slower)

Anyway, as I understand it that fringing is outside the scope of AnTi’s patch, since it’s caused by the colorspace the image itself is using.


#18

try painting white on black. The white comes out ‘too strong’; I’ve tried this myself.

I have been using linearized reference spaces forever and I am unfamiliar with the issue you are citing. It sounds more like a broken algorithm, or likely an unmanaged interface element?

The bottom line is that if you separate the reference from the view transform, and use energy based algorithms, everything holds up perfectly fine, and is the basis for solving a number of mixing issues. Without it, every single colour operation is horrifically broken.


#19

You don’t need to explain gamma correction/companding to me. I am extremely familiar with the problem and frequently advocate linear mixing. But I haven’t encountered a single colorspace that does ‘correct’ mixing (LAB is the closest I’ve seen, and is noted to have clear perceptual irregularities in spite of being designed to be perceptually regular). Bottom line, linear RGB blending is ‘less broken’ (ie. not mathematically nonsensical), not ‘fixed’.

Why don’t you look at the branch yourself? I don’t think it’s algorithmically wrong, as you can compare with eg. GPick’s blending (which uses linear RGB when you select ‘RGB’ colorspace) and observe the same ‘white is too strong’ effect in a black -> white blend; and results are otherwise as expected (no undue darkening/muddying, a gradient black->yellow passes through brownish midpoint instead of ugly greenish, etc).

The white on black thing shouldn’t be a deal breaker, btw. Blender copes with the same issues as it lets the user paint textures in linear space. It’s just a question of whether this is too surprising for the user or not. I think the user can adapt (which again, is different from assuming the issue is not present).


#20

I’m happy to revisit linear spaces for the model in MyPaint.

Really, libmypaint ought to be independent of how pixels are stored; its job is to calculate a mask for each dab [… alpha is always linear …] and splodge down a colour as quick as it can.

It’s the job of MyPaint’s custom Surface implementation to feed libmypaint the right colour for smudge-type blending.

The stumbling block for MyPaint has always been what we do with legacy ORA documents which have non-linear assumptions. Layer modes, for example, need to retain the same appearance whatever that is.