Roundtoe

How two pixels become one

Blending looks like one multiply. Underneath it sit forty years of pixel formats, palette tricks, integer maths and one famous 1984 paper.

16 min read
  • graphics
  • engineering

Every piece of software you looked at today ran the same tiny function millions of times: take the pixel that is already there, take the pixel that wants to be there, and produce the pixel that will actually be shown. Text over a background. A shadow under a window. A health bar over a battlefield. A countdown timer over a gradient.

That function is called a blend, and it is one of the small number of computations that has been rewritten for every machine, every language and every decade since screens existed. It looks like one line of arithmetic. It is actually a stack of decisions about memory, precision, colour and history, and when any one of those decisions is wrong the picture is wrong in ways that are maddeningly hard to describe. Slightly dark edges. A green fringe on white text. A gradient with bands in it.

This is a tour of the whole stack, from what a framebuffer even is, through the formats pixels have been stored in, to working blend code in JavaScript, C and C#.

First, what a surface is

A surface is a rectangle of pixels in memory. Three numbers describe its shape: width, height, and stride. Width and height are what you expect. Stride is the one that bites: it is the number of bytes from the start of one row to the start of the next, and it is frequently not what the width implies, because rows get padded to align nicely for the hardware.

The pixel at column x of row y lives at:

address = base + y*stride + x*bytesPerPixel

Walk rows with the stride and columns with the pixel size, and never assume stride equals width times pixel size. Every graphics programmer learns this by drawing a picture that comes out sheared into diagonal stripes, exactly once.

A blender, then, is a function applied at matching positions of two surfaces:

for every y, for every x:
    destination[y][x] = blend(source[y][x], destination[y][x])

Everything else in this article is about what goes inside blend, and about the inconvenient fact that source[y][x] is not one number but an encoding.

A short history of putting one thing on top of another

The earliest framebuffers had no notion of blending because they barely had the notion of a pixel value. One bit per pixel: ink or paper. Overlap meant OR, or XOR when you wanted to erase by drawing twice, which is how early cursors worked.

The 1980s brought indexed colour. Memory was too expensive to store a full colour per pixel, so the pixel became a small number, an index into a table of colours called a palette. EGA offered 16 entries, VGA's famous mode 13h offered 256. The Amiga shipped a dedicated chip, the blitter, that could combine three bitmaps with an arbitrary logical function as fast as memory allowed, which is why so many Amiga effects feel like set theory in motion.

Then in 1984 Thomas Porter and Tom Duff published a short SIGGRAPH paper called Compositing Digital Images, building on the alpha channel that Ed Catmull and Alvy Ray Smith had added to pixels a few years earlier. The paper did two things that stuck. It treated transparency as a fourth number stored with every pixel rather than a property of an object. And it worked out the complete algebra of combining two such pixels: not just “A over B” but twelve operators, each a pair of coefficients. Every compositor you have ever used, from a browser to a film pipeline, is running Porter and Duff operators. Almost always the one called over.

The 1990s games industry, still stuck with palettes, faked all of it with lookup tables, and it is worth pausing on how, because the trick still matters. You cannot blend palette indices. Index 5 plus index 9 is not a colour, it is nonsense. So engines precomputed a table: for every pair of palette entries, which single entry best approximates the mix. The Build engine behind Duke Nukem 3D shipped one. The Boom source port added one to Doom, a 64 kilobyte file called TRANMAP, which is exactly 256 times 256 entries of “if this is on top of that, show this”. Translucency as a multiplication table.

Only when direct colour became standard, every pixel carrying its own red, green and blue, did blending become the arithmetic we now write casually.

The blend itself

The operator you want almost every time is over. Source, with opacity alpha, goes over destination:

out = src*a + dst*(1-a)        with a in 0..1

Alpha is coverage: what fraction of this pixel the source actually occupies. At the crisp centre of a glyph alpha is 1. In the empty space beside it, alpha is 0. On the smooth edge it is somewhere between, and that in between is what antialiasing is.

Since a surface is just numbers, here is one. This is the top left corner of a rendered letter O at small size, shown as its alpha channel, 0 to 255. Each cell is lightly tinted with its own value, so the table is also the picture:

x0x1x2x3x4x5
y00024118189226
y1047213255255255
y22421325519712188
y31182551972100
y4189255121000
y522625588000

Read the diagonal and you can see the curve of the stroke: solid ink at 255, bare background at 0, and a two or three pixel ramp between them.

Now watch the blend actually happen, one channel at a time. Say the ink is white, value 255, and the destination is a background gradient that brightens towards the bottom of the image. Here is that destination, the same 6 by 6 region:

x0x1x2x3x4x5
y0404040404040
y1707070707070
y2100100100100100100
y3130130130130130130
y4160160160160160160
y5190190190190190190

Apply out = (255*a + d*(255-a)) / 255 at every position, using the alpha table above, and this comes out:

x0x1x2x3x4x5
y0404060139199231
y170104225255255255
y2115229255220174153
y3188255227140130130
y4230255205160160160
y5248255212190190190

Every cell tells the same story at a different mix. Where alpha was 0 the background survives untouched, still reading 40 at the top and 190 at the bottom. Where alpha was 255 the ink lands at full strength. And along the edge of the stroke each pixel settles somewhere between its background and 255, in proportion to its coverage: the 24 at the top corner lifts a 40 background only to 60, while the same ramp two rows down, over a 100 background, lands on 115. That is the whole trick of antialiasing: the edge pixels are not a special colour, they are the honest average of ink and whatever happened to be underneath. Blend that ramp badly and the eye reads the result instantly as “cheap”.

Over is not the only operator. The common ones, with destination d and source s in the 0..1 range:

Name Formula What it is for
Over s*a + d*(1-a) Everything. Text, sprites, UI, layers
Additive min(1, d + s) Fire, glows, lens flares, lasers
Multiply d * s Shadows, tinting, lightmaps
Screen 1-(1-d)*(1-s) The bright twin of multiply
Min / Max min(d,s) max(d,s) Masks, erosion effects

Additive blending is why every 1990s explosion looks the way it does: it can only make the image brighter, so overlapping particles bloom towards white. Multiply can only darken, which is why it makes convincing shadows. Over is the only one that needs alpha at all, and it is the one the rest of this article cares about.

Pixel formats, or what a pixel even is

The formula above pretends a pixel is a tidy set of numbers between 0 and 1. In memory it never is. A sample of the formats you will actually meet:

Format Bits Layout in memory Alpha Where you meet it
Indexed 8 8 one palette index no GIF, PNG8, classic games
RGB565 16 5 red, 6 green, 5 blue no embedded screens, old phones
RGB888 24 one byte per channel no JPEG decode buffers
ARGB8888 32 four bytes per pixel yes everything modern
RGBA f32 128 four floats yes GPU pipelines, HDR

Three details in that table have consumed entire careers.

First, RGB565 gives green the extra bit. The eye is most sensitive to green, so when sixteen bits must split three ways, green wins. Blending 565 means unpacking three differently sized fields, blending each, and packing them back, and there is a beloved trick that does two channels at once by spreading the fields apart inside a 32 bit register so their carries cannot touch. People shipped entire games on that trick.

Second, byte order. A format name like ARGB8888 describes the value as a 32 bit integer, but memory has an opinion of its own, and on the little endian machines everything now runs on, the integer 0xAARRGGBB lands in memory as the bytes BB, GG, RR, AA. So the same buffer is “ARGB” to someone thinking in integers and “BGRA” to someone reading bytes. Neither is wrong. Entire afternoons vanish here, usually into pictures where people have gently blue faces.

Third, indexed formats are not dead. GIF is indexed. PNG8 is indexed. Which means the 1990s problem of “you cannot blend indices” is a 2026 problem for anyone who renders animated GIFs, and the solutions are still the 1990s ones: blend in true colour first, then map the result to the palette at the end, or precompute the mapping. Keep that thought for the end of the article.

Which way is up

You would expect at least the direction of the rows to be settled by now. It is not.

System Row 0 is at Why
Nearly everything the top matches reading order
BMP and Windows DIBs the bottom OS/2 heritage, positive height means bottom first
OpenGL textures the bottom maths convention, y increases upward
Screen scan out the top electron beams started at the top

A BMP with a positive height stores its bottom row first, and the way you ask Windows for a top first DIB is to declare a negative height, which is exactly as pleasant to discover in a debugger as it sounds. None of this changes the blend function. All of it changes which two rows you hand to it, and a vertically flipped composite is the classic symptom of two libraries disagreeing about up.

Stride returns here too. Windows DIB rows are padded to four byte boundaries. A 3 pixel wide RGB888 image occupies 9 bytes of pixels and 12 bytes of row. Assume 9 and the shear stripes come for you.

There is one more layout that deserves a mention because it shaped a decade: planar. The Amiga and EGA did not store a pixel's bits together. They stored bit 0 of every pixel in one plane of memory, bit 1 in another, and so on. A single pixel was scattered across five or six places. Wonderful for the blitter's logic operations, miserable for arithmetic, and the reason the industry stampeded to chunky formats the moment memory allowed.

The integer problem

Now the arithmetic. Alpha arrives as a byte, 0 to 255, and the formula wants 0 to 1. The natural translation of s*a + d*(1-a) is:

out = (s*a + d*(255-a)) / 255

Correct, but that divide by 255 sits in the hottest loop in the program, and integer division was, for most of computing history, spectacularly slow. The tempting shortcut is shifting right by 8, which divides by 256, and which is wrong by just enough to matter: white blended over white comes out 254, and repeated blends drift visibly darker.

Jim Blinn published the fix, and it is three operations:

t = s*a + d*(255-a) + 128
out = (t + (t>>8)) >> 8

That is an exact, correctly rounded division by 255 for the whole input range, done with two adds and two shifts. Numbers worth checking by hand:

s d a Naive >>8 Blinn Exact
255 255 255 254 255 255
255 0 128 127 128 128
200 40 100 102 103 103

One more refinement and the classical toolkit is complete: premultiplied alpha. Store every pixel with its colour already multiplied by its alpha. Over then simplifies to:

out = s + d*(255-a)

One multiply instead of two, but the real reasons run deeper. Premultiplied pixels can be filtered and scaled without dark fringes creeping in from transparent neighbours, and composites of composites keep working, because the operator becomes associative. Every serious compositor works premultiplied internally. The classic giveaway that something does not is a grey halo around antialiased edges on a transparent background.

The gamma trap

One more trap, and it is the one that quietly degrades the most images. The byte 128 does not mean half as bright. Stored image values are gamma encoded, roughly the physical brightness raised to the power 1/2.2, because that spends the precious 256 levels where the eye can see them. Blending is physics, and physics happens in linear light, so strictly speaking every blend should decode, mix, and re encode.

What happens if you skip that and blend the encoded bytes directly, mixing full red with full green at 50 percent:

Approach Result bytes What the eye sees
Blend encoded bytes (128, 128, 0) dark muddy olive
Blend in linear light (188, 188, 0) the yellow you expected

Nearly everything on your screen right now takes the first row. Browsers blend in encoded space, most UI toolkits do, and at small alpha differences on similar colours nobody can tell. It becomes visible exactly where text antialiasing lives, thin bright shapes on dark grounds, which is why so many rendering systems special case text and why dark mode made a twenty year old compromise suddenly noticeable. The honest engineering position: know which space you are blending in, and choose it on purpose rather than by default.

Blending in JavaScript

The browser gives you two levels. The high level is globalCompositeOperation on a canvas context, which is the Porter and Duff table by name: it accepts source-over, multiply, screen and friends, and the GPU does the work.

The low level is getImageData, which hands you honest bytes:

function blendOver(dst, src) {
  const d = dst.data, s = src.data;   // Uint8ClampedArray, RGBA, always
  for (let i = 0; i < d.length; i += 4) {
    const a = s[i+3];
    if (a === 0) continue;
    const na = 255-a;
    let t;
    t = s[i]*a   + d[i]*na   + 128; d[i]   = (t+(t>>8))>>8;
    t = s[i+1]*a + d[i+1]*na + 128; d[i+1] = (t+(t>>8))>>8;
    t = s[i+2]*a + d[i+2]*na + 128; d[i+2] = (t+(t>>8))>>8;
    d[i+3] = Math.max(d[i+3], a);
  }
}

Three things the platform decided for you. The array is RGBA in that byte order on every machine, so the endianness swamp is drained before you arrive. The array is Uint8ClampedArray, so out of range writes saturate instead of wrapping, which forgives additive blending its overflow. And the canvas stores premultiplied pixels internally while getImageData hands you straight alpha, so a round trip through a nearly transparent pixel is lossy: write alpha 3 with colour 200, read back, and the colour has been quantised to one of the few values that survive multiplying by 3 and dividing again. Browsers are allowed to do this by spec. It is not a bug, it is a format conversion you did not order.

Blending in C

C is where the format stops being somebody else's decision. You get a pointer, a stride, and a promise about layout that you had better believe:

void blend_over(uint8_t *dst, size_t dstStride,
                const uint8_t *src, size_t srcStride,
                int width, int height)
{
    for (int y = 0; y < height; y++) {
        uint8_t *d = dst + (size_t)y*dstStride;
        const uint8_t *s = src + (size_t)y*srcStride;
        for (int x = 0; x < width; x++, d += 4, s += 4) {
            unsigned a = s[3];
            if (a == 0) continue;
            unsigned na = 255-a;
            unsigned t;
            t = s[0]*a + d[0]*na + 128; d[0] = (t+(t>>8))>>8;
            t = s[1]*a + d[1]*na + 128; d[1] = (t+(t>>8))>>8;
            t = s[2]*a + d[2]*na + 128; d[2] = (t+(t>>8))>>8;
            if (a > d[3]) d[3] = (uint8_t)a;
        }
    }
}

The shape of the loop is the lesson: rows outside, columns inside, pointers rebased from the stride each row. That order is not style, it is the cache. Pixels adjacent in x are adjacent in memory, so this loop streams; swap the loops and every access is a stride apart and the same function runs many times slower on the same data.

This scalar code also marks the doorway to the modern era. The next step, which every production blitter takes, is to process pixels in groups with SIMD instructions, sixteen or more at a time, and on the GPU the blend left software entirely: the over operator is configured into the output stage as state, not written as code. The arithmetic being configured is still exactly the table above.

Blending in C#

C# spent its first fifteen years doing this with Bitmap.LockBits, an IntPtr, and an unsafe block, which is to say doing it in C with ceremony. Modern C# has Span<byte>, which keeps the bounds checking and the safety and, through the JIT, most of the speed:

static void BlendOver(Span<byte> dst, ReadOnlySpan<byte> src)
{
    for (int i = 0; i < dst.Length; i += 4)
    {
        int a = src[i+3];
        if (a == 0) continue;
        int na = 255-a;
        int t;
        t = src[i]*a   + dst[i]*na   + 128; dst[i]   = (byte)((t+(t>>8))>>8);
        t = src[i+1]*a + dst[i+1]*na + 128; dst[i+1] = (byte)((t+(t>>8))>>8);
        t = src[i+2]*a + dst[i+2]*na + 128; dst[i+2] = (byte)((t+(t>>8))>>8);
        if (a > dst[i+3]) dst[i+3] = (byte)a;
    }
}

Libraries like ImageSharp and SkiaSharp wrap all of this, and most days you should let them. The reason to know the loop anyway is that the failure modes of the wrapped world are the failure modes of this loop: a fringe means straight and premultiplied alpha got mixed up, a muddy edge means the blend ran in the wrong colour space, a shear means somebody lied about stride, and an upside down result means the two of you disagreed about which way is up.

Why we care

Our countdown timers are delivered into email as animated GIFs, and GIF is an indexed format. A frame may use at most 256 colours from a palette. So the renderer lives on both ends of this article's timeline at once: it blends antialiased digits over backgrounds in full colour, exactly the arithmetic above, and then it must express the result in the 1980s terms of a palette, exactly the constraint that TRANMAP was invented for.

The design of that palette is its own story, and the short version is that we build it from the colours the customer actually chose and the blends between them, rather than from a fixed universal cube, so that a two colour design stays exactly two colours plus its own edge ramps. But that is another post.

The blend function is a good place to end because it never went away. It survived palettes, planes, endianness, gamma and the GPU, and every one of those eras left a mark you can still trip over this afternoon. One multiply, one add, forty years of context.