Grading by Distance

A colourist who wants a subject to stand forward grades it apart from its background: warmer and a little punchier in front, cooler, darker and flatter behind. To do that in an editor, you draw a mask, and the mask is nearly always an ellipse. Depth Grade does the same job with the depth map instead of the ellipse. It is the first of the Rake Plus Pro tools, the four that sit behind one in-app purchase while Clear, Relight, Air and Lens stay free, and this article is about the two things it had to learn before it beat the ellipse: how much of the scene counts as subject, and where the subject’s edge really is.

What an ellipse can’t know

A radial grade knows nothing about the scene: the floor in front of a person is as much “outside” as the wall behind them, and a head that pokes past the curve is graded as distance. Depth Grade weights every pixel by how far behind the subject it is, in inverse depth, so the subject and everything in front of it take the near grade and the far grade comes in smoothly with distance.

The grade itself is ordinary. Each zone has warmth (a blackbody tint from about 8000 K to 3900 K, luminance kept), exposure in stops, saturation, and contrast as a power curve about middle grey. The app’s four looks are pairs of zones; these are their far halves.

case .recede: return GradeZone(warmth: -0.4, exposure: -0.6, saturation: -0.35, contrast: -0.35)
case .pop: return GradeZone(exposure: -0.3, saturation: -0.7, contrast: -0.1)
case .dusk: return GradeZone(warmth: -0.9, exposure: -1.3, saturation: -0.3)
case .fade: return GradeZone(exposure: 0.35, saturation: -0.5, contrast: -0.7)

Everything interesting is in the number that decides, per pixel, how much of the far zone to use.

The kill test

The tool would have died if grading by distance looked the same as a radial grade, or if it left halos or cold outlines at the subject’s edge. The rig applied one look through two masks: the depth weight, and an ellipse fitted around the subject and feathered outward. The ellipse was drawn generously, the way a careful hand draws it, around Apple’s Portrait matte where the photo has one and around the subject’s own depth where it doesn’t. The photos were the nine used for the earlier tools, four with Portrait depth, among them a man standing in a café, and five with estimated depth. The results were laid out side by side for each photo: the photo, the radial grade, the depth grade and the two masks.

Depth won on nine of nine. The ellipse failed in three ways. Where it was wider than the subject, the wall just behind kept the warm grade: a bright ring around the shoulders of a woman at a brick wall, where the depth grade cooled the wall evenly up to her hair. It couldn’t tell the ground in front from the wall behind, so the café man’s floor, the laptop’s desk edge and the hand holding the ice cream all took the far grade. And distance that happened to fall inside the ellipse was left alone: the room past the café man’s shoulders, the path behind the cat.

Most of the numbers the rig prints are scored against the depth itself, which makes them a check on the ellipse’s geometry rather than independent evidence. The café man is the one photo with independent truth, Apple’s person matte, and on him the numbers are blunt.

The real competition is not a radial filter, though. Lightroom has Select Subject, Select Background and a depth range mask for photos that carry depth, and that comparison needs Lightroom on a device. It has not been made. What Depth Grade offers over those masks is one tap and a ramp that follows distance, rather than a switch between subject and background; the kill-test doc calls that a thinner moat than Lens’s.

How much is subject

The first two versions looked right on the vase and wrong elsewhere. At first the depth behind the focus point that still counted as subject was a share of the scene’s range at 0.06. That held about 5 cm at 1 m, and the back of a head and the rim of a round subject took the far grade.

The obvious repair is to hold a distance in metres. It doesn’t work, for a reason the first series met early: a Portrait map is metric only up to a scale that changes from photo to photo. The café man stands about 2 m from the camera and his map reads 0.8 m, with the back of the room at 1.1 m. A hold of 30 cm, sensible for a person, held the whole room.

What the code holds now is a share of the inverse-depth range between the subject and the farthest point in the frame, 0.2 of it by default. A share of that range carries no units, so the per-photo scale cancels out of it. The settings carry the history as a comment:

/// How far behind the focus point is still subject, as a share of the inverse-depth range
/// between it and the farthest point. Not metres: a Portrait map is metric only up to a
/// scale that varies by photo (a man standing 2 m away reads 0.8 m, the café behind him
/// 1.1 m), and 30 cm held the whole room. At 0.06 the back of a head and the rim of a round
/// subject took the far grade.
public var hold: Float = 0.2

Behind the hold, a second share, the ramp, decides how quickly the far grade arrives: 0.5 of whatever range is left. Both are sliders in the app, under Falloff, as Hold and Ramp. The weight is a smoothstep between them, and anything nearer than the subject is never graded as distance at all, which is why the café man’s floor stays warm.

let ds = s.focus.map { lens.focusDisparity(at: $0) } ?? (1 / frame.subjectDepth)
let start = ds - max(s.hold, 0) * max(ds - lens.minDisparity, 1e-4)
let span = max(s.transition, 0.02) * max(start - lens.minDisparity, 1e-4)
let raw: [Float] = disparity.map { d in
    let x = min(max((start - d) / span, 0), 1)
    return x * x * (3 - 2 * x)
}

Whose edge

The second failure was finer and more annoying: a thin, cold outline around the subject, as if the far grade had been traced along the rim with a pen.

An earlier version graded on Lens’s disparity, and that drew the outline. Lens deliberately pulls the near side of every depth edge in by two pixels, since a depth map’s edge sits a few pixels outside the subject’s and a strip of background taken as subject glows when blurred. For a blur, erring toward the far side is safe: the blurred foreground’s own spill covers what it loses. For a grade nothing covers it. The two pixels Lens gave to the background took the background’s grade, and the subject wore them as a cold line.

So Grade keeps Lens’s scene for its matte handling and its focus lookup, and grades on the map’s own edge unless there is a matte:

let far = lens.minDisparity
disparity = lens.alpha != nil ? lens.disparity : frame.depth.map { $0 > 0 ? 1 / $0 : far }

With a Portrait matte the edge is the matte’s, and it is used as it stands. Without one, the weight is pulled onto the photo’s own edges with a guided filter, the same idea that snapped the depth map to the picture in the first series. A guided filter would only soften a matte’s edge, so that path skips it.

Where it sits in the stack

On the GPU the grade is one more step in the composite that Clear, Relight and Air already share. It runs after Air and Weather’s veil and before Lens, so the blur mixes graded colour, as a real lens would mix whatever light reached it:

c = c * v + g.veil.rgb * (1.0f - v);
c = gradeZone(c, g.near, g.gamma.x) * (1.0f - k) + gradeZone(c, g.far, g.gamma.y) * k;
return clamp(c, 0.0f, 1.0f);

The weight map k is made on the CPU and sent in the g channel of the texture that already carried Clear’s transmittance, so it rides through the same guided upsample and keeps to the same silhouettes. That texture became rg32Float for Grade, and rgba32Float when Weather’s veil joined it in b. The map is remade only when the subject, Hold or Ramp changes; a colour change only resets the two zones. Measured against the CPU reference, the GPU grade is a mean of 0.16 of a step off, with a worst pixel 1/255 out.

On the Mac, building the scene took 41 to 93 ms on the CPU at the working size, most of it Lens’s matte fields. How that behaves on a phone has not been measured.

Limits, and the purchase

The kill test also wrote down what it didn’t fix. A subject off the centre has to be tapped, so Depth Grade reuses Lens’s focus tap. A landscape with no subject gains least: on a forest path the grade is a near and far split that a graduated filter comes close to. And estimated depth softens the edge a little in hair, not visible at full size in this look, though a stronger one would show it.

Depth Grade was also the tool that got the purchase working. Pro tools are free to try in the preview; sharing with one switched on opens the unlock, which also offers to share without them. In the Simulator, a test purchase unlocked at once.

What the tests hold

• On the synthetic scene, with the focus on the sphere, the sphere’s mean far weight is below 0.05 and the wall’s, everything with depth beyond 4.5, is above 0.95.
• With no lights, air or haze, the stacked GPU composite with the Dusk look is within a mean of one 8-bit step of GradeRenderer.render, and with Grade switched off again it is the photo, no pixel more than two steps off.
• A neutral zone changes no colour by more than 1e-4.
• Warmth at −1 and +1 keeps a grey’s luminance to within 1e-3, and full warmth makes red exceed blue.
• The kill test: depth beat a radial grade on all nine photos, compared side by side; on the café man, the one photo with an independent matte, the far grade on the floor in front was 0 with depth and 0.93 with the ellipse.

A grade changes the colour of what is already there. The next article puts something new into the scene, rain and snow falling at real distances, and has to hide every drop that would be behind the subject.