Focus After the Shot

Lens moves the plane of focus after the photograph is taken. It was built last of Rake Plus’s first four tools, because it has the strongest competitor: Lightroom already has a Lens Blur. Its kill test was about one place, the place where refocus effects usually give themselves away, which is hair and fur against a blurred background.

A circle from inverse depth

A thin lens blurs a point into a circle whose diameter is proportional to the difference in inverse depth between that point and the plane in focus. Inverse depth is disparity, the unit a Portrait photo’s depth map comes in (the first series read it), so the blur radius is a straight line in what the file already holds: r = k·|disparity − focus disparity|.

public func circles(_ s: LensSettings, longSide: Int) -> [Float] {
    let focus = focusDisparity(at: s.focus)
    let range = max(maxDisparity - minDisparity, 1e-4)
    let k = s.aperture * 0.02 * Float(longSide) / range
    let band = s.depthOfField * range / 2
    return disparity.map { max(abs($0 - focus) - band, 0) * k }
}

At full aperture, the most out-of-focus part of the photo is blurred by a disc 4 % of the long side across.

The band was not in the first version. With one sharp plane, a tap on a face on estimated depth softened the face: the estimate makes a head deep, the face well in front of the hair, and a single plane cannot hold both. Real lenses have a depth of field, so Lens has one too, 10 % of the scene’s range by default, adjustable as “In focus”.

Gathering what would have spread

A lens scatters every point into its own disc; a GPU prefers each pixel to gather from its neighbours. The cheap gather blurs every pixel over its own circle, and it gets edges wrong. A background pixel beside a sharp head has a large circle, so it gathers the head and smears it outward as a halo. That naive blur is kept in the code, for comparison.

Lens gathers as if it were scattering. A pixel collects the neighbours whose own disc reaches it, with one rule about order. A neighbour nearer the camera spreads by its own circle, so a blurred foreground spills over a sharp subject as it does through glass. A farther neighbour is held to the pixel’s own circle, so a blurred background never reaches across a sharp edge.

// Nearer neighbours spread by their own circle; farther ones are held to ours.
let nearer = d[j] > di + 1e-4
let r = naive ? ri : (nearer ? rj : min(ri, rj))
guard r >= 0.5 else { continue }

The kill test

The test as planned was to move the focus across hair and fur and compare with Lightroom’s Lens Blur at 100 %; Lens would die if Lightroom’s edges were cleaner. Lightroom cannot be run where Rake Plus is built, so the comparison made was against the naive blur, which is the shortcut behind most refocus effects. The Lightroom half is still to be done by hand, and the verdict does not claim it.

The photos are in the repository under samples/lens/, all CC0: long hair against brick, curls and a beard, a cat, and two dogs, with the Portrait samples alongside. A command-line run writes the naive blur and Lens side by side for comparison. Against brick, the naive blur bled the dark hair out into the wall; Lens kept the hair crisp against a smoothly blurred wall. White fur against a white chair glowed outward under the naive blur and stayed clean under Lens. The verdict, on September 23, was continue.

One weakness was written down with it. Fine strands beyond the depth map’s edge are taken for background, which gave the chihuahua’s neck a slight cut-out look. A depth map cannot see a strand of hair; a matte can.

What the bokeh needed

Lens character is the shape of the disc: round or five to nine blades, an anamorphic oval up to twice as tall as wide, a cat’s eye clipped toward the frame edge as a barrel clips the beam, and swirl, which stretches discs around the centre the way old Petzval designs do. A chart of bright points on a far wall, rendered round, with six blades, anamorphic 1.8, and swirl with cat’s eye, read correctly in every shape, after three fixes.

Samples read a prefiltered copy of the image. Ninety-six samples spread over a disc of radius R are each about R·√(π/96) apart, and read sharp, a small bright point was hit by one sample in a few, so every disc came out as grain. Each sample now reads a box-blurred level matched to that spacing, and stands for its share of the disc.

The sample pattern, a golden-angle spiral, is turned by a different angle at every pixel. Fixed, it printed a rosette into every disc. And the search widens for anamorphic ovals, which otherwise reached past their circle and were clipped back into one.

White discs on a cheek

Bright points have to bloom, so highlights are pushed up in linear light before the blur, up to about thirty times. In the app, focused on the wall behind the man in the café photo, a Portrait sample, that boost put white discs on his cheek and collar. The boost now starts at 92 % instead of 75 %, and only where a pixel stands out from its local mean, as a lamp or a glint does and a wall does not. Bloom was switched off near depth edges, where a pixel may belong to either side. The edge erosion above came from the same photo. A light patch on the cheek remained.

The matte

Portrait photos of people carry Apple’s portrait-effects matte: the person, from 0 to 1, at about half the photo’s resolution, drawn to the strand at the hair. Rake Plus loads it with the depth, through AVPortraitEffectsMatte, into DepthFrame.matte. The café photo’s is 1512 by 2016.

Right beside the matte is exactly where the depth map runs over, and sampling the subject’s depth there copied the map’s error faithfully. So LensScene builds two smooth fields, each push–pull filled from where both matte and map are sure:

let margin = max(6, max(w, h) / 80)
let solid = minFilter(al.map { $0 > 0.9 ? Float(1) : 0 }, width: w, height: h, radius: margin / 2)
let clear = maxFilter(al.map { $0 > 0.1 ? Float(1) : 0 }, width: w, height: h, radius: margin)
let fg = pushPullFill(raw, weight: solid, width: w, height: h)
let bg = pushPullFill(raw, weight: clear.map { 1 - $0 }, width: w, height: h)

Near the person, the depth used is the two mixed by the matte; away from them, the map stands, so a table in front keeps its own depth. On the GPU, lensDisparityMatte reads the matte at its own resolution for the output-size depth, so the edge follows the matte and not the coarse map. A test grows a synthetic subject’s depth edge by 6 px; the disparity error in that ring falls from 0.49 to 0.013 with a matte, and the subject keeps its own depth.

The patch that was a lamp

The light patch on the cheek was not the face. It was the lamp’s clipped glints just behind the head. In the gather they were weighted by their own small circle, which outweighed the blurred face around them ninefold. A farther neighbour is seen through the pixel’s own disc, so it now counts as one share of that disc. Bloom is also off within a blur radius of any depth edge, 1/60 of the long side, and on the matte’s soft edge. And only clipped highlights bloom:

guard l > 0.96, amount > 0 else { return c }
let t = (l - 0.96) / 0.04
return c * (1 + amount * 60 * t * t)

At the default 0.5, a pure white point comes back 31 times as bright, enough to fill its disc as a lamp does. At 0.92, a faint reflection in a pair of glasses had bloomed across a face.

Only one sample has a matte; no other openly licensed Portrait photo with one was found. Pets get none, because Apple makes the matte for people only, so the cat and the dogs keep the eroded-edge path.

In the app

Lens runs last on the GPU, after Clear, Relight and Air, over the photo as they left it. Tap or drag to focus, with a focus box; the controls are Aperture, In focus, Blades, Oval, Cat’s eye, Swirl and Glow. Refocus on the café photo and on the vase, and tap to focus, were checked in the Simulator. Frame time on a device, at 96 samples, and Lens on estimated depth in the app were not.

What the tests hold

• Focused on the synthetic sphere at full aperture, the sphere’s interior comes out within 0.02 of the photo.
• The sharp sphere bleeds less into the blurred wall around it under Lens than under the naive blur.
• The GPU matches the CPU reference to a mean under 2.5 steps; the kill test recorded 0.13.
• The café photo’s matte loads at 1512 × 2016 with a person in it, and the depth sample without one loads no matte.
• With a matte, the ring around a 6 px dilated edge has under a quarter of the error it had without, and the subject’s core keeps its own disparity within 0.02.
• With the matte, GPU and CPU agree on the café photo to a mean under 3 steps; the kill test recorded 0.39.
• The kill test’s verdict is continue, against the naive blur, with the Lightroom comparison still open.

The scene Lens builds, with its matte and its two depth fields, turned out to be useful beyond focus. The next article grades colour by distance, and with a matte it grades by the same depth, from the same LensScene.