Four Widths, Chosen by Eye

A focus stack in Sharp has one setting that decides how the picture looks, and it is the one setting nothing in the app has been able to choose. The app calls it Detail size; the engine calls it the radius of the sharpness window. Three ways of working it out from the frames were built and all three failed. Sharp 1.2 stops trying to guess: it stacks a small piece of your bracket at four widths, puts them side by side, and lets you pick the one that looks right.

The one number

To decide which frame a pixel comes from, the stacker measures how much detail sits around it in every frame: the squared Laplacian, averaged over a square window. The frame with the most detail there wins the pixel. The width of that window is the radius.

/// How much detail sits around each pixel: the squared Laplacian, averaged over a window.
public enum Sharpness {
    public static func map(of buf: PixelBuffer, radius: Int) -> [Float] {
        …
        return boxAverage(energy, w: w, h: h, radius: radius)
    }
}

The window has to be about as wide as the defocus blur. Narrower than the blur, and the blank area beside a hard edge goes to the wrong frame: in a blurred frame the edge’s contrast has spread onto that blank area, while in the sharp frame there is nothing there at all. So the blurred frame scores higher, wins a ring of pixels around every edge, and the ring is a halo. Wider than the blur, and a thin object such as a pin, a bristle or an antenna dominates the window for some distance around it, dragging the background beside it to the object’s own frame. The background goes soft in a band that traces the object.

The right width is not a matter of taste, and it is not constant. Judged by eye at 100% on the test brackets, a set of pin headers on a circuit board wanted 6 to 12 pixels and smeared behind the pins at 26. An insect in amber wanted about 30, and at 10 its legs and antennae got white outlines. Flies in amber were clean at about 10. A simulated bracket with blur reaching 22 pixels wanted about 26, and at 8 every block of text in it had a halo. Sets of similar size and subject, three times apart.

Three measures that failed

Each failure is written down in the repository, so that a fourth attempt does not repeat them.

The first watched the stacker’s own decisions: grow the window until the map of which frame wins each pixel stops changing, on the theory that halo rings keep moving until the window covers the blur. It never stopped changing. Noise flips pixels at every width, so the rule returned the widest window tried on every bracket: 94 pixels for the pin headers, which wanted 6 to 12.

The second compared whole frames, looking for the Gaussian blur that turns the sharpest frame into the softest by matching their total edge energy. It measured almost nothing, because every frame in a focus bracket has something in focus, so the totals barely differ. It returned 4 pixels for the scene whose blur reaches 22.

The third made the comparison fair: an all-sharpest composite against an all-softest one. That at least put the brackets in a sensible order, but it still read the blur two to three times too small, and it could not tell apart two sets whose right answers differ threefold. A single energy figure falls away too fast with blur to pin the scale down.

The same record ended with a short list of what to try next, and the first item on it was the plainest: show, do not guess. A photographer can see a halo in a second, where no measure could.

Four widths

The widths tried are fixed, and there are four of them.

/// Widths to try. They span the threefold range seen across real brackets, and each is about
/// three quarters again the last, which is roughly the smallest step anybody can see.
public static let candidates = [5, 9, 16, 28]

Each width is about three quarters again the last, which the comment gives as roughly the smallest step anybody can see. The engine’s default, 10, sits beside the 9.

A piece decoded once

The obvious way to make four previews is to run the stacker four times on a crop. For a RAW bracket that would decode every frame four times, and the decode is most of the cost. Cropping does not make it cheaper either: an earlier measurement found that a quarter-scale render of a RAW file costs as much as a full one, because the demosaic happens at sensor resolution whatever is asked for.

So DetailSize.previews decodes each frame once, cuts out the piece, and writes it to a 16-bit TIFF in a scratch folder. All four widths are then stacked from those small files.

let image = try io.load(url)
let extent = image.extent.integral
// Top-left pixels to Core Image’s bottom-left ones, clipped to the frame.
let rect = CGRect(x: extent.minX + crop.minX, y: extent.maxY - crop.maxY,
                  width: crop.width, height: crop.height)
    .intersection(extent).integral
guard rect.width >= 16, rect.height >= 16 else { throw SharpError.noFrames }
let piece = image.cropped(to: rect)
    .transformed(by: CGAffineTransform(translationX: -rect.minX, y: -rect.minY))
let pixels = io.render(piece, extent: CGRect(origin: .zero, size: rect.size))
try io.write(pixels, to: file, format: .tiff16)

The crop arrives the way a photographer points at a picture, from the top left; Core Image counts from the bottom left, so the rectangle is flipped before cutting. The piece is then moved to the origin, which the aligner assumes of every frame it is given.

Then the loop that makes the four pictures, in which only one thing changes:

for (n, radius) in radii.enumerated() {
    var each = options
    each.tool = .focus
    each.radius = radius
    each.cacheFrames = false
    results.append((radius, try Stacker(options: each, io: io).stack(urls: pieces).image))
}

Alignment, how decisively the merge favours the sharpest frame, the consistency filter on the map of winners and the noise reduction all come from the photographer’s own settings. The previews differ in the width and nothing else, which is the only way a choice between them means anything.

In the app

Under Detail size in the focus settings there is a button, Choose by Eye…. It opens a sheet showing the middle frame of the bracket with a 640-pixel square on it, which moves to wherever you tap. The sheet asks for a place where an edge meets the background: a petal against the dark, a bristle against the sky. Try four sizes here stacks that square at the four widths and lays the results side by side, the current setting marked “now”. Tap the one with no halo along the edge and no smear beside it, and the button at the top becomes Use 16 px, or whichever width was tapped.

The 640 pixels are a compromise written into the picker: big enough to hold an edge and the background beside it at the widest size tried, small enough to stack in seconds.

The frame you tap on is not a thumbnail from ImageIO. ImageIO turns a portrait JPEG upright, and Core Image, which the engine reads frames through, does not. A square chosen on an ImageIO thumbnail of a portrait frame would be cut from somewhere else entirely. So the overview is rendered by Core Image, scaled down from the same pixel grid the engine reads.

let extent = image.extent.integral
let scale = min(1, maxSide / max(extent.width, extent.height))
let small = image.transformed(by: CGAffineTransform(translationX: -extent.minX, y: -extent.minY)
    .scaledBy(x: scale, y: scale))

This was checked in the iPad simulator by tapping through it: the sheet opens on the middle frame, the four previews appear, choosing 16 px sets Detail size to 16 px, and the result already on screen is marked “Settings changed since this ran”, since it was made at the old width. No timings on a real iPad have been taken yet.

The picker is free. Sharp’s pricing has one rule, that quality is never what is being sold back to you, and a way to stop haloes is quality.

From the command line

The same function is behind a command in the sharp tool, which writes one PNG per width:

sharp sizes <frames…> [--crop x,y,w,h] [--out dir]

Without --crop it takes a 640-pixel square from the centre of the first frame, and it stacks at the default focus settings. On nine frames of flies in amber, a public focus bracket, the four widths take 2.1 seconds on a Mac.

What the tests hold

• A 100 × 80 crop of a 200 × 160 synthetic four-frame bracket, stacked at a radius of 9, comes back 100 × 80, and matches the whole stack’s same region to within 0.003 in every channel of every pixel at least 12 pixels in from the crop’s edges.
• In that test the options passed in carry a radius of 1 and the preview is asked for at 9, so it also holds that the width tried is the one asked for, not whatever the settings held.

That is the property the whole picker rests on: choosing on the piece is choosing for the photograph. The next article leaves focus for exposure, and for Blend, the tool new in 1.2.