A Photograph That Moves

Every tool so far has changed a still. Motion makes a short clip from one instead: the camera pushes towards the subject, sways from side to side, or pulls back while the lens zooms in. The plan named one risk, tearing at the subject’s edges, and expected that only Portrait photos with Apple’s person matte would survive it. The kill test passed on all eight photos it tried, and the reason was a second source of outlines the plan had not counted on.

One sheet, the cheap way

The usual way to move a camera through a still is to treat the whole photo as one sheet and warp it by its depth map. For each pixel of the new frame, six rounds of fixed-point iteration on the depth find the source pixel that lands there.

A subject’s edge is not smooth. The depth steps there from near to far, and a pixel on the step is half subject and half background. Warped as one sheet, the edge stretches like rubber into thin doubled contours. In the kill test this one-sheet warp is kept as the control, and it tore every time: round the vase’s neck, the ice cream, the hair of the woman at the brick wall, the head of the man outdoors, and the cat’s cheek and ears. The beach woman’s hair trailed a loop.

Two layers

Motion cuts the photo into two layers. The subject is one, with a coverage value for each pixel. The background is the other, with the part the subject hid filled in. Each frame re-projects both for the moved camera at their own depths and lays the subject over the background.

The subject’s colour is the part that took three tries. Laying the photo itself over the background counted the background twice at the soft edge. Solving for the subject’s own colour meant dividing by a small coverage at the edge of the band, which had to be clamped and lost it. What works is to carry the subject premultiplied: the photo minus the background’s share.

/// The subject layer's colour, premultiplied: photo − background·(1 − a), so photo = this +
/// background·(1 − a) and the unmoved frame is the photo exactly.
foreground = (0..<n).map { i in frame.albedo[i] - bg[i] * (1 - a[i]) }

let c = fg + bg * (1 - a)
return SIMD3(max(c.x, 0), max(c.y, 0), max(c.z, 0))

Where the camera has not moved, the two layers add back to the photo exactly, which is what one of the tests checks.

Filling what the subject hid

When the subject moves, it uncovers background the camera never saw. Motion does not invent it. It fills the hole by stretching the background round it inward, with the push–pull fill that Lens uses behind a subject: average the known pixels down level by level until the hole is gone, then bring the coarse levels back up and keep the known pixels where they were. Only pixels well clear of the subject are counted as known.

Lens’s version enlarges each coarse level by copying. Behind a moving subject the fill came out in blocks. Motion has its own version that enlarges bilinearly, and Lens keeps the old one.

let coarse = smoothPushPullFill(cv, weight: cwt.map { min($0, 1) }, width: cw, height: ch)
return (0..<v.count).map { i in
    let x = i % w, y = i / w
    let up = sampleBilinear(coarse, cw, ch, SIMD2((Float(x) + 0.5) / 2, (Float(y) + 0.5) / 2))
    let k = min(max(weight[i], 0), 1)
    return v[i] * k + up * (1 - k)
}

Each fix exposed the next

After the blocks came a ghost: a faint outline of the subject’s rim left standing on the background as the subject moved away. Every mask stops a pixel or two short of the photo’s own edge, where the last pixels are subject and background blended, and that rim stayed with the background. The subject layer now takes a feathered band round its mask, two pixels for Apple’s subject lift and wider for a cut from depth, and the background is filled under all of it.

The next was the mask itself. A subject cut from depth, everything clearly nearer than the background, sits well inside the real edge. Where a photo carries Apple’s person matte, that is the outline. For every other photo, SubjectMask.lift asks Vision for the same subject lift behind “Lift subject from background”, on the device, and picks the instance under the point the user tapped.

let request = VNGenerateForegroundInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: photo, options: [:])
guard (try? handler.perform([request])) != nil, let result = request.results?.first,
      !result.allInstances.isEmpty else { return nil }

The last was the subject drawn again beside itself, in parallel outlines. Warped by the depth inside it, the reverse lookup found the subject’s edge more than once wherever its own depth stepped. So the subject moves as a flat card at its own distance. Over a camera move of a few centimetres its relief would amount to a pixel or two, and only a strong Push shows the flatness.

Three moves

Each move runs out and back, eased, so the clip loops. Sway moves the camera sideways, Push moves it towards the subject, and both shift the image so the subject stays where it was in the frame. Vertigo is Hitchcock’s: the camera pulls back while the lens zooms in by exactly the ratio of the distances, so the subject keeps its size and the background swells behind it. The first version went the other way, in and wider, which needs more picture than the photo has; the edges smeared in from the border.

case .vertigo:
    offset.z = -0.3 * S * s.amount * ease
    // The subject keeps its size: focal length × (distance after / distance before).
    zoom = (target.z - offset.z) / target.z
}
// A slight crop, so a sideways move never needs picture from beyond the photo's edge.
zoom *= 1 + 0.04 * s.amount

From the finished picture

Motion is the last tab, and it moves whatever the other tools made. That order mattered. The first fill was taken from the photo as shot, so a relit subject moved away from a patch of unlit wall, and a halo of it showed round them. Every other tool is now applied at the clip’s size first, and the fill behind the subject is taken from that finished picture.

// The fill comes from the finished picture itself, brought down to the working size.
let small = scene.filled(linearPixels(photo, width: ww, height: wh))

The layers are made once on the CPU at the working size. MetalMotion then draws each frame on the GPU at the clip’s size, one dispatch a frame, and writes it to an H.264 movie through AVAssetWriter. On the GPU the subject is plainly a card: one disparity for all of it.

// The subject: a card at one distance.
float2 fs = q;
for (int k = 0; k < 3; k++) fs += q - landM(fs, u.shift.z, u);
float a = bilinear(alpha, fs).r;
float3 c = a > 0.001f ? max(bilinear(foreground, fs).rgb + bg * (1.0f - a), 0.0f) : bg;

The numbers recorded are all from the Mac. Building the layers took 55 to 100 milliseconds at the working size on its CPU, and the CPU reference about 20 milliseconds a frame at the working size, which is why export belongs on the GPU. There, a 1440 × 1920 clip of four seconds took about 1.8 seconds, and the GPU’s frame differs from the CPU’s by a mean of 0.04 of a step. Export time on a phone and the subject lift on a phone are both still listed as open; the Simulator check, a Vertigo preview of the relit café man that played cleanly, used that photo’s own matte.

In the app, Motion is a Pro tool with Push, Sway or Vertigo, Amount, Length and a tap for the subject. It has no on/off switch, since it changes no still. Preview makes a 720-pixel clip that loops in a sheet; Share Video makes it at up to 1920 and asks for the unlock.

The kill test

Eight photos: the café man with Apple’s matte, the vase and the ice cream with Portrait depth, and five with estimated depth: the woman at the brick wall, the man outdoors, the cat, the girl on the boardwalk and the woman on the beach. Each was rendered in Sway, Push and Vertigo, layered and as one sheet, and judged by eye on paired stills of Sway and Vertigo at their farthest, a close-up of the cat’s cheek, and a 30 fps movie of every move. The control was the one-sheet warp; no other app was compared.

Layered, every edge held: hair against the brick wall, the cat’s whiskers, the man’s curls. Where the subject uncovers background, the fill shows as a soft band, green beside the cat’s chin in Vertigo, pale beside the girl’s head and the beach woman’s hip. It reads as out-of-focus background, which it is. The verdict was continue.

Three limits were measured and left. The uncovered background has the right colours but no texture, hidden by gentle moves and visible at the largest amount. The subject is flat, so a strong Push shows no parallax inside it. And where there is no matte and the subject lift finds nothing, the cut falls back to depth, with its mask inside the edge.

What the tests hold

• On a synthetic sphere before a wall, all three moves keep the subject within half a pixel of where it was. In Vertigo, a point 40 pixels to the side at the subject’s depth stays 40 × (1 + 0.04 × amount) pixels away, to within one, while the same spacing on the wall behind grows past 44.
• At the working size, the GPU’s Vertigo frame at amount 1.5 matches the CPU reference to a mean below one step.
• With the camera unmoved, the frame is the photo to within 0.02.
• The kill test: layered held its edges on 8 of 8 photos, and the one-sheet warp tore them.

Looking back

The first series took one depth map a long way: reading it, estimating one where a photo had none, turning it into surfaces, and lighting those with three lights, cast shadows and four looks, without ever losing the original. This one started again from a binary and asked what else the same map could carry: fog, haze taken out, focus moved, a grade by distance, rain, window light, and a camera move.

Each tool had a kill test before it went into the app, and one failed it: the sun’s cast shadows were right in two photos of ten, and the window took its place. Much of the rest was work at edges: where a depth map steps from a subject to what is behind it, and who owns the pixels on the step. For Motion the answer was an outline from a matte or the subject lift, a flat card and a background filled from the finished picture, and that is where both series end.