The Depth a Photo Carries

A Portrait photo is two pictures in one file. There is the one you see, and there is a second, much smaller one that records how far away every part of the scene was when the shutter went. The camera builds it by comparing what two lenses see, or by reading the sensor's own per-pixel focus information, and iOS tucks it into the file alongside the photo as an auxiliary image. Open the photo anywhere and you get the picture. Ask for the auxiliary image and you get the shape of the room.

Everything that follows in this series rests on that second picture. It is worth being precise about what it actually contains, because it is not quite the thing people assume.

Disparity is not distance

The number stored per pixel is usually disparity, not depth. Disparity is how far a point appears to shift between two viewpoints, and it is proportional to one over the distance. Near things shift a lot; far things barely move; something at infinity has a disparity of zero.

The conversion to distance is a reciprocal, and the consequence of that reciprocal is the whole practical story of working with these maps. Precision is not spread evenly. A map that resolves distance beautifully across the near half-meter of a face will lump the far wall, the door behind it and the garden outside into two or three indistinguishable values. That is the right tradeoff for the job the camera was doing — it was separating a subject from a background, not surveying a room — but it means the far end of the scene is a guess wearing the costume of a measurement.

It also means you cannot treat the stored values as linear. Averaging two disparity values does not give you the point halfway between them in space. Smoothing a disparity map smooths something other than distance. Anything that needs real geometry has to convert first and work in distance afterward.

It may not be in meters at all

Depending on how the photo was captured, the values may be relative rather than absolute: correct in their ordering and their proportions, but with no scale attached. This face is nearer than that wall by a factor you can trust; how many centimeters that is, the file does not say.

For relighting, relative depth gets you most of the way. Which surface faces which way, and what blocks what, are questions about proportion. But the moment a light has to fall off with distance, an actual scale has to be chosen. Pick one that is too small and the whole scene sits inside the light's falloff, so everything blows out together. Pick one too large and the light behaves like the sun, flat and directionless. Somewhere in between is a scale that makes a photo of a face behave like a face a meter from a lamp, and landing on it is a judgment, not a calculation.

Small, soft, and full of holes

The depth map is a fraction of the photo's resolution — a coarse grid behind a very detailed picture. It has to be enlarged to match, and how that enlargement is done decides whether the light lands on the subject or slightly beside it.

Enlarging it on its own gives soft, rounded edges that do not line up with anything. A depth edge that has drifted a few pixels off the real one puts the boundary between near and far in the wrong place, and a light crossing that boundary produces a shadow that starts just inside a shoulder or just outside a cheek. Nothing else in the picture is wrong; the eye still knows immediately that something is.

The way out is to use the photo to guide the enlargement, on the assumption that a sharp change in the image is where a sharp change in depth is likely to be. The assumption is usually right and occasionally badly wrong — a strong pattern on a flat wall is an edge in the image and not in space — but it is far better than ignoring the photo entirely.

There are also gaps. Shiny and transparent surfaces return nothing usable, thin structures fall between samples, and anything beyond the camera's working range comes back empty. The map arrives with holes in it, and something has to go in them before a renderer can use it.

Why only some of your photos show up

When Rake offers a filtered picker, it is filtering on exactly this: whether the file still carries its depth. That is a property of the file, not of the photo, and it is easy to lose.

The clearest example is AirDrop. With Format set to Most Compatible, the photo is converted to a plain JPEG on its way out, and the auxiliary image does not survive the conversion. What lands on the other device looks identical, is identical as a picture, and has nothing behind it. Setting Format to Current in the share sheet's options keeps the file as it is, depth included.

Plenty of other paths do the same thing quietly — messaging apps that re-encode, exports that flatten, anything that treats a photo as pixels rather than as a file. The photo survives every one of them. The measurement does not.

What you actually have

So the input to everything that follows is a low-resolution, non-linear, possibly unscaled, partially missing measurement of the front surfaces of a scene, which is often lost in transit and which people reasonably assume is just part of the picture.

That sounds like a complaint. It is not. It is an extraordinary thing to find inside an ordinary photo, and it is enough to light a scene convincingly. But every artifact in the rest of this series traces back to one of those five properties, so it is worth knowing them before the lights go on.