From Depth to Surface

Knowing how far away something is does not tell you how it is lit. A wall and a nose can sit at exactly the same distance from the camera and take light completely differently, because the wall faces you and the nose does not. Distance is only half the geometry. The other half is orientation, and it has to be derived.

Brightness is about angle

A surface lit at a glancing angle receives the same light spread over more area than one facing the light square on, so it is dimmer. The relationship is the cosine of the angle between the direction the surface faces and the direction the light comes from: straight on is full brightness, forty-five degrees is about seventy percent, edge-on is nothing.

That single rule is most of what makes something look lit rather than brightened. The gradual dimming across a cheek as it turns away from a lamp is a cosine, and the eye reads it instantly as a curved surface in a real place. Brighten the same region uniformly and it reads as a stain on a photograph. So the direction each point faces — its normal — is the quantity that matters, and the depth map does not contain it.

Getting orientation out of distance

What it does contain is enough to work it out. Take a point and its neighbors to the right and below. Those three points, being at slightly different distances, define a small patch of surface, and a patch of surface has a direction. Do that everywhere and you have a normal for every pixel.

There is one step that cannot be skipped. The neighbors have to be turned into real positions in space first, using the camera's field of view, before they are compared. Pixels are not a unit of length — a pixel covers a fingernail up close and a doorway across a room — so differencing depth values directly measures something that changes meaning with distance. Skip the conversion and near surfaces come out wrongly steep, far ones wrongly flat, and a flat wall receding into the distance appears to curve. The error is subtle enough to survive a first look and obvious the moment a light moves across it.

Differencing makes noise louder

Comparing neighbors amplifies whatever error sits between them. A depth map that looks perfectly smooth can carry small amounts of noise, and the difference between two nearly-equal noisy numbers is mostly noise. A depth map that is quiet produces a normal field that shimmers, and shimmering normals under a hard light produce a surface that looks like beaten metal.

So the depth has to be smoothed before the normals come out of it. But smoothing has a cost that is the exact opposite problem: run it across the boundary between a shoulder and the wall behind it and the two get averaged into a ramp, which is to say the edge becomes a slope. Light that slope and you get a soft bevel around every object, as though the whole scene were molded in wax.

The resolution is to smooth within surfaces and not across them — to treat a large jump in depth as a wall the smoothing does not cross. Getting that threshold right is most of the practical work, and it is a compromise rather than a solution: too eager and edges melt, too cautious and the noise survives.

You only have the front

The deeper limitation is that none of this is a model of the scene. It is a single surface, viewed from one place, with a distance at every point. Everything the camera could not see is simply absent. There is no back of the head, no floor continuing behind a chair, nothing at all in the region a person occludes.

This is why relighting from a photo is not the same problem as lighting a three-dimensional scene, and it is where the next article starts, because it is shadows that feel the absence most. A light placed behind a subject wants to know what is behind the subject, and the honest answer is that nobody knows.

What it buys

With normals in hand, a light stops being a brightness adjustment applied in a region and becomes something the scene responds to. Move it and the highlight travels across a face the way a highlight travels. Drop it lower and the shape of a jaw appears, because the surfaces under it have turned away. Nothing in the photograph has been cut out, masked or selected; the picture is simply being treated as a surface, which is what it always was a picture of.