Fog That Knows How Far

Air is the first tool Rake Plus added to the old relighting. It puts fog, haze or mist into a photo that thickens with real distance, and it lets a sun shine through that fog so the scene’s own branches and frames cut it into shafts. Both are the depth map put to a new use: the fog needs to know how far away each pixel is, and the shafts need the same march toward a light that cast the shadows in the first series.

Fog is a distance, not a gradient

The model is a uniform medium in front of the camera. A surface at distance d is seen through the air between the fog’s start and itself. Its own light is dimmed by a transmittance T, and the fog adds its colour in proportion to what was taken away. The CPU reference says it in four lines:

Swift

let sigma = s.density * 2 / max(scene.subject, 1e-3)
let start = s.start * scene.subject
…
let fogLen = max(0, d - start)
let t = exp(-sigma * fogLen)
var e = fog * (1 - t)

The result is composited as the photo times T, plus the in-scattered light. Both settings are measured in the subject’s distance, so the same setting means the same thing on a portrait and on a street. Density runs from 0 to 1, and its comment pins down the top of the range: at 1, air one subject-distance deep past the start hides 86 %. Start runs from the camera at 0 to the subject at 1, defaults to halfway, and its slider goes on past the subject to 2.

sigma at density 1 2 / subject past start by 1 subject e^−2 = 0.135, so 86 % hidden default density 0.45 sigma = 0.9 / subject default start 0.5 subject at 1: T = e^−0.45 = 0.64 3 subjects: T = e^−2.25 = 0.11

How much of a surface survives the fog, against its distance along the ray: the default density of 0.45 (solid) and density 1 (dashed), with the fog’s start at half the subject’s distance. Curves from the code above.

Because d is the measured distance, a face in front stays clear while the wall behind it fades. A gradient painted over the photo cannot do that: it only knows how far down the frame a pixel is.

The test against a painted gradient

That difference is what the kill test asked about. Air would die if its fog read as a flat gradient instead of depth, or if its light shafts banded. The command-line rig writes, for each photo, the fog, a thick fog, fog with shafts, the open-sky mask, and a control: the same mean amount of fog painted top to bottom, the way a filter would. The pictures were then compared side by side.

The plan asked for five Portrait photos and five with estimated depth. Only four MIT-licensed Portrait photos with depth were found, a vase, a man in a café, a laptop and an ice cream, so the fifth is still wanted. The estimated set is five scenes from Wikimedia Commons, public domain or CC0: sun through trees at a creek, a window-lit stairwell, a town street and two forest paths.

The fog passed on all nine. On a forest path the distant pines go grey while the grass and bush in front stay crisp, and the gradient control fogs the near dune just as much as the far trees. On the ice cream the hand and cone stay sharp against a fogged forest. The start distance turned out to matter on Portraits: at 0.5 of the subject’s distance the person is lightly hazed, at 1.0 they stay clear. That is why it is a control and not a constant.

Why the fog came out beige

The fog needs a colour, and the obvious source is the far end of the photo, since that is what real haze is already over. The first version took the mean colour of the farthest fifth of the pixels literally. On the forest it came out the colour of the leaves, on the stairwell the colour of the wall: in the kill test’s words, a dirty beige.

Real fog is lit by the sky and reads nearly neutral. So the far colour now contributes only its hue, and only a quarter of it, and the brightness is lifted into a light grey:

Swift

let lum = max(0.2126 * mean.x + 0.7152 * mean.y + 0.0722 * mean.z, 1e-4)
let hue = mean / lum
let target = min(max(lum * 1.6, 0.55), 0.85)
fogColour = (SIMD3(repeating: 1) * 0.75 + hue * 0.25) * target

Seventy-five per cent neutral keeps only a hint of the scene’s cast, and the brightness is held between 0.55 and 0.85, a light grey whatever the scene. Glow multiplies it and Warmth tints it, where 5600 K adds no cast.

A sun that scattered nothing

Shafts are single scattering from one light. Along each camera ray the renderer takes 32 steps through the fog, and at every step it asks whether that bit of air can see the sun, using the same depth-map march Relight uses for shadows. Air that can see it scatters some light toward the camera, weighted forward by a Henyey–Greenstein phase function; air in the shadow of a branch stays dark, and the dark and light streaks are the shafts.

The first attempt treated the sun the way Relight treats a lamp, as a point whose light falls off with distance. Placed several metres back, behind the scene, it scattered about 1 % of the fog’s own brightness, and the shafts vanished. The fix was to stop treating it as a lamp:

Swift

// The sun is far enough away to be treated as parallel light: its radiance does not
// fall off across the scene (a point light several metres back scattered about 1 %
// of the fog's own brightness, and the shafts vanished). Its position still sets the
// direction the light comes from and what blocks it.
let sunPower = s.sun.map { pow(2, $0.stops) * 0.35 } ?? 0

The position still matters; the distance no longer costs anything. The 0.35 is itself a correction. The first defaults, forward scattering at 0.6 and shaft power at 0.6, were too hot near the frame and washed out the Portraits. They are now 0.5 and 0.35.

Beams need gaps

With a sun that reaches, beams appeared where the scene had gaps. On a forest path, a long beam through the gap between two stands, with fine striations from the trunks. At the creek, trunk shadows streaking down through the mist. In the town, the bus throwing a dark wedge through the lit fog. On the vase, a shadow volume through the fog behind it.

The limits were measured and left as they are. An indoor Portrait with a person gets a glow and a darker shadow volume, not beams, because beams need occluders with gaps and a person against a wall has none. The automatic sun, placed at the brightest far, open pixel, is only a starting point: the best results came from placing it by hand, above and behind the gap.

Before you read on

The window-lit stairwell has a roof. With the sun placed above it, what happens to the fog inside?

It floods with light. The march toward the sun stops as soon as it leaves the picture, and whatever it found in the frame is all it has. The roof is out of shot, so the air under it sees an open sky.

Swift

let q = project(s, f)
if q.x < 0 || q.y < 0 { break }
let px = Int(q.x), py = Int(q.y)
if px >= w || py >= h { break }
let j = py * w + px
if scene.open[j] { continue }

Anything leaving the frame counts as open air. It is the same edge-of-the-picture problem the first series ran into with shadows, met here as light where there should be shade.

No bands

Thirty-two steps along a ray is few, and taken at the same spacing for every pixel they show as contour lines across the beam. Each pixel’s steps are instead offset by a fraction from Jimenez’s interleaved gradient noise, so neighbouring pixels sample the ray at different places and the step spacing reads as fine grain. No banding was seen in the kill test, which settled its second question.

On the GPU and in the app

On the Mac’s CPU, eight threads, the shafts took about 40 to 150 ms per working-size frame. The Metal version is a separate shader holding the same maths, and its test against the CPU reference measured a largest difference in transmittance of 1.5e-7 and in the in-scattered light of 0.0008. One composite pass applies Relight’s light, then the air, with one guided upsample to the full photo, and Relight’s 1.0 shader is left untouched.

In the app, Air sits after Relight in the tool switcher, and the first visit turns it on. The fog has Density, Start, Glow and Warmth. Shafts of light add a sun drawn on the photo, which can be dragged past the frame edge, since light so often comes from outside the picture, and pinched nearer or further, with Power, Depth, Size, Warmth and Beam below it. It was checked in the Simulator on the vase: fog, a sun, and a drag that swings the shadow volume.

Three things were not checked. Frame time on a real device is unknown, and the shafts are 32 × 32 depth reads per working pixel. Pinch on the sun has not been tried. And Air on estimated depth in the app could not be seen, because the Simulator’s depth network returns a map of zeros.

What the tests hold

On the synthetic scene, the sphere in front keeps a mean transmittance more than 0.3 above the wall behind it at the default settings.
With a sun on, the GPU air map matches the CPU reference: transmittance within 1e-4 everywhere, in-scattered light within 0.001 on average and 0.02 at worst.
With no air, the stacked composite is Relight’s own composite, to within one level in 255.
The kill test: fog followed depth on nine of nine photos against the gradient control, and the jittered shafts showed no banding.

Air puts haze in by distance. The next article runs the same model backwards, to take it out.