The Sun That Stopped

Every tool in Rake Plus had to pass a kill test before it went into the app: a question written down first, a set of photographs, a control, and a verdict of continue or stop. This article is about the one that failed. Sun was planned as the third tool in Rake Plus Pro, a distant sun at any hour throwing long shadows across the scene. On ten photographs its cast shadows were right in two. It was stopped, and a different tool took its place.

What Sun was meant to do

Relight’s lamps sit a metre or two from the subject. The sun is parallel light, and low, it throws shadows many times as long as the things that cast them. The plan asked for two things: long cast shadows from a low sun across a floor in a Portrait photo, and across terrain in a photo with estimated depth.

The light itself was the easy part. Sunlight lands on top of the photo’s own light, which is kept at 35 % and tinted blue as skylight, and the sun’s colour follows its height, orange when it is low and white when it is high:

/// The sun's colour temperature at this height: low sun is orange, high sun is white.
public var kelvin: Float {
    (2000 + 3600 * (1 - exp(-max(elevation, 0) / 15))) * pow(2, -warmth * 0.5)
}

The shadows were the point. The test rendered three hours for each photo, low behind the subject at 8°, low from the side at 12° and higher from the front at 30°, each with cast shadows and, as the control, with the same sunlight and none. The photos were four Portraits (the vase, the café man, the laptop, the ice cream) and six with estimated depth: the town street, and five new CC0 photographs of people and things standing on sand, grass, boards and pavement.

Two of ten

Two worked. A bicycle leaning against a bar front throws its shadow on the stone base behind it, the pillar’s shadow crosses the door, and the crates’ fall on the wall; it reads as late-afternoon sun. On a boardwalk, with the sun from the side, the railing posts lay their shadows across the boards.

The rest were wrong in ways anyone would see at once. On the Portraits, the vase’s wall and table and the café’s walls got jagged dark patches: the 8-bit map’s steps became shadow edges. On the beach the sea wasn’t read as ground, so the waves “stood” and threw blobs across the sand. The bus cast a shapeless patch. In the field, waist-high grass left no ground to cast onto. Even without cast shadows the sunlight was blotchy on the Portrait maps, because directional light shows every wobble in the normals. And one failure was correct: a low sun behind the café man is behind the room’s back wall, and puts the whole room in shade, which is right and useless.

Six fixes, six new failures

The test was not run once and abandoned. Each change fixed one failure and showed the next. The first was the one Shadows That Something Blocks warned about: a depth map has no back to anything. Relight treats everything as solid behind its surface, which is right for a lamp beside a face. Across the ground it isn’t: a person cast a shadow as deep as the scene. So every surface was given a thickness, and the ground was told not to cast at all:

let reach = 6 * S
// Every surface is this thick: about 70 cm for a subject 2 m away.
let thick = 0.35 * S
…
// The ground does not cast.
if r.valid[j] && dot(r.normals[j], scene.up) > 0.75 { continue }
let bias = 0.01 * S + 0.02 * t
let intrusion = q3.z - bias - z
let cone = max(spread * t, 1e-4)
var o = min(max((intrusion + cone) / (2 * cone), 0), 1)
// Past the back of the thing, the ray is in the open again.
o *= min(max((thick - intrusion) / (0.25 * thick), 0), 1)

That second rule came from the second failure. An estimate squeezes a beach 30 m deep into 4 m, so the rebuilt sand rises like a ramp, and a low sun sits below its far end: the sand went black. On a Portrait map the steps striped it instead. Once the ground no longer casts, though, hills can’t shadow each other, and the “across terrain” half of the test can no longer pass.

Four more followed. The ground’s normals wobble by 10° or more, so at a low sun half of them faced away; the ground is now lit as flat. Marched from the ground, a person’s shadow was a wide slab of sand; it is now traced down from each standing pixel, which gives outlines on the boardwalk and blobs wherever the ground is misread. The sky, lit as a surface, blew out orange. And the sun didn’t read as sun until it was 1.5 stops stronger, which fixed the look, not the shadows.

Why it stopped

A cast shadow is the one thing Sun adds that a warm grade can’t, and it needs two things a single depth map doesn’t have: the back of every object, and a true ground plane. The Portrait maps are stepped and shallow, and the estimates squeeze distance. Relight gets away with this because its lamps sit beside the subject and its shadows fall across the subject itself, where the depth is at its best. Sun’s own unit test, a sphere’s shadow on a wall at 5 m, passes: the march is right, and the photographs’ depth is what failed it.

What worked was narrow: a sun on things just in front of a wall, a scene Relight already handles with a key light behind and to the side. So the verdict was stop. The code stays in the repository as the record of what was tried, Sun.swift, its test and the test rig’s --daylight, but it isn’t in the app.

Light through a window

The kill-test doc ends by naming two replacements that don’t need the back of things: a depth export, and window light. Window light is Relight with a large, shaped source, its blinds, panes or leaves projected onto the subject and whatever is right behind them, where the bicycle’s shadow fell and depth is reliable.

Its kill test asked a sharper question. Gobo presets and “window light” overlays lay a pattern flat over the photo. Does the same pattern, put into the scene by depth, read as light falling on things? If the two look the same, the tool dies. So each pattern was rendered twice: in the scene, and flat, as if everything stood at the subject’s distance facing the light with nothing shading anything.

In the scene, every point looks back along the light to a window 1.2 subject distances away, and takes the pattern from where its ray crosses it:

// Where the ray toward the light crosses the window, and how far that is.
let t = simd_dot(centre - p, dir)
let crossing = p + dir * t - centre
let q = SIMD2(simd_dot(crossing, right), simd_dot(crossing, upOnPlane)) / unit
// Edges at least a pixel and a half wide, as seen on this surface.
let footprint = 1.5 * p.z / f.fx / max(abs(simd_dot(r.normals[i], dir)), 0.3)
let blur = max(spread * max(t, 0), inScene ? footprint : 0) / unit
var through = transmission(q, s, blur: blur)
// The window's frame: light only through its opening.
let half = SIMD2<Float>(0.65, 0.8) * s.size

Edges soften with the distance to the window, the light falls only through its opening, and the subject shades what is right behind them, with Relight’s march and Sun’s thickness.

Nine of ten

The ten photos were the four Portraits again and six with estimated depth, among them a woman at a brick wall, a cat and the bicycle. Depth won on nine. The flat control is a striped rectangle over face, wall and sky alike. In the scene a window bar’s shadow bends over the cat’s muzzle, the panes break at different angles across the laptop’s desk, keyboard and screen, and the stripes arc over the woman’s hair and shoulders. The weakest was the ice cream: only the hand and the cone are near enough to catch the window, so the effect is small, though right.

A unit test measures what the eye saw. On the synthetic sphere, toward the rim where the surface turns away, the stripes crowd together; laid flat, they don’t.

What the test changed

The other numbers each came from something going wrong. Without a tap on the subject, the centre-weighted guess on the woman low in the frame found the wall at 2.6 where her face was at 0.54, nearly five times as far; the window was placed and sized for the wall, and the stripes on her face blurred to grey. So Window needs a tap, as Grade does. Solid all the way back, she threw a wedge of shadow to the edge of the frame, hence Sun’s thickness.

The sun is half a degree across, so crisp is the default: at 2° of softness the slats washed out a metre behind the subject. Sharper than a pixel and a half on each surface, the slats aliased into moiré across far foliage and grass. And unbounded, the pattern striped the whole scene to the horizon, as if the world stood behind blinds, so the window has a size.

Two limits were measured and left. Far backgrounds get no pattern, which is physically right and makes this a tool for portraits and still lifes, not landscapes. And on out-of-focus backgrounds the stripes follow the estimated depth’s ripples.

In the app

Window is a Pro tab after Relight: Blinds, Panes or Leaves, with Size, Turn, Opening, From, Height, Power, Shade, Softness and Warmth, and a tap for the subject. The light is made on the CPU at the working size whenever the settings change, at 13 to 23 ms per pattern on the Mac’s CPU across 8 threads, most of it the shadow march, and added to Relight’s light in the stacked composite on the GPU. It was checked in the Simulator on the woman at the brick wall. Its cost on a device while a slider is dragged is still open.

What the tests hold

• Sun, on the synthetic scene: the sphere’s shadow lands on the wall at 5 m where the ray from its centre meets it, with under 0.1 of the sun reaching that point, and the wall away from it keeps over 0.8.
• Window, GPU against CPU: the stacked composite matches the CPU reference within a mean of one step; the plan records 0.16, and a worst of 1/255.
• Window, stripes follow the surface: on the sphere the gap between stripe edges near the rim is under 0.8 of the gap in the middle, and laid flat it stays above 0.85.
• Sun’s kill test: stop, with cast shadows right in 2 of 10 photos. Window’s kill test: continue, with depth beating the flat pattern on 9 of 10.

Sun drew the line Window works inside: light from one depth map can be trusted on the subject and just behind it, and not much farther. The last Pro tool makes a short clip from a still, a push-in, a sway or a Vertigo, and it is the subject of the next article.