Stacking in Slabs

A long bracket in Sharp 1.2 can be stacked a slab at a time: the frames in groups, each group stacked with the chosen tool, and then the groups’ results stacked into one picture with the same tool again. Zerene Stacker does something like it for focus, and it would be easy to add it as a way of saving memory. That is not what it is for here, because memory was never what ran out.

What a long bracket runs out of

On an iPad (10th generation) the measurements say so plainly. A stack holds one frame at a time, and its footprint is flat in the number of frames: focus on seven 45MP frames used 1.22 GB, and focus on thirty-five used 1.22 GB too. Adding frames costs time, not memory.

What does grow is the scratch file. Clean, trails and pick are a single pass: a mean, a maximum or a sharpest-wins can be carried in an accumulator that never holds more than one frame. Focus and vanish read every frame twice, and they keep the pixels pass one has already decoded and aligned on disk for pass two, as half floats, three channels of two bytes: six bytes a pixel, for every frame.

static func size(frames: Int, width: Int, height: Int) -> Int {
    frames * width * height * 3 * MemoryLayout<UInt16>.size
}

The cache refuses rather than fills the disk: it takes at most a quarter of what the volume reports as available, and never more than 16 GB. Focus then falls back to decoding every frame a second time, which is slower but always works. Vanish cannot. The middle of a set of values is not known until the last of them has been seen, so it reads every frame’s share of a band back from the scratch file together, and on the iPad a vanish of thirty-five 45MP frames simply failed for want of 9.4 GB of scratch.

The second limit is a byte. Every stack also produces a map of which frame each part of the picture came from, and the map is one byte a pixel. A one-byte map has room for 254 frames, and past that a bracket cannot say where anything came from.

Equal groups on one grid

A slab needs only its own frames’ worth of scratch, and a slab stack’s map names slabs rather than frames. Those two facts are the whole case for slabs. The rest is making sure the picture comes out the same.

The first condition is that every slab lands on the same grid. A slab stacked on its own would align its frames to its own middle frame, and four slabs would give four results that each sit a little differently. So every slab is told to align to the middle frame of the whole bracket:

let reference = urls[urls.count / 2]
…
for (index, slab) in plan.slabs.enumerated() {
    var each = options
    each.slabSize = nil
    each.reference = reference
    // Noise reduction once, on the final picture, as the plain stack does it.
    each.denoise = 0
    let members = Array(urls[slab])
    let result = try Stacker(options: each, io: io).stack(urls: members) { … }
    alignments.append(contentsOf: result.alignments)
    let file = dir.appendingPathComponent(String(format: "slab%03d.tif", index))
    try autoreleasepool { try io.write(result.image, to: file, format: .tiff16) }
    slabFiles.append(file)
}

Each slab’s result goes to scratch as a 16-bit TIFF. Because they all share one grid already, the second stage has nothing to align: it is the same tool with the same settings and align = false. And because every frame was aligned straight to the bracket’s reference, not to a slab’s reference that was then aligned again, every frame’s alignment is direct. The stack keeps all of them, one per frame, and the retouch brush paints from exactly those.

Noise reduction is held back from the slabs and done once, on the final picture, as a plain stack does it.

A mean of means is the mean

The second condition is the size of the groups. Clean averages, and a mean of means is the mean only when every group has the same number of members. So the plan does not cut a bracket into slabs of the asked size and a remainder; it works out how many groups that size implies and then shares the frames out evenly:

// Equal groups, differing by at most one frame: the mean of means is then the mean.
let groups = (count + size - 1) / size
guard groups <= mostFrames else { return nil }
var slabs: [Range<Int>] = []
var start = 0
for g in 0..<groups {
    let length = count / groups + (g < count % groups ? 1 : 0)
    slabs.append(start..<(start + length))
    start += length
}

Twenty-four frames in slabs of six are four slabs of six, and the clean in slabs is the plain clean. Where the count does not divide, the groups differ by one frame at most: thirty-five frames with room for ten a slab become four slabs of nine, nine, nine and eight, and a frame in the slab of eight counts for a thirty-second of the picture rather than a thirty-sixth.

Trails is easier still. It keeps the brightest value at each pixel, or the darkest, and a maximum of maxima is the maximum however the groups are cut.

Focus is close rather than the same. A slab picks the sharpest detail among its own frames, and the second stage picks among the slabs’ winners; the result is not the plain stack to the last value, but it is what Zerene Stacker does with slabs, and the test holds it within 0.01 of the plain stack.

When it slabs

Three things decide it, and SlabPlan.plan asks all three. A photographer can ask: “Stack in slabs”, under Long brackets in the settings, part of the unlock, from 5 to 100 frames a slab in steps of five, 20 by default. The command line takes the same thing as sharp <tool> … --slabs n. Past 254 frames it slabs whether anyone asked or not. And for vanish only, when the scratch for the whole bracket would not fit, it slabs to what the disk affords:

if let asked = options.slabSize, asked >= 2, count > asked { size = asked }
if count > mostFrames { size = min(size, mostFrames) }
if options.tool == .vanish {
    let free = freeBytes ?? Self.freeBytes()
    let affordable = min(Int64(16_000_000_000), free / 4)
    let fits = Int(affordable / Int64(max(1, FrameCache.size(frames: 1, width: width, height: height))))
    if fits < count { size = min(size, max(fewestForVanish, fits)) }
}

Pick and Blend never slab. Pick merges nothing, so there is nothing to group. Blend works on exposure brackets of a handful of frames and its footprint is already flat.

Because the engine can decide to slab by itself, the app asks the same question before a run, with the same plan, so the progress can say what is happening. A slab stack reads “Frame n of total, in k slabs” through the first stage and then “Putting slab n of k together”. Afterwards the map names slabs, not frames, and the result carries a note saying so.

On the iPad

The vanish that failed was run again on the same iPad (10th generation): thirty-five 45MP frames, slabbed to fit the disk by itself. It finished in 107.6 seconds, using 1.03 GB of memory.

The bracket past 254 frames has not been run on the iPad. It is held only by a test, on frames sixteen pixels wide.

What the tests hold

• Clean and trails on 24 frames, plain and in slabs of six, give four slabs, a frame count of 24 and 24 alignments, and no pixel differs by more than 0.0002. One step of the 16-bit file between the stages is 1.5e-5.
• Focus on 12 frames in slabs of four is within 0.01 mean difference of the plain stack.
• 254 frames are not slabbed; 300 are slabbed into two slabs of 150 without anyone asking; Pick is never slabbed.
• An actual 300-frame trails run, on tiny synthetic frames, slabs itself into two and reports 300 frames.
• Vanish on 35 frames of 8192 × 5464 does not slab when the disk has room, and with room for only ten frames a slab, every slab holds ten or fewer and all 35 frames are covered.

Slabs keep every frame’s alignment direct so that a brush can go back to any one of them. The last article is about that brush, and about keeping a retouch you can come back to.