What Ran Out Was the Disk

Until version 1.2, Sharp had never run on a physical device. Every memory figure in its documentation came from a Mac or from the iPad simulator, and the documentation said so. The fear behind most of the engine’s design was memory: that iPadOS would kill the app halfway through a long bracket of large frames. When it finally ran on a real iPad, memory turned out to be the one thing that never ran short. What ran out was the disk.

A stack with no screen

Measuring an app by tapping through it is slow and hard to repeat, so the first thing 1.2 gained was a way to run a stack with nobody touching the iPad. It is a launch argument, -autorun, followed by pairs of a tool and a folder. The file it lives in is compiled only into Debug builds, or into a build given the AUTORUN flag; nothing that ships contains it.

The frames go into a folder inside the app’s Documents, copied there with xcrun devicectl device copy to --domain-type appDataContainer. Then the app is launched with its console attached, and it stacks every frame in the folder exactly as the app would, prints one line per run, and quits.

xcrun devicectl device process launch --console … me.pablin.sharp -- -autorun focus big45

The -- in the middle matters. Without it, devicectl read -autorun pick as options of its own and refused, with “The value 'pick' is invalid for '-t <seconds>'”. Everything after the double dash goes to the app untouched.

Two numbers about memory

Each line reports two different things. The first is the footprint: what the system charges the app for. Sharp reads it from task_info, and the field it reads is phys_footprint, not resident size.

public func memoryFootprint() -> UInt64 {
    var info = task_vm_info_data_t()
    var count = mach_msg_type_number_t(MemoryLayout<task_vm_info_data_t>.size / MemoryLayout<natural_t>.size)
    let result = withUnsafeMutablePointer(to: &info) {
        $0.withMemoryRebound(to: integer_t.self, capacity: Int(count)) {
            task_info(mach_task_self_, task_flavor_t(TASK_VM_INFO), $0, &count)
        }
    }
    return result == KERN_SUCCESS ? UInt64(info.phys_footprint) : 0
}

The difference is not academic. The engine keeps decoded frames in a mapped scratch file, and mapped file pages are file-backed: the system can drop them whenever it likes and does not charge them to the app. Resident size counts them anyway. Earlier, eleven 22MP RAW frames in the iPad Pro simulator showed 2.09 GB resident and 0.54 GB of footprint.

The second number is headroom: os_proc_available_memory(), how much more the system would let this process have before it is killed. A Mac has no equivalent, which is why none of the earlier figures could answer the real question. Headroom has to be watched, not read once. The tallest moments of a stack are inside a frame and at the very end, when the 16-bit file is written, and a sample taken at frame boundaries misses both. So a second thread samples every 20 milliseconds and keeps the lowest value it sees.

let lowestHeadroom = Lowest(os_proc_available_memory())
let before = lowestHeadroom.value
// Every frame boundary is not enough: the tallest moments are inside a frame and at the
// very end, when the 16-bit file is written. So something watches the whole time.
let watching = Watch()
Thread {
    while !watching.stopped {
        lowestHeadroom.sample(os_proc_available_memory())
        Thread.sleep(forTimeInterval: 0.02)
    }
}.start()

“Memory used” in what follows is the headroom before a run minus the lowest headroom during it: how close the iPad actually came to the edge.

The first run, and the build it ran in

The first run was a Debug build, because that is what pressing Run in Xcode gives you. Pick, the tool that only chooses the sharpest frame, took 186.7 seconds on seven 45MP frames. The same run in a Release build, with the AUTORUN condition set so the harness was still there, took 8.5 seconds.

The engine is tight scalar loops over pixels, and at -Onone they are unusable. The simulator had taught the same lesson a phase earlier, but it bears repeating: a Debug build is not a measurement. Every figure below is from Release.

The iPad and its frames

The device was an iPad (10th generation), with 4 GB of memory. Before a run, os_proc_available_memory() reported 3.0 GB of headroom. That is the budget: not the 4 GB on the box, but the 3.0 GB the system was prepared to give one app.

There was no 45MP bracket to hand, so the test frames were made. Seven real landscape photographs, a public focus bracket of heather and a valley, were upscaled to 5464 × 8192 with sips. Upscaled frames are honest about size and nothing else: they are as large to hold and to process as a 45MP camera’s, but they hold no detail a 45MP sensor would add. For the 35-frame runs, those same seven frames were repeated five times. That makes it a memory test, not a quality test, and it is used here only as one. The RAW test was 63 real 12MP DNG frames, 4048 × 3036, from the HDR+ dataset.

What it measured

The line that matters is the second. Five times the frames took more time, as it should, and not one byte more memory: 1.22 GB at seven frames and 1.22 GB at thirty-five. The engine was built so that what it holds depends on the size of a frame and not on how many there are, and on the iPad it does. At the worst moment of the seven-frame focus stack, the app still had 1.78 GB it could have asked for. Blend, the tool new in 1.2, used 0.98 GB on three 45MP frames in 6.3 seconds; it has an article of its own later in the series.

What failed

One run did not finish. Vanish, the tool that removes whatever moved, on the thirty-five 45MP frames, refused:

Vanish keeps every frame’s aligned pixels in a scratch file, because a pixel’s value under Vanish is not known until every frame’s value for it has been seen. The pixels are kept as half floats, three channels of two bytes, and the file is refused unless it is comfortably smaller than the disk can spare:

static func size(frames: Int, width: Int, height: Int) -> Int {
    frames * width * height * 3 * MemoryLayout<UInt16>.size
}

static func isAffordable(_ bytes: Int, in directory: URL) -> Bool {
    guard bytes <= 16_000_000_000 else { return false }
    let free = (try? directory.resourceValues(forKeys: [.volumeAvailableCapacityForImportantUsageKey]))?
        .volumeAvailableCapacityForImportantUsage ?? 0
    return Int64(bytes) * 4 < free
}

The rule is a quarter of what the volume says it has available for important use, and never more than 16 GB, so the 9.4 GB file would have needed more than 37.6 GB free. Focus uses the same cache, but only to save time: when the file will not fit, it decodes every frame a second time instead, which is slower and always works. Vanish has no such fallback. Without every frame at once there is nothing to take the middle of.

None of this is memory. The cache is mapped, and its pages are not charged to the app; the footprint of a run that used it would have looked like the others. The limit was a disk, and a sensible rule about how much of it to take.

The second limit

Looking for other things that grow with the number of frames turned up a smaller one. The map of which frame won each pixel, which is also the depth map, is one byte a pixel:

/// Which frame won each pixel: the depth map, one byte per pixel.
public let winner: [UInt8]

The largest frame number a byte can hold is 255, so a bracket longer than that cannot name all its frames. The retouch recipe had the same problem, worse. It stored the frame each pixel was painted from in a byte too, and used 255 to mean “untouched”. Frame 255 could not be painted from at all, and every frame after it was clamped into frame 254, quietly. In 1.2 the recipe’s source is two bytes, and untouched is the largest value two bytes hold:

/// No frame painted here — the pixel is the merge, as the stacker left it.
public static let untouched: UInt16 = .max

/// The frame a pixel is claimed for. Everything below `untouched` is a frame.
public static func claim(_ frame: Int) -> UInt16 {
    UInt16(min(max(frame, 0), Int(untouched) - 1))
}

What the tests hold

• On a 32 × 32 recipe, painting a pixel from frame 255 and then from frame 300 records each exactly, and afterwards the only frame the recipe uses is 300.
• For a vanish of thirty-five 8192 × 5464 frames, a disk with room for the whole scratch file at the quarter rule stacks in one piece; a disk with room for only ten frames’ worth is cut into slabs of at most ten frames that together hold all thirty-five.

Memory was never the limit. The scratch disk was, and the one-byte frame number, and both are answered by stacking a long bracket in pieces. Before that, though, the frames have to line up, and the next article is about a bracket shot by hand, where they do not.