The 4.7 MB File I Didn't Split

One line of error replaced my desktop app's whole window: Unterminated string in JSON at position 4188076.

The file the app had failed to read is the spine, the one JSON file my system compiles from my notes. The desktop app reads it, and my phone gets a slimmer copy of it. On the day of the crash, 10 November, it was 4.7 MB, up from 3.0 MB on 1 October. My first question was whether it had simply grown too big, and whether it would only get slower and more fragile from here.

I measured before cutting anything. The size cost 13 milliseconds. The crash came from somewhere else.

What the size costs

About 90 places in my code read the spine whole, and it grows by about 1.3 MB a month. With the error coming out of the parser, size was the obvious suspect, so I timed it. Reading and parsing the whole file takes about 13 ms on my Mac, which puts 20 MB at about 60 ms.

The usual savings were smaller still:

• Whitespace is 26% of the bytes, because the file is pretty-printed. Removing it saves under 1 ms of parsing.
• The largest slice, the calendar history, is 20% of the file. Moving it to a file of its own would save 2 to 3 ms.
• Git held 104 MB of old versions, only because they sat as loose objects. Packed, they take 5.7 MB, and git packs them by itself once there are 6,700 loose ones.

The crash was a torn read

My sync script rewrote the spine in place, and the app re-reads it on every change to my notes. A read that landed mid-write got part of a file. This one got 4,202,496 bytes of it.

I reproduced it on a copy of my notes: 2 of 569 reads came back torn over 15 syncs. Most torn reads saw a 0-byte file, so the size of the file had nothing to do with it. It had bitten before, at about 3 MB: on 29 September a torn read made a check treat the file as empty and throw away an answer I had given on my phone.

// Before: rewritten in place while the app reads it
writeFileSync(join(ROOT, "_data/spine.json"), json);

// After: write a temp file beside it, then rename it into place
const spineFile = join(ROOT, "_data/spine.json");
const spineTmp = `${spineFile}.${process.pid}.tmp`;
writeFileSync(spineTmp, json);
renameSync(spineTmp, spineFile);

The process id in the temp name keeps two overlapping syncs off one temp file. On the same copy of my notes: 0 torn reads in 1,121 over 30 syncs, and 0 with three syncs running at once.

Keep the last good screen

The second fix was in the app. Any failed read set an error, and the app answered an error by replacing the whole window, even with a good spine already in memory. Now a refresh that fails keeps what's on screen, and only a first load with nothing to show is an error. A test runs that code now, and 4 of its 10 checks go red on the old version.

The same day, three smaller files that another process reads got the same write-then-rename.

When I'll split it

So I fixed the fragility and left the file whole. I'll look at a split again past about 20 MB, or the day a screen is measurably slow. At 1.3 MB a month, 20 MB is about a year away.

I haven't timed it on a real iPhone. The phone reads its slimmer copy, about 2 MB, on an engine with no JIT, and since a Mac number once hid a phone's cost from me by about 40 times, I don't take my Mac's 13 ms for the phone's.

Next time a big file breaks, time the read and find the writer before you plan the split. Mine took 13 ms to read, and four lines to stop tearing.

Both fixes made it into my runbook for jobs that run while nobody watches, with 23 other checks.