Our updates were slow because we asked 617 times

We ship a Nightly build of Char most days. So most days, I watch an update download.

It was slow. Not "big file on hotel Wi-Fi" slow — slow in a way that didn't make sense. One nightly took about eight minutes. The bytes that changed fit in roughly ten seconds of my connection.

So where were the other seven and a half minutes going?

How a differential update works

Char's desktop app is Electron, and it updates through electron-updater. On macOS, the update is a zip of the whole app — about 437 MB for a recent nightly.

You don't want to download 437 MB every day, so electron-updater does a differential update. Each release ships a blockmap: the zip cut into chunks, with a hash for each. The updater compares the blockmap of the version you have with the new one. Chunks that match get copied from the zip already on your disk. Chunks that changed get downloaded with HTTP Range requests.

That part works. Between two nightlies, about 149 MB of the 437 MB had changed — a third of the file.

The problem was how those 149 MB were fetched.

Where the time went

The changed chunks aren't one neat block. They're scattered across the zip, so they come out as hundreds of separate byte ranges. And the updater fetched them one at a time: send a request, wait for the answer, write it, send the next.

Our updates are served from downloads.char.com, a Cloudflare Worker in front of R2. For each request it looks up the object and then reads the range. That costs a round trip or two — call it half a second before the first byte arrives.

Half a second is nothing once. It's a lot 617 times.

We measured a real update between two nightlies in a real Electron run:

• 617 Range requests
• 286 seconds to finish
• Almost all of it waiting, not downloading

Then I found something I didn't expect in the upstream code: every 100 requests, the updater sleeps for a full second. A polite pause for the server. We were paying it six times per update, on our own server.

The real question was never "how do we get more bandwidth." It was: why are we asking 617 times?

The obvious fix was off the table

HTTP already has an answer for this. You can ask for many ranges in one request, and electron-updater supports it with useMultipleRangeRequest.

But our Worker doesn't serve multi-range requests. The spec lets a server ignore them, so it does what's allowed: it answers with the whole file. Turn that option on and every "differential" update becomes a full 437 MB download.

So we kept single ranges and changed how we ask for them. We already carry a pnpm patch on electron-updater, so the fix went there.

Three changes

1. Merge ranges that are close together. If two changed ranges are separated by 64 KB or less of unchanged bytes, we fetch them as one request — unchanged bytes included. Downloading 64 KB extra costs a few milliseconds. A request costs half a second. That trade isn't close.

2. Split ranges that are big. Any range over 4 MiB gets cut into 4 MiB parts. That sounds backwards right after merging, but it's what lets the next step work: a big range no longer blocks everything behind it.

3. Keep eight requests in flight. Instead of one request at a time, the updater now runs up to eight ahead of the writer. Responses come back in any order. They're written to disk in order, interleaved with the unchanged chunks copied from the old zip, so the final file is identical.

And we dropped the one-second nap.

The result

Same two nightlies, same machine, same real Electron run:

286 seconds to 12 — about 24× faster. The updater still checks the finished file against the release's sha512, and the hashes matched.

At 12 seconds for 149 MB, we're finally bound by the connection instead of by waiting.

Making parallel safe

Running eight requests at once changes what can go wrong, so most of the patch is about failure, not speed.

• Memory is bounded. Each range is buffered before it's written, so eight requests of at most 4 MiB keep buffered data around 32 MiB, no matter how big the update is.
• A server that sends too much gets cut off. If a response claims a different length than we asked for, or streams past it, we abort right away instead of holding a whole archive in memory.
• Anything but a 206 is a failure. A 200 means the server ignored the range and is sending the whole file. We stop it instead of buffering 437 MB.
• One failure fails the download now. If the fifth range errors while the first is still loading, we don't wait for the writer to reach it. Every outstanding request is aborted and electron-updater falls back to a full download right away.

While we were there: resume

Speed made another problem obvious. If you quit Char mid-download — or cancelled, or went offline — the partial update was thrown away. The next launch started again at 0%.

Now the partial stays on disk next to a small sidecar file pinned to the target version's sha512. The next attempt picks up where it stopped: a differential download skips the parts already written, and a full download asks for the rest with a Range request.

A resumed file is hashed in full before it's used. If the hash doesn't match, it's discarded and the download starts from zero. Partials from other versions are cleaned up when a new download starts.

The trickiest bug was the one nobody would notice: a cancelled download could keep writing to the file for a moment after it reported "cancelled." With resume, the next attempt would then append to a file another writer was still touching. So a download now only reports it stopped after it has aborted the request and closed the file.

What I learned

1. Count requests, not bytes. We'd already made the update a third of the size. It didn't matter, because the cost was in round trips. If you fetch many small things, latency is your bandwidth.
2. Read your dependency's defaults. A one-second sleep every 100 requests is a sensible default for someone else's server. It was costing us six seconds on ours.
3. Check what your server actually does. Multi-range requests looked like the fix. Our server, correctly, answers them with the whole file.
4. Fast makes failure matter more. Parallel requests needed bounded memory, early aborts and fail-fast errors. That was most of the code.

Updates should be something you don't notice. Now, mostly, you won't.