← Blog

CI needs to be ready when I am

John Jeong

Depot finished our typecheck benchmark in 49 seconds. Boat took 5 minutes and 1 second. Both restored all 101 tasks from cache.

Depot was much faster. Then I looked at how I actually work.

Update, November 3: Boat lasted about a day. We now run all CI on Blacksmith. What happened is at the end.

I'm often working across several chat sessions. One is implementing something, another is fixing a test, another is reviewing a change. I come back and merge things as a batch. I don't sit in front of every CI run waiting for it to turn green.

That changes the question I want the benchmark to answer: how much validated work is ready when I come back?

CI still needs to be fast. If I return to review and half the changes are waiting on checks, it is too slow. If an agent needs a test result before it can make its next edit, that delay is part of its working loop. But a result arriving four minutes earlier doesn't automatically give me four productive minutes back.

After working on making CI faster, we started looking at Boat because Char's Depot bill felt too high. Char has JavaScript, Rust, desktop and mobile code. We moved Linux CI and deployment jobs to Boat, kept Apple jobs on Depot, and kept Windows builds on GitHub. The migration passed candidate CI and merged on November 1. We haven't measured a comparable period of invoices after the switch yet.

The warm benchmark gives us a starting point:

Same pinned application source, all 101 tasks cachedDepotBoat
Full job49s5m01s
Turbo execution0.488s2.806s
Checkout7s31s
Rust setup20s147s
pnpm setup11s103s

These are observations from one warm run per provider, not a distribution of production CI times.

Most of the gap came before the task results were used. Depot automatically routes standard Actions cache operations through its own cache service and connects Turbo to its remote cache. On Boat, we added explicit persistence for Turbo results, but still spent minutes restoring dependencies and preparing the environment. Depot cache integration, Turbo integration.

Most of the gap came before the check ran. Same pinned source, all 101 Turbo tasks restored from cache, one warm run per provider. Depot routes Actions cache and Turbo through its own cache; on Boat, dependencies and toolchains are restored step by step.

We also had to lower typecheck concurrency from 16 to eight. Both machines had 16 vCPUs, but Boat had 32 GB of advertised RAM versus Depot's 64 GB. Our first Boat attempt exhausted memory and swap. Matching the CPU count hadn't reproduced the environment.

This is a comparison of the configurations we would actually use, not an isolated CPU test.

Boat's dashboard attributed $0.017 of usage to the successful warm run. Depot's published 16-vCPU Linux rate is $0.048 per minute, which puts the 49-second run at $0.0392 at list price. That Depot number is calculated, not taken from our invoice. Depot runner pricing.

On those inputs, a dollar of compute corresponds to roughly 59 Boat benchmark runs or 26 Depot runs. That is an illustrative budget comparison using Boat's rounded dashboard figure and Depot's public rate. It excludes subscription floors, included credits, discounts, extra start charges and failed attempts. It also counts this one cached check, not complete PRs or shipped features.

Depot's speed narrows the price gap because we pay for fewer running seconds. The remaining premium can be easy to justify when someone is blocked on the result. The interesting part is what happens when the work is independent.

Suppose we have 20 independent checks like this one, ready at the same time. Assume the observed durations hold under load, with no failures, startup delay or shared bottlenecks. This is a model, not a load test:

Available concurrent job slotsDepot finishes all 20Boat finishes all 20
116m20s1h40m20s
101m38s10m02s
2049s5m01s

At ten slots, if I return to review after 15 minutes, both setups have finished the whole batch. Depot was ready much earlier. In this example, that earlier finish didn't change when I reviewed or how many results were ready.

If I return after three minutes, Depot has finished and Boat hasn't finished its first wave. The same infrastructure now creates a real interruption. If the checks arrive gradually, a change pushed just before I return may still be waiting even when the earlier ones are done.

What's ready depends on when I come back. A model of 20 independent checks on 10 job slots, using the observed warm durations. It assumes no failures, queueing, startup delay or shared bottlenecks; it is not a load test.

The review cadence determines whether the latency is visible.

Parallelism also changes how we should talk about saved time. Twenty jobs each finishing four minutes earlier does not mean I got 80 minutes back. They may all run while I am doing something else. Their compute costs still add up: under the model above, the batch costs about $0.784 on Depot versus $0.340 on Boat, regardless of whether we use one slot or twenty, provided durations stay constant.

The faster setup releases each slot earlier, though. With the same number of continuously busy slots, Depot can finish more checks per hour. Boat can buy more of this particular check per dollar of compute. Those are different constraints, and both can matter to the same team.

More parallel jobs cannot accelerate a chain where each step needs the previous result. An agent doing edit → test → inspect failure → edit still benefits from fast feedback. So does a person waiting to fix a release. A fast typecheck also matters if it unlocks later jobs. We need to measure the dependency path to a mergeable change, not add every job's duration together.

There are limits to running more things at once. Providers have start quotas and concurrency limits; jobs can compete for cache bandwidth or other shared resources. Our Boat plan has a $100 monthly floor with compute credit, and its included starts are limited per minute, hour and day. Our 20-job illustration assumes those limits have headroom. We haven't purchased extra starts. Boat limits and billing.

A matrix of test shards can consume many starts for one PR. Twenty independent checks are not twenty complete CI pipelines. And 101 cached Turbo tasks are not 101 new pieces of work accomplished.

This is why we changed job selection as well as providers. Our old workflow already skipped native mobile builds on PRs and selected them for affected pushes to main. The new selector distinguishes ordinary mobile JavaScript from native inputs, selects iOS and Android separately, and chooses affected Rust platforms before allocating their runners. Selected native checks now participate in the PR gate.

For frontend-only work, we keep the affected checks and mobile bundle exports on Linux. Native changes still get the appropriate platform builds. Skipping an unnecessary native job reduces compute demand and the chance that an unrelated long-running build holds up the batch. A green Linux check still doesn't prove Apple-specific code or device behavior works.

The provider choice and the selection policy can be evaluated independently. We can keep the selector even if we change runners again.

What I want to measure after the switch is the fraction of changes whose required checks are ready when I return to review. Alongside that: actual blocked time, the tail of CI completion times, and total cost per successfully validated change, including retries. That will tell us more than multiplying every saved runner-minute by an engineer's hourly rate.

We still need a comparable period of real workloads and billing before claiming monthly savings. That accounting needs to include queue time, failed attempts, complete PR pipelines, the Apple jobs we kept on Depot, and both providers' plan allowances.

I want CI fast enough that the batch is ready when I come back, and faster where another person or agent is waiting for its answer.

What happened next

The start quotas above weren't a footnote. One CI run uses about 15 starts, and I usually have several sessions pushing at once. Our first twelve hours on Boat used 1,072 starts. The busiest hour used 327 against an hourly limit of 210.

At 03:24 KST on November 2, the hourly cap ran out and every job failed at "Set up runner". Formatting, typecheck, builds and main all went red at once. It looked like a repo-wide break, but no code had changed. An hour and a half later the daily cap of 840 ran out too, and CI stayed blocked until 14:30. We skipped the morning release.

The real workload was also slower than the warm benchmark suggested. Compared with Depot's previous 24 hours:

16-vCPU jobsDepot, 24 hBoat, first 12 h
Jobs started893869
Failure rate0.9%13%
Job-minutes1,5667,269
Median unit test shard2.1min11.4min
Median TypeScript check1.3min8.9min
Median integration test shard1.5min7.3min

Of the 113 failures on Boat, 80 were refused starts. The same number of jobs used almost five times the minutes, so the cheaper minute bought much less than it looked like it would.

We upgraded to Boat's $500 plan to get the release out. That unblocked the queue but didn't make anything faster. Boat cost $518.84 for about a day and a half of CI.

The argument earlier in this post still holds: the batch only needs to be ready when I come back. But a runner that refuses to start isn't slow. It's absent. At five to seven times slower, the batch wasn't ready when I came back either.

We picked again:

  • Back to Depot. Our agents recommended this: no start limits, and we knew it was fast. But cost was why we left.
  • Namespace. Competitive per-minute pricing, but the plan we looked at capped us at 160 concurrent vCPUs, about ten of our jobs at once. That's too few for how we work.
  • Vercel CI. The cheapest listed rate, $0.0035 per CPU-minute, and native to Turborepo. But it was five days into beta and Linux only, with no Rust preinstalled. Our Electron tests would also need extra setup. It's worth a pilot, not all of our CI.
  • Blacksmith. List price per vCPU-minute about two-thirds of Depot's. No published start or concurrency caps, and the same 16-vCPU, 64 GB machines we had on Depot, so typecheck went back to concurrency 16. It also has macOS runners, so Apple jobs could leave Depot too. Blacksmith pricing.

We moved Linux CI and deploys to Blacksmith that afternoon and macOS that evening. That night's nightly was the first release built entirely on Blacksmith. Then we cancelled Boat and Depot. Blacksmith's YC startup discount applies to us, which makes our near-term bill better than list price, so I'm leaving it out of the comparison.

Here is the same job on main over the three days, split by which provider ran it (medians):

JobDepotBoatBlacksmith
TypeScript check1.1min7.8min4.4min
Unit tests, per shard2.2min9.2min5.0min
Integration tests, per shard1.5min6.2min2.9min
pnpm setup10s87s15s
Rust setup19s78s36s

These come from 14 to 56 runs of each job, not a controlled benchmark. Each run checks whatever its change affected. Still, the shape is clear: Blacksmith is about twice Depot's time on main and well ahead of Boat. Depot connected every job to its Turbo remote cache automatically. On Blacksmith, only typecheck keeps Turbo's results between runs. That gap is ours to close, and closing it is cheaper than paying for it.

The measurement I said I owed is still owed: how often the batch is ready when I come back, and the cost per validated change, over a full month on one provider. What I learned in the meantime: when you shop for cheaper CI, check its limits against how many runs you actually start, not only its price per minute.