We got one full CI run down to six minutes, then reverted a later change that made individual jobs faster.
The six-minute run bought shorter waits with more compute. The later change overloaded the runners and introduced timeouts.
At Char, we were pushing code with Claude Code and Codex faster than we could comfortably verify it. In September, we looked at PR runs taking roughly 14 to 24 minutes. One spent 7m47s on TypeScript checks before starting another roughly six minutes of iOS work.
My first instinct was to keep merge checks small and do the expensive validation before a release. Char has a desktop app; building every native target on every push felt wasteful. But we also had a web/API deployment path, so weakening the merge gate could affect production before a desktop release ever happened.
We moved mobile tests to Linux and kept native builds on affected main pushes, scheduled runs, and manual dispatch. The Linux mobile checks passed in 2m44s. Storybook then became the thing we were waiting for.
By October, we were measuring individual test steps and changing the tests' setup as well as the workflow.
We split the large CI job into parallel lanes for unit and integration tests, with separate checks for the native agent storage journey and critical browser interactions.
Tests run in parallel, but required results still meet at one gate. Native app builds and deployment ordering are omitted.
One run that selected the full package set reached the CI gate in 6m07s. The previous nine successful runs had a median of 20.3 minutes. Total runner time increased from about 20 minutes to 36.5 minutes.
That compares one new run with a historical median; it needs more measurements before we can generalize it to ordinary PRs.
One full-selection run after tiering, compared with the earlier baseline. Runner minutes measure compute consumed, not the bill or the time a developer waits.
I was willing to spend more compute to shorten the feedback loop. I also wanted to keep the bill visible: parallelism can make CI feel cheaper while using more runner time.
Then we tried giving each integration shard four Vitest workers instead of two. The early measurements looked good enough to merge. Timeouts followed in the native title viewer and memory-context tests, and we attributed them to runner overload after the worker increase.
We reverted it and returned to four shards with two workers each. A shorter successful run wasn't worth making failures harder to interpret.
The more durable savings came from setup that tests repeated without gaining additional coverage.
A CPU profile of the native agent storage test showed its fake SQLite driver preparing SQL statements repeatedly. Reusing prepared statements reduced the reported job time from 358 seconds to 323 seconds without changing its assertions.
We also moved 60 Page actions out of that long journey into a separate test file so they could run alongside it. Across those changes, the native test step went from about 303 seconds to 156 seconds. We kept the 10-second deadline for each Page action.
The integration suite had another repeated cost: running all 210 database migrations for every test, about 550ms each time. We changed ordinary fixtures to migrate once per worker and copy the resulting database, which took about 10ms. Tests specifically checking upgrades or partial schemas still ran the migrations themselves.
The local integration suite went from 277 seconds to 207 seconds.
Assigning files by sorted name left one shard with much more work than another. We used recorded durations to put the longest files onto the least-loaded shard first; the reported spread between the fastest and slowest shard narrowed from 117 seconds to 41 seconds.
| Change | Recorded comparison | What we preserved |
|---|---|---|
| Reuse prepared statements | Native storage job: 358s to 323s | The same assertions |
| Reuse statements and split Page actions | Native test step: about 303s to 156s | The 10s deadline per Page action |
| Copy a migrated database fixture | Local integration suite: 277s to 207s | Real migrations in upgrade and partial-schema tests |
| Balance shards by duration | Spread between shards: 117s to 41s | The selected test files |
The first two rows overlap. Job duration, test-step duration, local suite duration, and shard spread are different measurements.
None of those numbers should be added together. They measure different parts of CI, on different runs, while the test suite was changing.
The strangest slowdown happened before tests started.
Our package store restored successfully, yet pnpm installs took between 59 and 1,040 seconds during a period of slow registry responses. The logs said packages were reused, with nothing downloaded.
pnpm was still fetching metadata to verify the lockfile against our supply-chain policy, including a minimum release age. A local reproduction counted about 2,850 metadata requests with an empty verification cache. Preserving pnpm's roughly 4 KB verification record reduced that count to zero.
We cached that record with a key covering the lockfile and policy configuration, along with the files specifying the pnpm version and npm settings. Changes to those inputs still required verification. The first reported main run after the fix installed in 10 seconds, compared with a normal 15-17 seconds before it.
The larger benefit was avoiding those repeated metadata requests when the registry was slow. We kept the policy enabled.
We decided against shared test-result caching during this work. An analysis of 44 recent runs estimated potential hits for 21% of test-task minutes, but the package dominating the slowest shard would have hit only four times in 194 executions.
That package also read inputs it hadn't fully declared. Caching a successful test result would have required fixing those dependencies first, for relatively little improvement to the wait we cared about most.
Keeping the checks meaningful took separate work.
We replaced CSS-class assertions for scrolling with browser tests that observed actual overflow. Restoring the old constrained scrolling behavior made the replacement test fail. We also rejected moving editor tests from jsdom to Node for a roughly 16-17% saving: code branching on document or navigator could take a different path from the app.
A story appeared to lose multi-select answers when a card collapsed and expanded. The stories reused question IDs, so state from an earlier story affected the next one. Giving them distinct IDs fixed the test isolation problem.
We also broke the nightly workflow ourselves.
A new job required a permission that the reusable nightly caller didn't grant. PR CI passed, but the scheduled workflow failed at startup and ran no tests. We fixed the permission; the session record still had the next scheduled run pending as the real verification.
I don't have a clean final before-and-after number for the whole effort. The suite grew, registry latency distorted wall time, and the comparable nightly measurement was still pending in the records used for this post.
I would keep the prepared-statement reuse, database fixtures, and measured shard balancing. I would also keep the worker rollback. We shortened specific checks and made other checks more realistic, but we still owed ourselves a comparable full run before calling the whole CI project finished.