Keep your agent's brain outside the box

This is part two. Part one was about picking Charlie's computer.

Part one boiled down to a single question: does your agent loop run inside the box or outside it? I said that question does most of the work. It does. The box comes after.

This post is that question — the decision underneath the box.

(If you're new here: I'm building Char, an AI notepad, and Charlie is the assistant inside it. Its brain is the harness — the loop that reads your message, calls the model, picks a tool, and decides whether to stop and ask you first.)

In part one I also wrote that our next design would put the loop inside the box. The same week, I asked one more question: why can't each harness just run on its own machine?

We talked it through. We didn't move it.

Charlie's computer stayed on E2B anyway. Not for the loop — because we still wanted it to pause with its memory and pick up where it left off.

Here's why the brain stayed out, and what broke while we built the version we kept.

What the brain has to do

Before deciding where the harness lives, we listed what it actually does:

1. Hold the keys. The model API, your Gmail and Calendar tokens, write access to your data.
2. Decide every step. Including whether to stop and ask you.
3. Answer instantly. Most chats never touch a computer. They answer a question, edit a page, check your mail. They shouldn't wait for one.
4. Keep working while you're away. Schedules, incoming email, overnight memory work.
5. Survive a deploy. A turn in progress should finish, not vanish.

None of those needs a computer. Some of them need to stay as far from one as possible.

The options we weighed

Brain inside the box. This is the pitch every sandbox startup makes: a dedicated machine per agent. The loop runs in a VM next to the shell and the files, the way coding CLIs like Claude Code and Codex run on your laptop. It's tempting. Tool calls are local, every agent is isolated, and you can drop in any CLI agent unchanged.

It costs three things.

• The keys move in next to bash. Charlie's computer is where web pages, downloaded files and model-written code run — any of which can carry a prompt injection. Put the harness there and a hijacked page sits next to your tokens, and next to the code that decides whether a send needs your approval.
• Every message waits for a VM. About 2 seconds to resume one, 6–10 for a fresh one. Even for a chat that only wanted to say "done."
• The server doesn't go away. Something still has to hold the secrets, enforce approvals, find the right VM and wake it for a Schedule at 6 a.m. The trusted layer just gets thinner.

Brain outside, hands in the box. The loop runs on our servers and stays warm. Charlie's computer is something it calls: run this code, read that file, open this page.

But Devin gives every session a VM. It does, and for Devin that's right. Nearly every Devin session clones a repo, installs, builds and runs tests. The computer is the work. For Charlie it's the exception — most chats never need one. A VM per chat would add wait time and cost to every chat for nothing.

The real question was never which box. It was: what's allowed to touch your keys?

What we chose

The trusted side runs on Fly. Each person gets one harness — a small object in memory, an actor — and all of their chats live in it together. One machine holds many people. Every model call and tool call is checkpointed to the person's own database, so a crash or a deploy resumes the turn instead of losing it. A tool only reruns if it's safe to repeat.

The untrusted side is Charlie's computer, on E2B. One per person, not per chat, so the file you worked on yesterday is still there today. It starts only when Charlie reaches for it, pauses when idle, and comes back in about 2 seconds. It holds no keys and has no say over approvals.

Two machines first, for deploys, not for load. Each person belongs to one machine at a time, recorded in a table. A request that lands on the other machine gets a fly-replay header, and Fly's proxy resends it to the owner in a few milliseconds. The app never sees the hop. During a deploy, people are handed from the old machine to the new one.

Load wasn't the reason. We added the second because with one, every deploy took Charlie down. Three deploys in three days, one to two minutes each.

Then we measured load anyway. We load-tested 500 people with 10 chats open each. Memory was fine. CPU wasn't, and not where I expected. Waiting on the model is nearly free. Running the turn around it is not: turns were about 80% of the CPU, all on one Node thread. On the small shared machines we'd started with, one machine kept up with only about seven active people once its CPU burst ran out.

So we moved the brain to dedicated CPUs. That's about 16× the sustained CPU per machine, or roughly 75 people each when one chat in twenty is running. And because every request already finds its owner, a third machine works exactly like the second.

Then we profiled it. I'd assumed the cost was streaming the answer back token by token. It wasn't. The model stream was cheap. Our bookkeeping around it wasn't. On every partial update, ten times a second, a turn looked up its own run again in three places, rewrote the whole saved transcript, and bumped the chat's last-activity time. That last one rewrites the chat, its search entry, and sends a sync change to every one of your devices.

Now a stream checks its run once. The transcript is saved at most every half second, and always at the end. Activity is bumped at most every 5 seconds. CPU per turn halved, from 190 ms to 95 ms. Database calls per turn fell from about 2,600 to about 930, and sync changes per reply from 25 to 3.

Two smaller fixes came out of the same test. The small scripts Charlie writes to call its tools shared one sandboxed process per machine, so one slow script stalled everyone's. They now run across a pool, and a stuck one only fails itself. And chats now follow the live stream instead of polling for changes.

Then we let it scale on its own. The hands already did: E2B runs one sandbox per person and pauses the idle ones. Now the brain does too. The signal is running turns per machine, because that's what burns CPU — not request count, since a forwarded request lands twice.

Every 10 seconds, each machine reports how many people it holds and how many turns it's running. A person nobody owns goes to the least-busy machine. Anyone idle for 15 minutes is unloaded, and a machine well above the fleet average hands quiet people back, so a new machine actually fills. An autoscaler keeps between 2 and 14 machines running, based on running turns.

What broke

A red banner on a chat that was working fine. Right after we went to two machines, people started seeing "Cloud tasks are unavailable" in red. The chat was fine the whole time — "Thinking…" showed up a few seconds later. The status check the app runs while Charlie works had landed on the machine that didn't own that person, and that one route never learned to forward. It threw instead.

It wasn't the only one. File uploads had the same gap, and they failed quietly: the app said the file was attached, and it never reached Charlie's computer.

The fix was boring. Every request now checks who owns the person and forwards if it isn't the owner. If someone is mid-handover during a deploy, the app hears "busy, retry in 1 second" and retries without saying a word.

Approval prompts that made Charlie feel broken. Ask first used to ask before almost every write. So Charlie asked permission to start its own computer. Then it asked again to check the state of its own browser. Each prompt was defensible. Together they made Charlie feel broken.

I said it twice: don't ask unless it's critical.

The rule now: under Ask first, Charlie confirms only the steps that can't be undone and reach outside Char — sending an email or message, paying or ordering, posting publicly, deleting for good. A calendar event with guests asks, because it emails them invitations. One without guests doesn't.

Everything on Charlie's computer runs without asking: navigating, clicking, typing, running code. That's the payoff of the split. The box is disposable, so Charlie can move fast inside it. When a task on a website ends in a send, a payment or a post, Charlie stops right before that final click.

Two kinds of prompts stay. Charlie asks when it needs clarification. And it asks when it suggests something you didn't ask for — like connecting Gmail during setup — because that's its idea, not yours. Computer use on your own Mac still asks too. Your Mac isn't disposable.

What I learned

1. Keep the brain out of the box. Keys, approvals and the loop live where untrusted code can't reach them. The box only gets what the brain hands it.
2. Make the box cheap to lose. If nothing in it can hurt you, the agent doesn't need permission to work in it. Isolation is what buys you fewer prompts.
3. Ask when it can't be undone, not when it writes. Read versus write is the wrong line. Undoable versus not, and inside versus outside, is the right one.
4. Size the unit to how people use it. Devin's unit is a session, because every session is code. Ours is a person: one brain, one computer, files that outlive a chat.
5. Your second machine is for deploys. Before it's capacity, it's what lets you ship without an outage.
6. Route every request or none. With two machines, one route that forgets to forward is a bug that looks like an outage.
7. Measure before you buy machines. I'd have bet on memory. It was one CPU thread.
8. Then profile before you blame the model. The stream looked like the cost. It was our own bookkeeping on every frame, and half of it went away.

What's next

Wake the computer early. Today the sandbox wakes on Charlie's first computer call, so code starts about 4 seconds in: 2 for the model to plan, 2 to resume. If we start resuming the moment a chat looks like it needs code, those overlap and code starts at about 2 seconds. Chats that never need code still never touch E2B.

Make every turn cheaper, again. With our bookkeeping gone, the biggest remaining cost is the checkpoint itself: about 10 database calls per update, nearly half of what's left. Every call is a round trip to the person's database, so that's next.

The box got all the attention in part one. It's the easy half.

The brain is where the product lives. Keep it outside the box.