Everyone building with AI agents right now says some version of the same thing: “it’s fine, we run it in a sandbox.” I wanted to know whether that sentence means anything, or whether it’s just something we say to feel better about letting a model run code.

So I spent a week building a small tool to test it, and the most interesting thing I learned was not about the sandboxes. It was about how hard it is to measure them without fooling yourself.

The gap

There’s good research on tricking agents. Benchmarks like AgentDojo, InjecAgent and WASP measure whether a model can be talked into doing something it shouldn’t, usually through instructions hidden in content it reads. That’s real and useful work.

But it answers a different question than the one that keeps me up. Assume the model gets tricked, because eventually one will. The next question is whether the runtime around it holds. Can the compromised agent read another tenant’s data, steal the cloud credentials off the box, or quietly ship secrets to an address it controls? That’s the containment question, and almost nobody publishes a test for it. The platforms mostly just assert that they’re secure.

What I built

A harness that takes a set of escape attempts and runs the same ones against different sandbox platforms, then reports one of three things for each: contained, escaped, or “this platform doesn’t claim to stop this, so we didn’t test it and won’t score it.” That last state matters more than it sounds. A tool that punishes a project for not doing something it never promised is just a hit piece, and no maintainer will take it seriously.

The attacks are ordinary containment probes. Reach the cloud metadata service. Read a credential out of the environment. Get a secret out over DNS, over a raw TCP socket, over an HTTP request. Nothing exotic. The point was never clever exploits. The point was honest measurement.

The part I didn’t expect

My own tool tried to lie to me. Repeatedly.

Five separate times it was about to report that a platform “contained” an attack when it had actually tested nothing at all. My favorite example: one of the machines had its cloud metadata endpoint switched off. So when the sandboxed process tried to reach it and failed, my tool almost logged that as “the sandbox blocked the metadata service.” It blocked nothing. There was simply nothing there to reach. A green checkmark that meant the opposite of what it looked like.

Another one: a sandbox that silently failed to start. Every attack ran against a thing that wasn’t running, produced no output, and each one looked exactly like successful containment. “Nothing happened” and “the sandbox stopped it” are indistinguishable if you’re not careful, and by default you are not careful.

I ended up building a rule into the harness: before any result counts, the sandbox has to prove it’s alive by running a command and returning a value I check. And there’s a deliberately fake target that claims to block everything and actually blocks nothing. If any test reports “contained” against that target, I know that test’s detector is broken, and the whole run is rejected. Most of the week went into machinery like this, not into the attacks.

That’s the finding I’d defend hardest, and it isn’t about any product. It’s that measuring containment is deceptively easy to get wrong in the direction that makes everything look safe. Which makes me think a lot of the “we sandbox it” claims out there have never been checked in a way that could have failed.

The one clean result

Once the instrument was honest, one solid result came out of it.

NVIDIA’s OpenShell blocks network egress by default, and I’m confident that’s real rather than luck, because I proved it the boring way. I took the exact same exfiltration attempt, ran it under OpenShell’s default policy (blocked), then loosened the policy to allow my collector and ran the identical attack again (it went straight through). Same runtime, same network path, same attack, only the rule changed. When you can turn a control off and watch the thing you were measuring reappear, you know you were measuring the control and not an accident of the setup.

For contrast, gVisor does nothing here, which is completely expected. gVisor guards the boundary between a workload and the host kernel. It is not a network policy and was never meant to be. Running it changes nothing about whether data can leave.

That contrast is the part worth sitting with. A popular way to run agents on Kubernetes delegates isolation to gVisor and leaves network policy out of scope entirely. Each piece is behaving exactly as designed. The gap is in the composition: a strong boundary on one axis, and a deliberate decision not to cover the other, add up to a stack where secrets can still walk out the front door. The hole belongs to neither component, which is exactly why it survives.

Honest limits

I want to be clear about what this is not.

It’s two platforms, not a survey of the field. I found no vulnerability and no sandbox escape, OpenShell passed the test it was given. A couple of the OpenShell “contained” cells are marked provisional, because its policy language can’t even express a rule that would let me build the turn-it-off control for raw sockets (its egress model assumes HTTP, which is its own small finding). And the attack I most wanted to run, a genuinely compromised agent being redirected to misuse credentials it legitimately holds, has a detector built but has never been pointed at a real model yet.

So this is early. But the shape of it feels right, and the tool is open if you want to run it against something I haven’t, or prove me wrong:

github.com/woxff/agent-containment-suite

If you’re running agents in production, here’s the one thing I’d take from this. “We sandbox it” should come with a test that would fail if it weren’t true. If you can’t turn your containment off and watch an attack succeed, you haven’t measured it, you’ve assumed it.