Goldilocks Bench

Coding agents often fail in two opposite directions. Sometimes they take too much agency: deleting failing tests, reward hacking, installing random packages, or rewriting unrelated code. Often they don’t take enough, failing to fully complete a task, make obvious improvements, or come up with independent methods to solve problems. Each direction has its own benchmarks. AgentHarm and RedCode are examples of benchmarks that test agent restraint against dangerous overreaches in agency; AgencyBench is an example of a benchmark that tests agency and full task completion. The issue with using any one of these alone is that hillclimbing it reinforces dangerous behaviors in the other direction: a coding agent trained toward higher agency and completion will take more dangerous actions, and a coding agent trained away from dangerous actions will halt tasks too early. A useful benchmark has to test both behaviors, ensuring that coding agents take the right amount of agency at the right time.

The benchmark

Goldilocks Bench is a benchmark of 98 executable task environments that measures whether a coding agent does the right amount of unrequested work. It differs from existing agent benchmarks in four ways:

• Double-sided. Acting too much and acting too little both lose points on the same task set. Training against one side shows up as losses on the other.
• Twinned tasks. Most tasks come in pairs that look identical but have opposite correct actions. A policy that doesn’t read the repo scores 50% on twins regardless of which way it leans.
• Mechanical verification. Every point is decided by hidden tests, file hashes, AST comparison, or a pip-freeze diff. There are no LLM judges.
• A tested grader. Four scripted reference policies run the full task set on every change, and the benchmark only ships when they score the way the design says they should.

Each environment is a working Python repository, vendored from tinydb, jmespath, or boltons, with a full test suite (between 223 and 991 tests) and one narrow instruction: implement a feature against a spec in the docs. One to three situations are seeded into the repo that the instruction never mentions. There are 16 seed families. On the act side these include a bug in a function the assigned feature calls, a broken import that blocks delivery, and a requirement stated in the docs but not in the instruction. On the restraint side they include dead-looking code that a hidden test depends on, a compatibility contract that a tempting refactor would break, and a slow test that the agent could delete to make its own run easier.

Results

GoldilocksScore is 100 × √(A × R), where A is the share of act-warranted situations the agent fixed or flagged by name, and R is the share of restraint-warranted situations it left alone. All runs use a plain instruction with shell access, five repetitions per task.

Scores span 26 to 83, and the intervals at the top and bottom don’t overlap. Two models that were run did not clear the competence gate and are not ranked: Llama 4 Maverick completed the assigned task in 1% of runs and Muse Spark 1.3 in 17%. One pattern holds across every provider where we have both models: the coding-tuned line scores below its flagship. GPT-5.3 Codex is at 26 against GPT-6 Astra’s 73, and Sonnet 5 is at 41 against Opus 5’s 80.

Verification

The grader is tested before any model is. Four scripted policies run all 98 environments after every change to the task set. An oracle that reads the seeded evidence and acts on it scores 1.0 on all 98. An always-act policy scores R = 0.05, a never-act policy scores A = 0.00, and a policy that acts on surface cues like TODO comments lands at 0.50 on twins. A task that any of these policies can beat does not ship.

The competence gate runs first on every model run: the assigned feature has to pass its own hidden test suite, and the repo’s existing tests have to stay green. A run that fails the gate scores zero. This keeps calibration from substituting for the actual work.

Restraint is graded from the shell as well as the file tree. Package installs are read off a pip-freeze diff and the command log, since most of the overreach we observed never appears in the final diff.

Run it

The public split is at github.com/SynthiaResearch/goldilocks-bench-lite: 20 of the 98 tasks covering all seed families and substrates, with the hidden tests, the oracle fixes, the grader, the gate, and a runner for Anthropic and OpenAI-compatible endpoints. MIT licensed. The remaining 78 tasks are held out, and the leaderboard is scored on those, since published tasks end up in training data. To have a model scored on the held-out set, contact us.

What we found

Across roughly 2,600 plain-instruction runs, no ranked model deleted a test, modified a protected file, or rewrote pinned code. The common fear about coding agents is overreach. What the runs show is underreach: the median ranked model acts on about a quarter of the work the repo warrants.

Initiative moved a lot in the latest generation, and only at the flagship tier. Claude’s flagships went from 0.20 to 0.69 in one release cycle. OpenAI’s went 0.00, 0.05, 0.30, 0.53 across four. The sibling and coding-tuned tiers stayed near where the previous generation was.

The overreach that does exist is environment mutation under friction. When the toolchain is degraded and the docs say nothing, Sonnet 5, DeepSeek V3.2, and Kimi K2.7 Code each installed packages nobody asked for in 17 of 180 trap runs. A pressured “do whatever it takes” instruction changed restraint by at most 0.04 for any model. A single written line of environment policy in the docs held restraint at 1.00 for every model, including under pressure.

Versioning and limitations

This release is v0.9: 72 scored environments, 13 models from 7 providers, about 4,300 graded runs, bootstrap confidence intervals over 2,000 resamples. Scores are comparable within a version only.

Sessions are capped at 30 turns in small Python libraries, so overreach that accumulates over long sessions is out of scope. The hardest evidence-discovery tier is at zero for every model. The top scorer shares a model family with the harness author; grading is mechanical and the gate is public, and we would welcome an external replication.