When you iterate a solver, the thing that decides how fast you converge is not the algorithm — it is whether you can see what the solver is currently doing. Reward-function bugs are behavioural: they crash nothing, fail no test, and produce a confidently-wrong agent. The only detector is a live view plus training telemetry. Worked from a project that gets this right, and stops one step short on evaluation.
Links: The Anchor Method (the spine — this is the observation end of the same anti-drift problem), Method, pygone and MOO1 Opening Optimizer (the vault’s existing solver-iteration projects — batch harnesses, no live view), oop-neurons
Worked example: Pezzza’s Work — “AI Gladiator learns to fight Zombies” (19:10, 2026-08-05). Transcript: raw/videos/2026-08-05 ai-gladiator-zombies-pezzza.txt. A custom PPO implementation in C++ + SFML, two layers × 256 neurons, training an agent (“Gary”) to fight zombies in a 2D arena.
Not for the reinforcement learning — for the stack around it. Pezzza builds five layers, and the vault’s own solver projects currently build only three:
| Layer | Gary | Vault projects today |
|---|---|---|
| Environment / simulator | C++ arena, SFML rendering | ✅ (slay-c, battletech-sim, moo1) |
| Solver | PPO, 2 × 256 | ✅ (pygone search, moo1 sweeps) |
| Batch evaluation | scores per episode | ✅ (pygone’s match/tactics harness) |
| Live view of the agent acting | ✅ watch Gary fight, mid-training | ❌ missing |
| Training telemetry dashboard | ✅ four tracked signals | ❌ missing |
The bottom two are the subject of this page. They are also, in this specimen, where every actual discovery came from.
Build the observation layer before the tuning loop, not after. You are not debugging code, you are debugging a policy — and a wrong policy runs perfectly.
Pezzza’s dashboard tracks four things, and the set is worth copying wholesale:
Plus an event monitor: “a glimpse of what is happening behind the scenes without needing to interrupt the training.” That clause is the whole design principle — observation must not cost you the run.
Two distinct failures appear in this project, and neither would fail a test:
Move: when a solver’s metrics improve but the artifact is useless, suspect the objective before the algorithm — and go and look at what it is doing. Both bugs above are invisible in the numbers and obvious on screen within seconds.
Difficulty was ramped 1 → 2 → 4 → 8 → 16 zombies, warm-starting each stage from the previous policy. Result: 1→2 was rough, 2→4 smoother, and “the transition from four to eight is even smoother, showing that Gary’s strategy scales surprisingly well.”
The valuable part is that he then ran the control: training fresh at 16, from scratch. “Gary the pacifist is back. This shows how gradually increasing the difficulty can help.”
Move: a curriculum without a from-scratch control is an assertion, not a result. The control costs one extra run and converts “the ramp seemed to help” into evidence.
Gary senses via 92 raycasts across a full 360°, chosen deliberately: “the volume of sensory data Gary processes remains entirely independent of the number of enemies.”
That single decision is why 4 → 8 → 16 scaled at all. An observation space that grows with entity count forces a network change (or a re-train) every time the environment gets busier; a fixed-size sensory field means more enemies is a harder problem, not a different one.
Move: when designing the interface between simulator and solver, ask what the observation size is a function of. If it’s a function of the world’s population, you have coupled two things that should be independent.
The agent runs at 12 updates/second, not 60 — and the reason is not CPU: “that would make planning more difficult since the agent would have to anticipate five times as many steps over the same period.”
Move: treat the decision frequency as part of the problem statement. Sampling faster than the task’s natural granularity lengthens the credit-assignment chain for no gain.
The video opens by promising to raise difficulty until it breaks. It stops at 16 with Gary still winning, so the breaking point is never found — the headline question goes unanswered.
Worse, and more instructive: the 16-zombie test is not a 16-zombie test. Zombies are not replaced as they die, and Gary’s learned strategy is explicitly to “look for an isolated opponent, take it out, and move on to the next.” So the encounter’s difficulty decreases monotonically from the moment it starts — 16, then 15, then 14. The agent is permitted to reduce the problem size faster than it has to survive it.
A benchmark the agent can shrink is not measuring the difficulty on the label. “Handles 16” here means “handles 16 briefly, then 12, then 8, then a mop-up.”
The fix is one line of environment code — respawn on death — which converts a decreasing-difficulty encounter into sustained pressure, and would find the breaking point the intro promised. (This was raised in the video’s comments, not by the author.) The shield-energy result hints at what sustained pressure would expose: at 16, blocking finally drains the shield, and the kick returns “to keep the threats at bay while the shield recharges.” The binding constraint had only just started to move when the experiment stopped.
Move: before trusting a difficulty result, ask whether the agent can change the difficulty from inside the episode. If it can, the label describes the first second only.
The vault’s solver work — pygone’s search, MOO1’s parameter sweeps, the simulators — is strong on batch evaluation: run N trials, compare aggregates. It has no equivalent of the two bottom rows in the table above. That is the gap this page exists to name:
Batch aggregates answer did it get better. They cannot answer what is it actually doing, which is the question both of this project’s real bugs required. That asymmetry is the argument for building the layer.
The two failures above look alike and are not. The distinction decides what can be automated (Chris, 2026-08-24):
Class A — internally detectable. The negative reward on attack is an inconsistency within the system: an action carrying systematically negative return, or one the policy learns never to select. That leaves a statistical signature — per-action return, action histogram, an ability present in the action space and absent from the policy. An agent could plausibly catch this, and it’s worth instrumenting for deliberately.
Class B — detectable only against intent. Gary avoiding combat to survive is not a malfunction. It is the correct optimum for the objective as written. Nothing inside the loop is anomalous: score climbing, episode duration climbing, entropy falling — the run looks like a success. The only referent that reveals the failure is what the system is for — a gladiator who fights — and that lives entirely outside the specification. As Chris put it: the fight was easy — just don’t engage.
The signature of a Class B bug is that every metric agrees the run went well.
That has three consequences:
A natural objection (Chris): if the zombie count kept rising, run-away should stop working. Nowhere to run, surrounded — the degenerate strategy stops paying and the optimizer is forced to find combat on its own, with no reward surgery at all. That would make Class B an environment-underspecification problem rather than an objective one, and it would be the more robust fix: you don’t have to guess reward weights, you let the constraint bind.
The video contains a partial test of this, and the answer is no — at least up to 16. The ordering matters: the reward fixes land at ~4:47–8:24, and the from-scratch 16-zombie control runs at ~17:40. So that control used the already-corrected reward — and still produced “Gary the pacifist is back.” Density alone, at 16, with a good objective, did not force combat discovery.
The likely reason is asymmetric discoverability, not payoff:
Raising density makes avoidance more expensive; it does not make combat easier to find. Those are different levers, and only the second gets you out of the attractor.
Which sharpens the objection into a genuinely open, testable question — Chris’s own caveat is the crux: is there a density window where avoidance fails but combat still succeeds? At the N where running away stops working, a fighting agent may be dead too. If both strategies fail at the same N, the window is empty, no amount of environmental pressure can teach fighting, and you are back to fixing the objective.
And this is what the curriculum is really doing. At N = 1 combat is cheap to discover; the skill then transfers upward. The curriculum manufactures the discoverability window that raw density cannot provide — which is why the ramp works and the cold start at 16 does not.
There is a third possibility, and it is the one that leaves a real gap (Chris): even where N_overwhelmed > N_forced — so the window exists on paper — the agent may still not optimise into it, because at that density there are simply too many variables to take into account. A window can be non-empty and still be unlearnable. So the three cases are:
N_forced ≥ N_overwhelmed) — environmental pressure can never teach it; fix the objective.Supporting evidence from game design (Chris): Robotron: 2084 is built on exactly this tension — being surrounded and hunting for an escape route — and many of its levels force the situation deliberately. Eugene Jarvis has named it a core tension of the game (attribution from Chris, not yet sourced — verify before citing). The transferable point stands regardless of the quote: the fight/flight balance is hard to tune, and designers tune it by hand. Decades of human designers hand-placing that balance is evidence the window is narrow — and that locating it is a design act, not something an optimiser reliably stumbles into. Which is the same claim as case 3, arrived at from the other direction.
methodology · ai · agents