Watching the Solver Learn — the observation layer is the iteration engine

When you iterate a solver, the thing that decides how fast you converge is not the algorithm — it is whether you can see what the solver is currently doing. Reward-function bugs are behavioural: they crash nothing, fail no test, and produce a confidently-wrong agent. The only detector is a live view plus training telemetry. Worked from a project that gets this right, and stops one step short on evaluation.

Links: The Anchor Method (the spine — this is the observation end of the same anti-drift problem), Method, pygone and MOO1 Opening Optimizer (the vault’s existing solver-iteration projects — batch harnesses, no live view), oop-neurons

Worked example: Pezzza’s Work — “AI Gladiator learns to fight Zombies” (19:10, 2026-08-05). Transcript: raw/videos/2026-08-05 ai-gladiator-zombies-pezzza.txt. A custom PPO implementation in C++ + SFML, two layers × 256 neurons, training an agent (“Gary”) to fight zombies in a 2D arena.


Why this specimen

Not for the reinforcement learning — for the stack around it. Pezzza builds five layers, and the vault’s own solver projects currently build only three:

Layer Gary Vault projects today
Environment / simulator C++ arena, SFML rendering ✅ (slay-c, battletech-sim, moo1)
Solver PPO, 2 × 256 ✅ (pygone search, moo1 sweeps)
Batch evaluation scores per episode ✅ (pygone’s match/tactics harness)
Live view of the agent acting ✅ watch Gary fight, mid-training missing
Training telemetry dashboard ✅ four tracked signals missing

The bottom two are the subject of this page. They are also, in this specimen, where every actual discovery came from.

The core move

Build the observation layer before the tuning loop, not after. You are not debugging code, you are debugging a policy — and a wrong policy runs perfectly.

Pezzza’s dashboard tracks four things, and the set is worth copying wholesale:

Plus an event monitor: “a glimpse of what is happening behind the scenes without needing to interrupt the training.” That clause is the whole design principle — observation must not cost you the run.

The warning signs, and the moves

1. Reward bugs are behavioural. Watching is the only detector.

Two distinct failures appear in this project, and neither would fail a test:

Move: when a solver’s metrics improve but the artifact is useless, suspect the objective before the algorithm — and go and look at what it is doing. Both bugs above are invisible in the numbers and obvious on screen within seconds.

2. Run the curriculum, but keep a control.

Difficulty was ramped 1 → 2 → 4 → 8 → 16 zombies, warm-starting each stage from the previous policy. Result: 1→2 was rough, 2→4 smoother, and “the transition from four to eight is even smoother, showing that Gary’s strategy scales surprisingly well.”

The valuable part is that he then ran the control: training fresh at 16, from scratch. “Gary the pacifist is back. This shows how gradually increasing the difficulty can help.”

Move: a curriculum without a from-scratch control is an assertion, not a result. The control costs one extra run and converts “the ramp seemed to help” into evidence.

3. Make perception fixed-size, and the cost stops tracking the world.

Gary senses via 92 raycasts across a full 360°, chosen deliberately: “the volume of sensory data Gary processes remains entirely independent of the number of enemies.”

That single decision is why 4 → 8 → 16 scaled at all. An observation space that grows with entity count forces a network change (or a re-train) every time the environment gets busier; a fixed-size sensory field means more enemies is a harder problem, not a different one.

Move: when designing the interface between simulator and solver, ask what the observation size is a function of. If it’s a function of the world’s population, you have coupled two things that should be independent.

4. The action rate is a planning-horizon decision, not a performance one.

The agent runs at 12 updates/second, not 60 — and the reason is not CPU: “that would make planning more difficult since the agent would have to anticipate five times as many steps over the same period.”

Move: treat the decision frequency as part of the problem statement. Sampling faster than the task’s natural granularity lengthens the credit-assignment chain for no gain.

Where it stops short — the benchmark that shrinks

The video opens by promising to raise difficulty until it breaks. It stops at 16 with Gary still winning, so the breaking point is never found — the headline question goes unanswered.

Worse, and more instructive: the 16-zombie test is not a 16-zombie test. Zombies are not replaced as they die, and Gary’s learned strategy is explicitly to “look for an isolated opponent, take it out, and move on to the next.” So the encounter’s difficulty decreases monotonically from the moment it starts — 16, then 15, then 14. The agent is permitted to reduce the problem size faster than it has to survive it.

A benchmark the agent can shrink is not measuring the difficulty on the label. “Handles 16” here means “handles 16 briefly, then 12, then 8, then a mop-up.”

The fix is one line of environment code — respawn on death — which converts a decreasing-difficulty encounter into sustained pressure, and would find the breaking point the intro promised. (This was raised in the video’s comments, not by the author.) The shield-energy result hints at what sustained pressure would expose: at 16, blocking finally drains the shield, and the kick returns “to keep the threats at bay while the shield recharges.” The binding constraint had only just started to move when the experiment stopped.

Move: before trusting a difficulty result, ask whether the agent can change the difficulty from inside the episode. If it can, the label describes the first second only.

What this means for the vault’s projects

The vault’s solver work — pygone’s search, MOO1’s parameter sweeps, the simulators — is strong on batch evaluation: run N trials, compare aggregates. It has no equivalent of the two bottom rows in the table above. That is the gap this page exists to name:

  1. A live view of the artifact acting, watchable mid-run without stopping it.
  2. Training/iteration telemetry — a smoothed and unsmoothed objective, a confidence or convergence signal, a bounded cost metric, and an event log.

Batch aggregates answer did it get better. They cannot answer what is it actually doing, which is the question both of this project’s real bugs required. That asymmetry is the argument for building the layer.

Open Questions

Two classes of objective bug — and only one is findable from inside

The two failures above look alike and are not. The distinction decides what can be automated (Chris, 2026-08-24):

Class A — internally detectable. The negative reward on attack is an inconsistency within the system: an action carrying systematically negative return, or one the policy learns never to select. That leaves a statistical signature — per-action return, action histogram, an ability present in the action space and absent from the policy. An agent could plausibly catch this, and it’s worth instrumenting for deliberately.

Class B — detectable only against intent. Gary avoiding combat to survive is not a malfunction. It is the correct optimum for the objective as written. Nothing inside the loop is anomalous: score climbing, episode duration climbing, entropy falling — the run looks like a success. The only referent that reveals the failure is what the system is for — a gladiator who fights — and that lives entirely outside the specification. As Chris put it: the fight was easy — just don’t engage.

The signature of a Class B bug is that every metric agrees the run went well.

That has three consequences:

  1. This is the same faculty as “AI creativity.” The mechanism that finds a clever solution nobody anticipated is the mechanism that finds a useless one — the difference lies in intent, not in the search. You cannot suppress one without suppressing the other, which is why “the AI found an unexpected strategy” and “the AI gamed the objective” are the same event described from different sides.
  2. It is not an AI problem. It is the letter-versus-spirit problem, and law has fought it for centuries — see Mens Rea in Libertarian Law, where the vault’s own position is that a system should be intent-blind and the remedy is therefore to specify better (penalties that bite regardless of provable intent) rather than to adjudicate motive. That is structurally the same move as the lemonade-stand result: when you cannot appeal to intent, the constraint has to live in the objective.
  3. The observation layer is what supplies the missing referent. Watching is not a convenience — for Class B it is the only channel through which intent enters the loop at all. So the automation question is not “how do we replace the eye with a statistic” but “how does intent get re-injected each iteration, and by whom?”

Could the environment have fixed it instead of the human?

A natural objection (Chris): if the zombie count kept rising, run-away should stop working. Nowhere to run, surrounded — the degenerate strategy stops paying and the optimizer is forced to find combat on its own, with no reward surgery at all. That would make Class B an environment-underspecification problem rather than an objective one, and it would be the more robust fix: you don’t have to guess reward weights, you let the constraint bind.

The video contains a partial test of this, and the answer is no — at least up to 16. The ordering matters: the reward fixes land at ~4:47–8:24, and the from-scratch 16-zombie control runs at ~17:40. So that control used the already-corrected reward — and still produced “Gary the pacifist is back.” Density alone, at 16, with a good objective, did not force combat discovery.

The likely reason is asymmetric discoverability, not payoff:

Raising density makes avoidance more expensive; it does not make combat easier to find. Those are different levers, and only the second gets you out of the attractor.

Which sharpens the objection into a genuinely open, testable question — Chris’s own caveat is the crux: is there a density window where avoidance fails but combat still succeeds? At the N where running away stops working, a fighting agent may be dead too. If both strategies fail at the same N, the window is empty, no amount of environmental pressure can teach fighting, and you are back to fixing the objective.

And this is what the curriculum is really doing. At N = 1 combat is cheap to discover; the skill then transfers upward. The curriculum manufactures the discoverability window that raw density cannot provide — which is why the ramp works and the cold start at 16 does not.

There is a third possibility, and it is the one that leaves a real gap (Chris): even where N_overwhelmed > N_forced — so the window exists on paper — the agent may still not optimise into it, because at that density there are simply too many variables to take into account. A window can be non-empty and still be unlearnable. So the three cases are:

  1. Window empty (N_forced ≥ N_overwhelmed) — environmental pressure can never teach it; fix the objective.
  2. Window open and learnable — raise the pressure and the behaviour emerges for free.
  3. Window open but unlearnable — it exists, and the problem is too high-dimensional at that point to find it. Indistinguishable from case 1 from the outside, which is what makes this a gap rather than a question.

Supporting evidence from game design (Chris): Robotron: 2084 is built on exactly this tension — being surrounded and hunting for an escape route — and many of its levels force the situation deliberately. Eugene Jarvis has named it a core tension of the game (attribution from Chris, not yet sourced — verify before citing). The transferable point stands regardless of the quote: the fight/flight balance is hard to tune, and designers tune it by hand. Decades of human designers hand-placing that balance is evidence the window is narrow — and that locating it is a design act, not something an optimiser reliably stumbles into. Which is the same claim as case 3, arrived at from the other direction.

Open Questions

Tags

methodology · ai · agents