CyborgDJ as Controller — the Interface Between Inference and the Engines

The dj-score library already has the shape of an MCP server: one service per use case, results as JSON-ready records, a cache as shared state. Decided: CyborgDJ becomes the controller (inference holds the set) and the code does the busywork. The design question is where inference sits in each loop. It calls tools directly when planning, as with REA; it writes a deterministic search it then stays out of when tuning, as the Zelda bot did. The second loop is where the open “tuning knob” problem lives.

Links: The RE Build Tool — the same controller pattern for disassembly projects (2026-10-26), Two Libraries Out of a Grown Tool — the restructuring that produced the two tool surfaces, dj-render — the render library (2026-10-22), the second tool surface next to dj-score, The Cyborg Model (the Cyborg DJ worked example this extends), An AI Speedruns NES Zelda (inference at authoring time, a 4-verb socket at run time), REA (inference driving tools over MCP at run time), Crossover Calibration (the tuning-knob problem, designed, not built), DJ EQ Blending (real-time knobs, “the spec is the stable contract”), Eyes and Ears for LLMs (why the model’s ears must be a feature pipeline), Oracles Are Objective Functions, Programmatic DJ Mixing Tools, CyborgDJ, dj-score, Well-Structured Python at Scale

The idea, in Chris’s words

“After some recent projects I have seen, I was thinking that this library can be or is similar to an MCP server. If so, the AI inference can figure out instructions which are then fed to the MCP to process, this way Cyborg DJ focuses on controlling the task at hand while the code does the busywork. I was thinking this was similar to the Zelda controller and the decompiler projects we reviewed recently.”

Asked whether CyborgDJ becomes the agent (holding the set, calling dj-score and render tools) or stays the renderer: “closer to 1.” On speed: “Zelda was real-time so the need for speed was apparent there.. since CyborgDJ’s output is a wav file, I am not sure we need complete speed, but we still have not solved the ‘tuning knob’ problem, so there might be a time and place for that.” On the interface: “Zelda had to poke lua, REA was tools driven by inference, so this seemed like the same pattern.”

Why dj-score is already most of an MCP server

Two placements of inference, from the two references

The Zelda run had two interfaces: the agent wrote Python at authoring time, and that Python drove the emulator through a four-verb Lua socket at run time. Inference never touched a frame. REA is the other placement: inference calls MCP tools over Ghidra at run time, question by question. CyborgDJ needs both:

Loop Speed needed In the loop Interface
Plan the set (order, keys, overlap bars) minutes is fine inference MCP over dj-score, from the cache (REA-style)
Render the spec to a WAV seconds to minutes CyborgDJ engine an MCP verb: render(spec)
Verify the render seconds dj-score on the output an MCP verb: measure the WAV (beat alignment, bass handoff, collision): the Zelda replay check
Tune a knob (crossover Hz, EQ curve) fast per trial a deterministic sweep; Chris judges Zelda-style: inference configures the sweep once; code turns the knob

Turning a knob one tool call at a time would put inference in an inner loop, which is slow and costly. This is the REA teardown rule again: “only use tokens when we need inference.” The Zelda analogue is segment search from a bookmark: 60–90 seeded attempts, each scored to one number, best kept. For the DJ case: from cached analysis and pre-stretched audio, render an ~8-bar window per candidate (the calibration design’s window), score each by measurement, and loop the best few for Chris’s ear. Inference picks what to sweep and reads the results. The human holds the criteria, as the Zelda owner did (he wrote no code and supplied every big correction).

Two Zelda lessons that apply to the knob loop

Limits

What this changes next

The CyborgDJ audit (before migration step 7) sorts each script by the loop it serves: planning, measuring and verifying go to dj-score tools; rendering stays in CyborgDJ as tool verbs; the analyze_btb_* experiments are likely early knob-tuning attempts and evidence for what the sweep should score. cyborgdj/loader.py’s own downbeat detector belongs with step 6 (the grid swap).

Open Questions

Tags

music, agents, claude, ai