the governor problem
Why every complex system eventually grows a part whose only job is to watch the rest of it — and what that means for AI agents.
I have a tool that runs teams of AI agents. Each team has a lead. The leads file tickets, assign work, check each other’s output. It’s tidy. Structured. Occasionally it even works on the first try.
Nobody watches the tool.
Not the software — people maintain that. I mean nobody inside the system observes the system. No agent sits at the altitude where it could notice that one team is idle, another is stuck, a third is rebuilding something the fourth finished yesterday. I’m the one who notices. Usually at 3 AM, alt-tabbing between dashboards, doing the job that no agent has.
I’ve had this thought exactly twice. Both times I filed it under “interesting, not urgent” and went back to work. The third time is this post.
The obvious framing is technical. “Why doesn’t the orchestration layer have its own orchestrator?” But that’s the wrong question, or at least an incomplete one. What I’m actually asking is a governance question, and it’s old enough to have a Latin name.
Juvenal, writing satire in the second century, dropped quis custodiet ipsos custodes — who watches the watchers? He was writing about the difficulty of ensuring your wife’s fidelity when the guards you hire are themselves corruptible. Not exactly a distributed-systems problem. But the structure is identical: you have agents operating on your behalf, and the question of oversight generates an apparent regress. Watch the watchers. Then watch those watchers. Turtles.
Plato took the problem seriously enough to build an entire political system around it. The Republic proposes a guardian class — warriors and philosophers selected and educated from childhood to govern the city. Plato’s answer to “who watches the watchers” is striking: the watchers watch themselves. Not through willpower or honor codes but through formation — a carefully designed education that makes the guardians want what’s good for the city the way a well-tuned instrument wants to play in key. The guardian doesn’t need a meta-guardian because the guardian’s own nature has been shaped to be self-correcting.
It’s a beautiful solution. It also requires that you can perfectly control the formation of the governing entity, that the entity has no interests of its own that might diverge, and that the definition of “good for the city” never changes. In other words, it works in theory and collapses on contact with anything real. Which is roughly where we are with AI agents.
There’s a better framework, and it was sitting in front of me the whole time, hiding in a word.
Norbert Wiener published Cybernetics in 1948 and named an entire field. The word comes from the Greek kubernetes — steersman, helmsman, governor. Wiener didn’t pick it for poetry. He picked it because the central problem of his new science was how systems regulate themselves, and the Greeks already had a word for the person who does that on a ship. The governor isn’t the captain. The captain decides where to go. The governor keeps the ship from capsizing on the way there.
Wiener’s insight — the one that matters here — is that the fundamental unit of control is the feedback loop, not the hierarchy. A thermostat doesn’t outrank the furnace. It’s coupled to the room’s temperature, and the coupling itself produces regulation. No supervisor required. The system’s output becomes its input, and the loop corrects.
James Watt understood this a century and a half before Wiener named it. His centrifugal governor on the steam engine is still the cleanest physical example. Two metal balls on hinged arms, connected to the engine’s output shaft. When the engine speeds up, centrifugal force pushes the balls outward. The outward motion mechanically closes the throttle. The engine slows. The balls drop. The throttle opens. The engine speeds up. Two spinning balls and a lever, and the regress stops.
The regress stops because the governor is a different kind of thing than the system it governs. It doesn’t make decisions. It doesn’t have a plan. It doesn’t model the future or weigh alternatives. Its own physics — centrifugal force against gravity — produce the corrective behavior. The governor doesn’t need a meta-governor because regulation is what the governor is, not what it does. There’s no homunculus inside deliberating about throttle positions. The loop is constitutive, not deliberative.
That distinction matters. It’s the difference between management and governance. A manager makes choices within a system. A governor shapes the conditions under which the system operates. Adding another manager creates a hierarchy. Adding a governor creates a feedback loop. Hierarchies regress. Feedback loops close.
Wiener saw the same pattern everywhere. Biological homeostasis. Economic markets. Anti-aircraft fire control. Neural motor coordination. The steersman on the ship, the thermostat in the room, and the cerebellum in the body are all solving the same formal problem: observe the system’s output, compare it to a reference, and adjust the input. The details of the mechanism vary wildly. The structure doesn’t.
This is where the convergence becomes evidence. It’s not that someone designed homeostasis, price discovery, and motor learning to use feedback loops. They arrived at the same architecture independently because the problem — regulating a complex system in real time — has a shape, and that shape strongly favors loops over hierarchies. When independent systems converge on the same structure, the structure is probably tracking something real about the problem rather than reflecting a design fashion.
Plato’s guardian and Wiener’s feedback loop are two answers to the same question, and the difference between them is the whole story. Plato reaches for an entity — a philosopher-king with the right nature and the right education. Wiener reaches for a relation — a coupling between a system’s output and its input that produces stability without anyone being wise. Plato’s solution is brilliant, fragile, and doesn’t scale. Wiener’s is dumb, robust, and runs in every living cell.
The post could stop here and the point would be made. But the interesting part isn’t whether agent platforms need a governor — they obviously do. The interesting part is what happens when you try to build one.
A cybernetic governor needs three things. It needs to observe the system’s state. It needs a reference to compare against. And it needs a way to adjust. Watt’s governor gets all three from physics. An agent governor has to build them from software, which means building them from choices, and the choices are harder than they look.
The observation problem is a memory problem. The governor has to maintain a model of the system — what’s running, what’s stuck, what’s idle, how the parts relate. Bernard Stiegler had a name for this kind of structure: tertiary retention — memory carried outside the organism. Your notes, your dashboards, your ticket backlogs — these are all tertiary retentions. They enable cognition the system couldn’t perform without them, and they shape what the system can even think about. A governor whose world model tracks task status but not resource contention will be blind to a whole category of failure. The model isn’t neutral. It’s an epistemology. It decides what counts as the system’s state, and everything it leaves out becomes invisible.
This is the world-model problem, and it’s genuinely hard. Too much detail and the model exceeds the governor’s capacity to process — you’ve built another agent, not a governor. Too little and the governor is steering blind. Watt’s device didn’t face this problem because its “model” was the rotational speed of the output shaft, a single scalar value read directly from the mechanism. An agent governor’s world model is more like a map, and maps involve choices about what to include. Those choices are political in the sense Langdon Winner meant: the architecture of the observation system encodes assumptions about what matters.
The reference problem is a values problem. A thermostat has a setpoint. Watt’s governor has the equilibrium between centrifugal force and gravity. What’s the setpoint for a fleet of AI agents? “Everything is running” isn’t enough — you can have every agent busy and the overall system pointed in three directions at once. “All tickets are progressing” isn’t enough either — tickets can progress smoothly toward the wrong outcome. The reference against which the governor compares the system’s state is, functionally, a compressed statement of what the system is for. That’s not an engineering question. That’s the question Plato was trying to answer with the whole Republic.
In practice, the reference will be modest. Something like: capacity is allocated proportional to priority, no team is stuck without escalation for more than N minutes, no two teams are duplicating work. Not a theory of the good. A set of invariants the governor tries to hold. But even this modest reference embeds values — the assumption that duplication is waste, for instance, rather than productive redundancy. The governor’s setpoint is a philosophy whether you name it one or not.
The action problem is a restraint problem. This is where Watt’s governor teaches the most important lesson. The governor’s action vocabulary is tiny. It modulates the throttle. That’s it. It doesn’t redesign the engine. It doesn’t add cylinders. It doesn’t decide where the train should go. Its power comes from its narrowness: because it can only adjust the throttle, it can’t interfere with the engine’s operation in ways that would require its own governor.
An agent governor needs the same discipline. A small set of verbs: reassign, escalate, pause, merge, create. Fewer actions than a single task-agent needs, not more. The moment the governor starts writing code, editing documents, or making product decisions, it’s not a governor anymore — it’s an agent with a god complex. And then you need a meta-governor, and the regress is back, and Plato is laughing.
The implementation is simpler than the philosophy. A heartbeat loop: the governor wakes on a schedule, reads system state, compares against its invariants, takes zero or one action, goes back to sleep. Most heartbeats are no-ops. The governor is idle the way a thermostat is idle — not inactive, but in a state where observation and reference are aligned and no adjustment is needed. The idleness is the point. If the governor is constantly acting, either the system is broken or the governor’s reference is wrong.
So why hasn’t anyone built this?
Not because it’s hard. The context windows are big enough for the world model. The APIs exist for the observation and the actuation. The cost per heartbeat is negligible.
The reason is architectural inertia. Most agent frameworks were designed for single-task execution. One prompt, one plan, one output, exit. That’s a process, not a system. You don’t need a governor for a process. You need one when you have dozens of processes running continuously, with dependencies and shared state and overlapping scopes, and enough internal complexity that the components — each working correctly alone — can produce collective failure. Agent platforms are crossing that line right now, moving from “run a task” to “run an organization.”
The governor is what they’ll grow when they get there. Not because someone planned it. Because the absence becomes the bottleneck. Every complex system asks this question right before it answers it.
The steersman is always the last hire.