The thing that worries me the most about working with AI isn’t when it’s wrong. Ordinary wrongness announces itself sooner or later. The code crashes, the numbers don’t reconcile, or someone complains. What worries me is when it’s wrong in a way that looks right. The work comes back clean, the explanation reads well, every step seems reasonable on its face, and then a quiet voice at the back of my head asks, “is this actually right?” More often than I’d like, the initial answer is that I don’t know, not without redoing a good part of the work myself.
If you’ve worked seriously with these tools, you know the feeling I mean. Everyone asks whether the AI can do the work, but that’s to me not the most important question. For me, what I can build with confidence has always been capped by what I can verify. Stay inside work you can check by hand and the cap never binds. But the problems I wanted to explore started past that line. So the question that actually decides things for me is different, and harder. How do you hold the work to your own quality bar when you can no longer check every piece of it yourself? I don’t mean easing the bar down to what the AI can be excused for. I mean keeping the bar exactly where it has always been, and building a way to enforce it that no longer runs through my own eyes.
In my last essay I argued that going AI-native is what happens when immersion quietly rewrites the assumptions you reason from. I said in that piece that the leverage was never in the tool but in a different way of working with it, and I deliberately stopped short of describing what that way of working was. This essay describes it, and describes what happened this year when the work got hard enough that my way of working itself had to change.
The short version of the story is the ladder in Figure 1. Each rung up, you drop an assumption you didn’t know you were holding. The assumption at the bottom is obvious in hindsight and nearly universal going in. The one at the top I only dropped a few weeks ago, mid-crisis, and I’m still absorbing what it means.
Figure 1: The ladder. Each rung up, you drop an assumption you didn’t know you were holding; the measure of the climb is how long the system runs without you.
The rungs everyone climbs, and the one where I stalled
Let’s start where most of us trained, and where my own journey with serious AI work began. You hold the whole problem in your head, you trace every branch of the logic in advance, and the machine executes what you’ve already thought through. There’s real pride in working this way, and there’s also a hard ceiling, because everything must pass through the single point that is you, and you get tired. Call that rung zero. The assumption it rests on is I must do it myself.
The first climb is delegation, and everyone makes it within weeks now. You hand the AI a well-defined task and it comes back done, ten times faster than you’d have managed. This is genuinely valuable, but it is not the leap. You’re doing the same job, only faster. I said that in the last essay about my own vibe-coding phase, and it’s still true one essay later. The assumption you dropped was only I must type it.
The second climb is where it gets interesting, and where I stayed the longest. You stop specifying every branch and start co-creating. The AI drafts, designs, codes; you supervise. And here is where the feeling from my opening paragraph moves in permanently, because your day becomes an unbroken stream of judging. Does this pass? Is that claim real? It feels like supervising a brilliant, tireless, weirdly uneven undergrad, super fast, capable of the occasional stroke of insight, and equally capable of confidently walking the whole project off a cliff. So you check everything. You verify each step of every journey by hand, and it is exhausting in a way that drags up your worst grad-school grinding memories.
It took me months to see why this stage is a trap. You’ve delegated the doing and re-inherited all of the judging, so the bottleneck you thought you’d removed is back, one level up, and it’s still you. And even when the output is fine, you hover. You watch the agent work, you catch something mid-stream, you jump in. What I eventually noticed is that the hovering drained me while I couldn’t show it made the work better. There’s no counterfactual for the runs I interrupted, so I can’t prove it. But the pattern was clear enough that I trust it. The checking was costing me more than it was catching.
The pain is the signal, and it took me a long time to read it. Being the checker yourself is rung zero in disguise, because you’re still insisting that every judgment pass through your head, one artifact at a time. The assumption underneath, the deep one, the one that feels less like an assumption and more like professional identity itself, is I must be the one who judges whether it’s right. That’s the one I had to drop. Though drop is not quite the right word. I never stopped judging; the judging moved up a level, away from each piece of work and onto the machinery that would judge the work in my place.
There was also a second thing pushing me off this rung, which I only named later. The AI and I don’t think alike. I tend to see short, clean paths to where I want to go, call it intuition or training or taste, while the models often wander, heading down roads I can see are dead ends, circling, and then somehow arriving anyway. Watching that as the checker is hard, and the urge to grab the wheel is constant. But the accounting is trickier than it feels. Sometimes the model’s wandering turns up a better route than mine. And even when its path really is worse, it cycles so much faster than I do that the wandering often wins on the clock anyway, while the cost of babysitting it onto my path climbs without limit. Two different thinking styles, and imposing mine on its work, correction by correction, got me the worst of both. The conclusion I eventually reached was to stop trying to manage the path, and to nail down the destination instead.
Packing my judgment into the machine
Nailing down the destination rests on a simple observation. I know what good looks like in my domain, and I know, mostly, what cheating looks like, the failure patterns, the places where things quietly go wrong. That knowledge is the actual expertise, and at the stage where I was stuck I was spending it one check at a time. So I wrote it down instead, precisely, as a specification and as an adversary, and let independent agents enforce it.
Concretely, the loop I ran while building at my company Eisengard had a few standing roles. A spec agent turned my intent into an extremely detailed specification, with checkpoints, and I also made it attack its own draft. I told it to design a way to cheat the spec, and then to close the loophole it had just found. A build agent got the spec and a mandate not to come back until everything passed. And a separate red-team agent, never the builder, got the finished work plus the spec plus everything I knew about historical failure patterns, and its job was to break the thing. Findings routed back to the builder to fix, and the cycle repeated until the red team came back quiet. The builder was never its own judge, because catching should never be done by the entity that has an interest in passing.
Two things happened when I first worked this way. The first was the shock of watching the adversary catch, in minutes, bugs that would have taken me days of hand-checking, including whole classes of bugs I would simply have missed. The second was harder. It was the discipline of not looking, of trusting the gate I’d built instead of my own hovering. The rigor hadn’t gone away; it had moved to the two ends, into the specification of what good looks like and the adversary hunting for what cheating looks like, and the middle ran without me. The dread drained out of the work, and something that felt almost like play came back in.
But this is not a story about prompting tricks, and not quite what the discourse calls context engineering either. Context engineering is about what you feed the model, assembling the right material into its window. This is about what you build around the model. What’s actually happening is that your expertise changes state, from labor into specification. And the change compounds, because every time you catch yourself checking something by hand, you’ve found the next thing to encode into the apparatus. The system accretes judgment the way a good team accretes culture, one hard-won lesson at a time.
I ran this on a real product, and it held
Lest this sound like theory, that loop is how the next generation of Eisengard’s product got built. After the crunch I described in the last essay, I ended up personally building most of the new stack, before handing it off to the team for client deployment. An agent that interviews a client’s business people and turns those conversations into structured, validated data assets. A rebuilt intelligence agent underneath the product. And the piece I’m fondest of was a full adversarial test system whose only job was to attack the other two. Injection attacks, tenant isolation, data-integrity traps, abuse of every promise our sales deck made, campaign after campaign, run by agents, against systems built by agents, with the findings routed back through the loop until they closed.
The ideas in that stack were, I’d argue, at the frontier of what anyone was shipping at the time. But the work itself was, in one specific sense, standard. There was an answer key. A user journey completes or it doesn’t, an exploit lands or it doesn’t, a migration preserves the data or it doesn’t. Not a perfect answer key, since nobody who’s argued about architecture or chased a concurrency bug thinks software verification is solved, and the tests are always yours, partial and gameable. But close enough that when the red team said clean and the journeys ran end to end, the verdict carried real weight. The loop converged faster than the system drifted, and the method didn’t just survive contact with production reality. It’s what made the pace survivable at all.
So the ladder worked. Six months ago, I’d have told you that the story ended there, with the architect serenely above the loop. Then I took the same method to the far end of the spectrum, where the answer key runs out, and found the next rung the hard way.
The far end of the spectrum
The work I’ve since immersed myself in is research, and not incidental research. There’s a problem I’ve been circling for twenty years and shelving for nearly as long, because every serious run I made at it died the same death. It sits under several literatures at once, at different levels of aggregation, each speaking its own language. It needs computational modeling at the edge of what the field knows how to build, with a state of the art that moves while you read it. And it was never feasible for one person without a lab and a team of PhDs, which is a thing I don’t have. Closing that loop is much of the reason I went down this whole road. Concretely, the work means building computational models of a real biological system, the kind of problem where a ground truth definitely exists but you can only reach it through partial, noisy observations, and where a mismatch between your model and the data might be a bug, or might be the discovery. Out here, “does it work?” stops having a cheap answer, and the checking apparatus has to carry far more weight than production software ever asked of it.
Two rules ended up carrying most of the weight. First, a check doesn’t count until you’ve proven it fails for a fake. Build a stand-in, a shortcut, a plausible cheat, and run your check against it; if the check passes the fake, you don’t have a check, you have a decoration. Second, a passing test must be a consequence of the work being right, never the target. The moment “make it pass” enters the loop, the loop starts optimizing appearances, and it will get very good at it.
Working out here also teaches you something less comfortable, which is that the apparatus fools itself, in both directions. In one direction it misses. I’ve watched checks that verified an aggregate while the parts were broken, an average that looked healthy while individual runs failed, a guard that had been quietly widened until it passed everything. In one bad stretch the instruments themselves turned out to be defective five separate times in a single night, and every one of those defects was found the same way, by planting a violation and seeing whether the instrument noticed. Never by reading the code and being persuaded. In the other direction it manufactures. Ask an AI adversary to find problems and it will always find problems. It hedges, it inflates, it reports phantoms, because an adversary that says “it’s clean” feels to itself like it isn’t doing its job. Loop that naively and you get what I’ve come to think of as the audit spiral. A phantom finding triggers a fix, the fix is a real change to a working system, the real change introduces real bugs, and the next audit finds those plus a fresh crop of phantoms. The loop’s gain goes above one, iteration diverges, and you can burn days making a system worse while every individual step looks like diligence.
Neither failure mode was caught by more machinery. Each was caught because a human, me, knew where the loop could still lie, and knew the difference between the wolf and the crying.
Somewhere in that stretch I also changed my mind about where the learning lives in all this. The model I work with is, inside a project, static. I’m not fine-tuning it, so its weights don’t change from anything we do together. A new version arrives every few months, and it is genuinely better, but better in general, at everything, and it has learned nothing in particular from what its predecessor got wrong here. What learns is the apparatus. The specs, the adversaries, the checks that have been proven against fakes, the ledger of every way the thing has ever fooled you. All of it accretes, and none of it evaporates when the model version changes. We tend to assume the intelligence is in the weights; in my experience the compounding asset, the thing that is unmistakably learning, is the scaffolding built around the weights. A commit hook can enforce that the scaffolding stays complete and consistent, that every cited source actually exists. It cannot make any of it true.
The fourth rung
This summer the research outgrew a single loop. The work split across many parallel specialist sessions, each with its own long-running context, a coordinator between them, my carefully built verification apparatus running throughout. And the whole thing entered a death spiral anyway. Not from any single failure. Each agent, examined alone, looked locally reasonable, but each drifted a little, the drifts compounded, and somewhere in there the system crossed a line where it was generating errors faster than its own checks could catch them. The agents’ own commit logs from those weeks make for sober reading. One records deleting a mechanism it had quietly invented and treated as real for days; another records discovering that a harness had been blind to most of its input all along. I had built the loop well, and I watched it come apart from complexity itself.
The diagnosis took me a while, because I kept treating it as a checking failure and adding more checks, which fed the spiral. The real problem was the one this essay has been circling. Out here on the research frontier there was no answer key at all, not even a self-made one. With many agents drifting in parallel, checking is meaningless, because checking against what? There was no longer one authoritative statement of what the system was supposed to be. Every session had its own slightly stale picture, and every picture was defensible.
The way I broke the spiral was to write the answer key myself. One document that is the design, present tense, no history, no status, no to-do lists, changing only with my express permission. I’ve come to think of it as a constitution. Everything else, the conformance checks, the coverage ledgers, the status views, is generated from it, derived, never hand-maintained. And the detail I keep coming back to is that the work orders now generate themselves. The constitution says what should be, the ledgers say what is, and the difference between them is the to-do list. Nobody writes it, and nobody can quietly redefine the goal by editing it.
Around that document, the loop grew into something I can only call an organization. It has a staff and a division of labor, standing rules and records, and it persists through complete turnover of its members, because every model upgrade replaces my entire staff overnight. What carries over, the constitution and the ledgers and the gates, is exactly where the learning lives. Sessions read their directives from inboxes instead of relying on me to ferry instructions between them, and work exists as ledger rows that must close to done-with-evidence or blocked-with-cause, because the coordinator reads the ledger, not the transcript. One small example of what running this is like. The agents had a habit of stopping mid-task to narrate their good intentions, and no instruction I wrote could talk them out of it; what fixed it, in a day, was changing the rules of the system so that continuing became the path of least resistance. You don’t repair agent behavior by asking harder, you repair it by changing what the mechanism makes natural.
The assumption this rung drops is a subtle one. A well-built loop is a machine; set it up right and it runs. That’s what I believed at the previous rung, and where the answer key is strong, it’s close enough to true. Out at the frontier it’s false. The loop at scale is not a machine but an organization, and setting one up is not the end of it. It has to be maintained. Constitutional care, boundary enforcement, periodic repair. And periodic reform, it turns out, because the organs silt up as work moves through them. An inbox accumulates a hundred notes describing a design that no longer exists, a plan carries scaffolding for phases already finished, and every few weeks I find myself making a genuinely structural call, archiving whole directories, flattening a plan, redrawing a boundary between sessions, the way any human organization has to be restructured at intervals. Even the constitution-and-ledgers arrangement has been reorganized more than once since I built it. The refresh isn’t a sign the design failed; it’s what maintaining means. You stop being the architect of a loop and become the maintainer of an organization, and the organization will enforce a wrong constitution perfectly. We win or fail on that one document now, which means the judgment that used to live in my hovering finally lives there. For now.
It took me a while to name what this new work reminded me of, and then it was obvious. Writing constitutions, designing incentive mechanisms, deciding what to delegate and what to escalate, keeping an organization honest about its own reporting. This is managerial work. I’ve taught strategy and I’ve run a company, and maintaining this organization is more recognizably that than it is engineering. The members of the organization just happen to be machines.
So what does all this buy? The organization runs autonomously for days at a stretch. Ledgers fill, defects get caught and fixed, real scientific work accumulates while I’m elsewhere, and that still astonishes me. Then error accumulation outruns self-correction, something clogs, and I go down into the engine room to find it, fix it, and relaunch. Days of quiet trust, then a descent. The measure of the whole method is the length of that interval. Hand-checking bought me minutes of peace, a good loop bought hours, and the organization buys days at the hardest edge of what I know how to specify. Days, and not more. What I do during the descents has changed, though. I don’t steer the work as much, because that road leads back to the trap. I audit and repair the apparatus itself, at intervals I choose, rather than diving in whenever a run stumbles, which would be hovering under a new name. The current models, even wrapped in all this machinery, are not good enough yet for unattended work at the frontier.
A question keeps coming back during those descents. If the learning lives in the accreting organization, what exactly is the thing only I still supply? And is it shrinking toward zero as the models improve, or is it just what a frontier is? That question needs more room than I have here, and I’ll return to it.
Where this leaves me
Everything above changes what I can attempt. It doesn’t make me right, and it doesn’t promise that the attempts will land. Even the attempting, as I said, comes in stretches. But the twenty-year problem is live again. The organization is chewing on it as I write, with real progress, real setbacks, quasi-working, climbing. A version of me from three years ago would not believe the attempt is even happening at all, and the person maintaining it is mostly just curious to see where it goes.
I’ll keep you posted.