AI copilots produce things fast. Dashboards get built in minutes. Pull requests merge cleanly. Everything looks done.
That speed is the trap. As one write-up on the phenomenon put it, teams are “getting faster at producing things that don’t always solve the problem” – dashboards that look great but don’t answer the right question (LinkedIn). Polish and correctness are not the same thing, and autopilot optimizes for the one you can see on a screen.
We run an AI-operated pipeline at Glad Labs – agents write code, review each other’s work, merge PRs, and post content on a schedule. We’ve watched this trap open up in our own system more than once. This post is about what it looks like from the inside, and the specific guardrails we’ve built to keep from falling into it.
Complacency Is a Mechanism, Not a Mood
There’s a useful parallel from construction safety. When someone has performed a task a thousand times without incident, their brain shifts into what one safety trainer calls “low-power mode” – it stops treating the task as high-risk. The danger isn’t that the person got careless in some abstract sense. It’s that repeated success without a visible failure retrains the brain to stop checking.
The same mechanism runs in agentic engineering pipelines. Merge #400 looks exactly like merge #40. CI is green both times. The diff resolves the ticket both times. Nothing in the interface tells you that merge #400 quietly broke an edge case merge #40 never touched. You stop reading diffs not because you decided to, but because a hundred clean merges taught your attention that this step doesn’t need it anymore.
That’s the version of the autopilot trap that matters for teams running LLM agents: it isn’t laziness, it’s calibration drift. The system trains you to disengage exactly where engagement still matters.
Where We’ve Hit This Directly
Our own agent stack – internally we call it Poindexter – runs scheduled sessions, isolated git worktrees per task, and a persistence layer (a “brain daemon”) that’s supposed to remember context across calls and container restarts. For a while, it did. Then a container restart hit an ImportError that crashed persistence silently. Nothing alerted. The room just quietly forgot everything and kept operating as if it hadn’t. We fixed it and added a resume-recovery mechanism (PR #1085), but the failure mode is the point: the system had been running fine for long enough that nobody was watching for exactly this.
That’s why we don’t let a single agent write code and merge it. Every task goes through a two-stage review – one pass for spec compliance, a separate pass for code quality – dispatched as fresh subagents rather than the same context that wrote the code in the first place. A model reviewing its own output inherits its own blind spots; a second, differently-primed reviewer doesn’t.
We also gate autonomy by edit distance. An agent making a small, contained diff gets more latitude to merge without a human in the loop. An agent proposing a large, sprawling diff gets stopped for review regardless of how green the tests are. Green CI tells you the code runs. It says nothing about whether the code does the right thing, and treating “tests pass” as “task is correctly done” is exactly the trap the LinkedIn piece describes at the org level – polished output standing in for solved problems.
The Same Trap, Three Names
We’ve written about adjacent versions of this before. The Fine-Tuning Trap is the modeling version: tweaking parameters and calling it done, without the ongoing evaluation that catches quality drift after the fact. The GitFlow Trap is the process version: a workflow that felt disciplined at five contributors becomes autopilot ceremony at fifty, because nobody re-examined whether the rules still matched the risk. Autonomy gates and two-stage review are our answer to the same failure mode showing up in agent orchestration: don’t let “this has worked before” substitute for “this is being checked now.”
There’s a leadership-side version of the same idea worth naming directly, because it explains why the trap is so easy to miss from inside a team. One analysis of professional autopilot describes it as a side effect of being good at your job – plateaued growth, a calendar full of escalations, “the most capable leaders” feeling most disconnected from the actual work, while every quarterly metric says they’re succeeding. Swap “leader” for “engineer running an agent pipeline” and the description holds. If your dashboards are green and your merges are landing and you can’t remember the last time you actually read a diff line by line, that’s not efficiency. That’s the trap wearing efficiency’s clothes.
What We Actually Check
The practices that keep us out of this are unglamorous. Isolated worktrees per task, so one agent’s mistake can’t silently bleed into another’s context. Fresh subagents for review instead of self-review. Autonomy tied to edit distance, not to a track record of past success. And a standing rule we apply to any new piece of automation: it earns its place only if it reduces future human ops load – if it’s just automation for automation’s sake, with no payoff in reduced future work, we trim it rather than let it accumulate as one more system nobody’s watching closely.
That last rule matters more than it sounds. It’s tempting to keep adding autopilot because each individual addition looks like a productivity win, the same way each individual AI-generated dashboard looks like a productivity win. The indie hacker money post makes a related point about sustainable income – it’s a byproduct of building something real, not a target you automate your way toward directly. Autopilot is the same shape of mistake: it’s a byproduct of a system working well, not something you should be optimizing for on its own.
The trap isn’t the agent. It’s forgetting that “it’s been running fine” was never evidence that it’s still correct – only evidence that nothing has forced you to look yet.
