What is human-in-the-loop AI?

Human-in-the-loop AI is a system where a person decides at the points where a mistake would cost something, and the model does the rest. In practice the useful place to put the person is the write boundary: the moment a system is about to change something outside itself, such as posting a journal entry, restarting a service, sending a message or opening a pull request. Reads, drafts and investigations run on their own; the person is asked a few times a day, not a few hundred, and only about actions.

How do you implement human-in-the-loop approvals for AI agents?

Make every outside write a typed action with a state machine (proposed, approved, running, done or failed, plus declined, expired and refused), not a free-form command. The model may only propose. A card asks one question with the reasons and the exact thing that will run. The workflow that raised the card waits durably, so the answer can come hours later and survives a restart. The approved action executes exactly once, verifies its own result, and is never retried on its own after a failure that may have reached the target.

What does human-in-the-loop mean for agentic AI?

For an agent that plans and acts in many steps, human-in-the-loop does not mean approving each step. It means approving the step that leaves the system, and letting the agent investigate, plan and draft freely before it. An alert agent can read logs and metrics, group duplicates and write up a verdict with no approval at all; the restart it proposes is the one thing that needs a yes. Put the gate on the consequence, not on the reasoning.

How does human-in-the-loop improve AI accuracy over time?

Only if the answer is kept. The pattern that compounds is one question per card whose answer becomes a rule: a bookkeeping system asks which account a new payee belongs to once, writes the answer as a rule and re-files every past entry it matches, and never asks that question again. A system that asks the same question every week is not learning; it is delegating its work back to you.

Does human-in-the-loop slow AI automation down?

It slows the write, not the work. Investigation, matching and drafting run unattended; the card waits for a tap. What actually slows a system down is approval fatigue: cards that ask for a decision the person cannot make from the card. In my own systems I skipped 6 of 8 meeting-commitment cards because each asked whether to add lines to a task list with no idea where they would go. A card has to carry a plan with a destination and one button, or it will be skipped.

What should an AI agent never do without approval?

Anything it cannot undo on its own and anything with money, people or production in it: posting to a ledger, sending mail or a social post, restarting or scaling a service, merging code, deleting or archiving in bulk, and acting on a protected resource at all. The exception worth making is one narrow, verified, reversible action with a dry-run record behind it, such as a single careful restart of a service that has a known healthy state to return to.

Human-in-the-loop AI: the approval patterns I run in production

For the last month I have run my own business on eight self-hosted AI systems. Bookkeeping, infrastructure alerts, mail and calendar, research, publishing, analytics, code, and a desk that holds them together. Every write that leaves any of them waits for my answer on a card. This post is what that looks like after a month of answering: the six patterns that held, the cards I stopped answering, and the mistake that changed the rules more than any design did.

"Human in the loop" usually means a review step bolted onto a model. What I run is narrower and, I think, more useful: a person decides at the write boundary, and nowhere else. The systems are AEGIS, the family I rebuilt in October 2026. The numbers are from their status and learnings documents on the day of writing.

Pattern 1: approve the write, not the thought

The first rule decides everything after it: a card is for an action that leaves the system. Not for a plan, not for a verdict, not for a draft.

The infrastructure system takes every alert, groups duplicates into one problem, reads logs, metrics and the service's own state, and writes up what it thinks is wrong. None of that needs me. The restart it proposes does. The bookkeeping system matches hundreds of statement lines to the ledger on its own; the one payee it has never seen becomes a card. The mail assistant reads, classifies and summarises an inbox unattended; archiving is the write, so archiving is what has rules and switches.

Put the gate on the consequence, not on the reasoning, and the number of cards drops to something a person will actually answer. Across the eight systems I see a handful of cards a day. The one time a system wanted a yes for every step, I stopped reading them, which is the subject of the section on skipped cards below.

Pattern 2: one question, the reasons, and the exact command

A card asks one question. It carries why the system wants to do this, what it will run, and what it will not do.

For the infrastructure system "what it will run" is literal. An action is a typed object, not a string a model wrote. It validates its inputs and renders the exact command line. Before it runs, it renders again and compares; if the two differ, it does not run. The card I tap shows the rendered command, so the yes is a yes to that command and nothing else.

Cards are one per subject, refreshed in place while the problem is open, and resolved after a grace period when it closes. Acknowledging a card silences it until the subject changes, and a flapping problem does not raise a new card on every flap. All of that is a pure function with a test, not a prompt.

Pattern 3: the answer becomes a rule

This is the pattern that compounds, and the one most "human in the loop" products miss.

When the bookkeeping system sees a payee it does not know, it asks one question: which account is this. The answer does not go into a chat log. It is written as a rule into a plain-text rules file, and in the same commit every past entry that the rule matches is re-filed. Next week the same payee produces no card. The system asks less every week by construction, and the rules file is readable by anyone who inherits the books.

A question becomes a rule. Bookkeeping asks about a payee once. The answer becomes a rule and re-files every past entry it matches.

The mail assistant does the same with meetings. A recurring meeting it has not seen asks once where its outcomes belong; the answer becomes the routing rule for the series. A system that asks the same question every time is not learning. It is handing its work back to you one card at a time.

Pattern 4: typed actions with a state machine

Every outside write is a row with a state: proposed, approved, running, then done or failed, with declined, expired and refused as the other exits. A model can only create the first state. The owner moves it to the second. The system moves it through the rest, exactly once, then verifies what it did and records the result.

The rule that matters most sits in the failure branch: a failure that may have reached the target is never retried on its own. A restart that timed out might have restarted the service. A journal push that lost its connection might have landed. Retrying either turns one unclear outcome into two. The row stays failed, a person looks, and stuck rows are reaped on a schedule rather than retried in a loop.

This shape was built once, for the infrastructure system, and copied to anything that writes outside itself: journal pushes, social posts, pull requests. It is the part of the kit I would keep if I had to throw everything else away.

Pattern 5: dry run before the switch

Nothing that writes is switched on in one step. It runs first in a mode where it records what it would have done, and the record is read before the switch is thrown.

Automatic restart of a failing service is the only action the infrastructure system may take without a card. Before it was allowed to, it ran for days writing "would have restarted" into the log. Those dry runs found the rule it had to learn: restarting a service that is crash-looping does not help, it only adds a failed task to the pile. So the rule now refuses crash loops, and that refusal was earned on the dry run, not in production.

The bookkeeping system took the same path at a larger scale. It wrote to a shadow copy of the ledger until a diff against the old system had no unexplained line, across all 15 statements. Only then was the old system switched off, and only after that did the new one write to the real journal. There was never a day with two writers.

The mail assistant has a smaller version: every archive rule can be dry-run on its own, even under a lane that is already live. A new rule under a live lane would otherwise go live on deploy, which is the kind of switch nobody meant to throw.

Pattern 6: the system may say no to me

An approval gate is usually drawn as the human stopping the machine. In practice it runs both ways, and the refusals are where the design proves itself.

The agent said no. Four times the systems refuse, and why that is the whole point.

The infrastructure system refuses to restart a protected database, however I ask. A node drain is refused while it would put the cluster's manager quorum at risk or strand a service pinned to that node. The publishing system, in the demo above, refuses to send a post whose text changed after I approved it: the yes was for those words, and different words need a new yes. The investigation agent refuses to give a verdict if it made no tool calls, because before that rule it invented replica counts.

A yes from me does not override a refusal. The refusal is the system's half of the contract, and it is what lets me answer cards quickly on the other half.

The cards I skipped

Out of eight cards about commitments made in meetings, I skipped six.

The mail assistant reads meeting notes and lifts out the things people said they would do. Its first version turned each meeting into a card that asked, in effect, whether to add those lines to my task list. I skipped most of them, not because the lines were wrong but because the card gave me nothing to decide with. Where would the tasks go? Were some of them already tracked somewhere else, as a ticket or a pull request in the client's own tools? Most were. A card that asks for a decision the person cannot make from the card is a notification with a button on it, and it will be treated like one.

The fix changed the card's content, not its shape. Each commitment now arrives with one proposed outcome: track it in a named project of mine, already tracked elsewhere with a link, hand it to the code system, draft a ticket for me to post myself, or just note it in the day log. The card has one button, Do this, and per-line edits for the rest. A meeting never starts work by itself, and a task closes only on an event, never on silence.

Approval fatigue is not caused by too many cards. It is caused by cards that cannot be answered.

The mistake that wrote the rules

The best argument for all of the above is the one time a system got a yes and still did harm.

On its first sweep of old mail, the assistant looked only at senders its rules already knew. It archived 2,466 mails, kept 1,290 of them back, declared the inbox swept and switched itself off. Thousands of mails that actually needed sorting were never selected at all. I noticed four days later.

The assistant’s first mistake. The inbox sweep got it wrong the first time. What changed, and why every switch now runs dry first.

Three things changed, and none of them was the prompt. The sweep now selects its input by what I see, inbox mail older than a number of days, not by what the rules already know. Categories from a model or a cache are candidates, not decisions: a second, cheaper check on sender, subject and summary has the last word, and a keep list (starred mail, mail marked important, senders with important mail on record) runs before anything is archived. And every archived mail carries a label, so the whole sweep can be undone with one search.

The lesson I took is uncomfortable for anyone selling approval as a feature. My yes to "sweep the inbox" was real and it did not help, because the yes was to a batch, not to the decisions inside it. For bulk actions the dry run, the second gate and the undo matter more than the approval. The card is still there, but it is the least important of the four.

What this costs, and who should not do it

The cost is a few taps a day from one person, and a design discipline that every write path goes through one typed, stateful, dry-runnable door. The model spend is bounded the same way: every call checks a daily budget before it runs, and the infrastructure system runs under a cap of $20 a month.

Who should not build this: a team that needs roles and sign-off chains out of the box, a process so standard that a vendor's approval flow already fits, or anyone with nobody to answer the cards. I am one person answering mine. If the person who would answer yours does not exist yet, build that first.

If you are building one

Six checks, in the order I would apply them to a system that is about to go live:

Does the person approve writes only, with reads and drafts running unattended?
Does each card carry one question, the reasons, and the exact thing that will run?
Does the answer become a rule, so the same question is never asked twice?
Is every outside write a typed action with a state machine, executed exactly once and never retried after an unclear failure?
Did it run in a dry-run mode first, and did someone read the record before the switch?
Can the system refuse you, and can a bulk action be undone with one command?

The AI workflow teardown asks these as part of a longer checklist you can run against your own stack, free and without an email. If your workflow does not fit the tools you have tried, AEGIS is what building around the workflow looks like, and the AI automation engineer page says what an engagement covers.