LLM agents in production: a teardown of AEGIS, my open-source automation layer

Every morning, a system I built triages my inbox of interruptions: it classifies new tasks, investigates infrastructure alerts, reconciles finance data, and drafts research summaries. Four LLM agents run 28 workflows over my tasks, money, knowledge, and infrastructure. I don't watch any of it happen.

The only time it needs me is when a decision actually matters — and then it doesn't act. It asks. A card lands in an approval queue: approve / choose / edit / discard. I tap one button and the workflow continues.

That last paragraph is the whole thesis of this post. LLM automation works in production when it's boring: deterministic workflows, typed approval gates, audit logs, bounded costs. Everything interesting about AEGIS is in service of making the AI parts boring. I recently open-sourced it (github.com/hikmahtech/aegis, MIT), so this teardown comes with receipts — you can read every line I'm describing.

Why the first version failed

AEGIS is a rewrite. The first version — Jarvis — worked, and then became a 20,000-line monolith I dreaded touching. The failure mode wasn't the LLM; it was everything around the LLM. Agent behavior was hardcoded. Adding an integration meant surgery. There was no principled boundary between "the model suggested this" and "the system did this."

The rewrite inverted the priorities. Before asking what can the agents do, I asked what happens when they're wrong — and built that path first.

The architecture: boring on purpose

AEGIS is three independently deployable Python 3.12 services sharing one Postgres 16 database (with pgvector for embeddings):

• Core — FastAPI API plus an admin React SPA. Owns the REST surface, knowledge retrieval, and the connectors.
• Worker — a Temporal worker running all 28 workflows and their activities.
• Comms — a thin Slack adapter. If you don't configure Slack, it idles as a no-op and approval cards land in the web inbox instead.

The load-bearing choice is Temporal. LLM calls are non-deterministic; everything wrapping them shouldn't be. Workflows get retries, timers, child workflows, and durable state for free — so an agent pipeline that takes three days (because a human approval sat in the queue over a weekend) resumes exactly where it stopped. No cron-plus-state-table archaeology.

Model access goes through a LiteLLM proxy with three tiers — fast, balanced, smart — that resolve to whatever you've configured: Claude, OpenAI, OpenRouter, or fully local Ollama. Embeddings run on a local model at zero per-token cost. A completely local deployment has $0 marginal LLM cost. That tiering matters more than it looks: classification steps don't need the smart model, and routing them to a cheap tier is the difference between an automation layer you leave running and one you nervously switch off at month-end.

Human approval as a first-class primitive

Most agent frameworks treat "ask the human" as an afterthought. In AEGIS it's the central abstraction: every workflow-to-human handoff is a single interactions row in Postgres, with one of five kinds — approval, choice, input, draft review, acknowledge. A child workflow delivers the card and waits. The human's response comes back through one uniform callback, identical whether it was tapped in Slack or clicked on the web.

One primitive, used everywhere, means every risky action in the system has the same shape: proposed, delivered, decided, logged. There's no second code path where an agent "just does it."

The same discipline extends to infrastructure access. AEGIS has a registry of SSH hosts, Docker Swarm and Kubernetes clusters, and cloud accounts — and each entry carries a read-only gate enforced identically across the admin UI, the REST API, and the agents' chat tools. Credentials are encrypted at rest, materialized to locked-down temp files only at call time, and deleted immediately after. An agent can inspect a cluster all day; mutating it requires the gate to be explicitly opened.

Agent behavior is data, not code

The four agents — an executive assistant, a researcher, a finance agent, and an infrastructure agent — aren't classes with hardcoded abilities. Behavior keys off capability tags, and the routing knobs (intent keywords, mention aliases, tool sets) live in JSONB metadata, editable live from the admin UI. The 42 chat tools are gated per agent: the assistant gets 15, the researcher 11, finance 13, infrastructure 32.

This replaced an earlier hardcoded-agent design, and it's a lesson I now apply everywhere: the parts of an AI system you'll tune weekly should be configuration, not deployments. The same goes for schedules — a reconciler polls the schedule table every few minutes, so changing a cron from the admin UI reaches the running worker with zero redeploys.

The flagship: alert investigation with a coding agent in the loop

The pipeline that best shows the philosophy is AlertInvestigationFlow — twelve steps between "an alert fired" and "something happened about it":

1. Signature dedup (does this stack trace vary, or is it the same crash?)
2. Fingerprint dedup against past investigations
3. Mute check
4. Human gate #1: approve investigating at all
5. A verification delay and recheck — transient alerts die here
6. Resource resolution: which repo, which org, which account
7. Context gathering: runbooks and prior incidents from the knowledge layer
8. The investigation itself — the infrastructure agent can SSH into a registered host and run a coding agent (Claude Code) against the right repo
9. A cheap-model verdict classification of what came back
10. Human gate #2: open a PR, mute, acknowledge, or discard
11. Notification
12. Audit log

Notice what the LLM is not allowed to do: skip the dedup, skip the gates, or act on its own verdict. The model does the expensive cognitive work — reading logs, correlating code, proposing a fix — and the workflow does everything else. When people ask me what "AI automation with guardrails" means concretely, I point at this pipeline.

Observability closes the loop: OpenTelemetry traces span all three services, and every LLM call lands in a table with model, tokens, latency, and time-to-first-token. When something misbehaves, I query it like any other production system — because it is one.

Three production lessons

These are the bugs that taught me the most. All three are invisible in a demo and inevitable at production volume.

Reasoning models can silently return nothing. A reasoning model bills its hidden thinking against max_tokens before any visible output. Set the cap tight and it burns the budget thinking, then returns finish_reason: length with empty content — which crashes whatever tries to parse the JSON that isn't there. AEGIS now detects this and raises a typed truncation error, and structured-extraction calls get at least 3× the expected output size as headroom.

Agents will talk themselves into loops. Agents reply to task comments; agents also read task comments. Without a strict producer/consumer contract, every agent reply re-triggers its own classification — every fifteen minutes, forever. The fix is unglamorous: every system-authored note carries a distinctive prefix, and every consumer filters it out. The lesson generalizes: any agent that writes where it reads needs an explicit "did I write this?" protocol.

Retrying the unretryable poisons your queue. An external API returned per-command errors nested inside an HTTP 200 envelope. Treating any failure as transient burned five retries per call on permanently invalid requests. The fix was an error classifier: retryable failures re-enter the outbox, permanent ones fail fast and loudly. Every external integration in the system now makes that distinction explicitly.

What it costs to run

The whole thing runs on hardware I already own, on a self-hosted Docker Swarm. Embeddings are local and free. The LLM tiers can point entirely at local models — $0 marginal cost — or at hosted APIs where the smart tier earns its price. The expensive steps (like verdict classification) deliberately run on the cheap tier. AI automation that only pencils out with unlimited API budget isn't an automation strategy; it's a subscription with extra steps.

It's open source — including the boring parts

AEGIS v0.1.0 shipped on 2026-07-10 under MIT. The whole application is public: all three services, migrations, seed config, and the design docs. What's deliberately not in the repo is my deployment pipeline — CI runs tests only, so a fork's GitHub Actions can never touch infrastructure it doesn't own.

Open-sourcing it honestly took real work, and the commit history shows it: hardcoded currency assumptions made configurable, my specific cluster names and contexts pulled out into the registry. That gap — between "works for me" and "works for anyone" — is the same gap between a demo and a product. I'd rather show the work than claim the polish.

If you want LLM automation in your operations

Strip away the personal-assistant novelty and AEGIS is a template for the systems I build for clients: LLM agents doing real work inside deterministic workflows, with human approval where it counts, audit trails everywhere, and costs bounded by design. The same shape fits invoice triage, document pipelines, alert response, and ops copilots.

If you're trying to get AI automation past the demo stage — safely, and without an open-ended API bill — let's talk. And if your data stack is the thing that hurts, my free book Local-First Analytics is where I'd start.