When the Bot Learns to Stay Quiet

Every monitoring setup I've ever run has followed the same arc. Week one: alerts on everything, because everything matters. Week four: alerts on most things, because some of those turned out to be noise. Week twelve: a muted channel with four hundred unread notifications and a vague sense that if something truly broke, I'd probably find out some other way.

A watchdog that cries wolf gets muted. And a muted watchdog is worse than no watchdog at all, because it keeps converting real signal into background noise while letting you believe you're covered.

So when I built the alerting side of AEGIS, I designed backwards from that failure. The goal is not to tell me about problems faster. The goal is to not tell me about most of them, ever — and to be trustworthy enough about it that when a card does arrive, I actually read it.

What happens between "alert fired" and "human paged"

The infra beat belongs to Pandora's Actor — the agent with the theatrical name and the SSH keys. The homelab it watches is a small Docker Swarm, three nodes in my house, reached over SSH through HomelabConnector. Nothing exotic: services drift, disks fill, containers crash, GitHub sends webhooks.

When something fires, it does not come to me. It goes into AlertInvestigationFlow, a Temporal workflow that stands between every alert source and my attention. The pipeline, roughly:

• Skip if resolved on arrival. Some alerts are already stale by the time they land. Those die immediately.
• Deduplicate. Have we investigated this exact thing recently? If so, don't do it again.
• Wait, then re-check. A verification delay, after which the flow asks the source whether the condition still holds. A remarkable fraction of alerts self-resolve — a container restart, a transient network blip, a health check that caught a deploy mid-flight. Those die quietly too.
• Ask an LLM for a verdict. Given the evidence gathered, is this actually a problem? Is it actionable, is it noise, did it already fix itself?
• Only then, a human gate. If the verdict says this is real and something should be done, I get an interactions card — a row in Postgres, a card in chat, a workflow waiting on my tap.

Because the whole thing is a Temporal workflow, the "wait fifteen minutes and look again" step is a durable timer, not a thread holding its breath. The worker can restart, the stack can redeploy, and the investigation resumes exactly where it left off. That durability is most of the reason this is a workflow and not a script.

The net effect is that the majority of alerts are born, investigated, and buried without me ever knowing they existed. Which is the point.

The three parts that are actually hard

None of this is conceptually difficult. All of it is difficult to calibrate.

The verification delay is a bet, and I keep losing small amounts on it. Too short, and you investigate blips — false positives with extra steps. Too long, and something real burns for twenty minutes before anyone looks. There is no correct number; there's a per-source number, and the flow asks for the delay per alert rather than hardcoding one. I've adjusted these more times than I'd like to admit, and I still don't fully trust them. The honest position is that the delay encodes a guess about each source's flakiness, and sources change.

Deduplication has to survive flapping. A service in a crash loop can fire alerts far faster than investigations complete. The first cut was fingerprint-exact dedup — don't re-investigate the same alert. Not enough: a flapping service produces near-identical alerts with slightly different fingerprints, and every variation slipped through the exact match. So there's a second, coarser layer that recognises the signature of an alert and attaches the new firing to the already-open task instead of spawning a fresh investigation. Ten parallel investigations of one crash loop isn't diligence. It's noise with a token bill.

The LLM verdict can be confidently wrong. Most of the time the model reads the gathered evidence and produces a reasonable call. But "most of the time" is carrying a lot of weight in a system whose whole promise is that silence means safety. A model that declares a real problem not_actionable fails silently, which is the one failure mode this design cannot afford. That is exactly why the human gate stays: the LLM decides what's worth my attention, but anything that would take a real action against the homelab still comes to me as a card. The verdict is a filter, not an authority.

The judgement call: deleting ambition

Today I rewrote the GitHub half of this, and the rewrite is a small confession.

GitHubAlertFlow started life as an issue-investigator. Webhook fires, agent picks up the issue, investigates, prepares analysis before I've even seen it. It sounded like the obviously ambitious version. In practice it was the wrong shape: it fired across every repository, investigated things I didn't want investigated, and produced output I skimmed at best. So it got muted.

Which — given everything above — is the failure mode. I had built a cry-wolf watchdog inside the system designed to prevent them.

The version I shipped today is duller and better: a pull-request notifier. It's scoped to the repositories I actually track in my workspace, it fires only when a PR is opened, reopened, or marked ready for review, and it deliberately ignores the synchronize event — that one fires on every push, and most pushes to PRs in my orbit are my own. The flow that used to investigate now just tells Pandora to mention that a PR wants eyes.

A notification I read beats an investigation I muted.

I resisted this for a while, because "the AI investigates your issues" demos better than "the bot tells you about PRs." But AEGIS isn't a demo, and the register that matters is whether I trust the channel. The clever version had already lost that trust. The boring version is earning it back.

Where this goes

Three directions, none of them glamorous. More alert sources, because the pipeline earns its keep in proportion to what flows through it. Smarter dedup, because the signature layer is still fairly blunt and I'd like it to understand recurrence over time, not just concurrent duplicates. And tighter verdict prompts, because every inconclusive verdict is an investigation that spent its effort and then shrugged.

The uncomfortable truth about this whole subsystem is that its success is invisible by construction. When it works, nothing happens. I don't get a card, I don't check a dashboard, and the only evidence is a workflow history showing that something fired at 3am, was watched, self-resolved, and was let go.

I've decided I'm fine with that. The code isn't public yet, but the pattern travels: put a patient, durable pipeline between your alerts and your attention, make silence the default, and reserve the interruption for things that survive scrutiny. More about the system lives at /aegis.