Why is alert deduplication so hard?

Because most systems dedupe on the notification rather than on the problem. If the ticket, page or chat message is what identifies an incident, then "is this the same thing as last time?" cannot be answered until a ticket already exists — and every producer of alerts invents its own key for doing so. The fix is to give the problem its own record, with one correlation key, before any notification is sent. The ticket then becomes a projection of that record, and closing or deleting it changes nothing about what the system believes.

Should a self-healing alert create a ticket?

No, and the cheapest way to enforce that is a per-class settle window: a problem younger than the window stays visible in the dashboard and the chat channel but earns no task, and one that resolves inside the window never earns one at all. It has to be a clock rather than an occurrence count, because a five-minute flap fires five times and still is not worth a chore. Measured in my own system: of 60 problems that earned a task, 15 were over inside fifteen minutes and 8 inside five.

Why did my health check stay green during an outage?

Almost certainly because the check runs inside the thing it is checking. A container healthcheck executed inside that container proves the process is alive; it proves nothing about the proxy, tunnel or load balancer in front of it. If the way in is broken, the application is healthy and unreachable at the same time, and an inbound monitor cannot tell you — being told is exactly what is broken. The fix is an outbound canary: from inside your own system, fetch your own public URL every cycle and alert when it stops answering.

Can an LLM safely group related alerts into one incident?

Let the model answer only the question it is good at — "are these the same condition?" — and put every safety rule in code around it. In AEGIS a grouping candidate must already be three or more live problems of one class and one subject kind; the model's yes is required to fold them; and code refuses to group across classes, to group hand-written tasks, or to group on a count alone. A "no" verdict is cached for a day, or until the cluster grows, so most ticks cost nothing at all.

Your Alerts Have No Identity

I went looking for why my homelab kept telling me the same thing eight times.

The answer was structural, and it is the kind of thing that hides in a system for years because every individual piece of it is reasonable. AEGIS — my self-hosted agent platform — had no concept of a problem. It had notifications. One probe failed on eight consecutive days and wrote eight Todoist tasks, because the only thing it could ask was "did I already write a task about this?" — and it asked by looking at its own most recent log line, which by then said something else.

The Todoist task was the identity of the incident. And that is the bug.

This is the first of three posts about three lanes of AEGIS I rebuilt between 5 and 13 September. The other two — the money lane and the knowledge lane — landed on the same rule from completely different directions, which is the only reason I trust it.

What the measurement said

I don't rebuild anything on a hunch any more. Before touching the code I measured what was actually there on 2026-09-07:

Three alerting systems that could not see each other. The investigation flow (a #alert task plus a chat card), a set of watchdogs deduping against the audit log, and domain tables with their own private keys for certificate expiry and config drift. The resolved-aware NOT EXISTS dedupe SQL was copy-pasted in three files.
Thirteen fingerprint or dedupe-key schemes, mutually incompatible. A vendor fingerprint. A synthesised alertmanager:{alertname}:{instance}. A sentry:{issue_id}. Three separate signature classes. Three mute-key namespaces sharing one primary key. A day-bucketed drift key. Plus three informal links — a LIKE '%' || task_id || '%' against workflow ids, title substring matching, and a comment footer used as an authorship marker.
Dedupe keyed on the task, not the problem. The dedupe index's task_id column was NOT NULL and joined to the task table. One flow refused to use it at all and built a fourth ledger inside a settings row. Because the signature was the primary key, recreating a task reset first_seen_at and the occurrence count — recurrence history was destroyed on every recreate. Only 12 of 42 open alert tasks had a signature row at all.
One producer bypassed the capture path entirely, hand-building its command and deduping against the newest audit row. The close-on-resolve function structurally could not reach it. That is where the eight copies came from.
No service state. No maintenance window, no deploy suppression. The GitHub webhook claimed deployment events and dropped them. Suppression existed only as incidental delays scattered across three files.

And the part that made it expensive to fix: of 72,499 source lines, the four files carrying this logic held 11,212 of them. The code doing all this was the code hardest to change safely.

The primitive: a problem is a row, a task is a projection

The replacement is deliberately small. Every operational signal — an Alertmanager webhook, a Sentry issue, a swarm heartbeat, a certificate about to expire, a stuck social post, a hand-written task — becomes an Event and goes through one function, ingest_event.

That function owns three things and nothing else:

Identity. One open problems row per correlation_key. If a row for that key is already open, this is another occurrence of the same problem, not a new one.
History. Every occurrence, every resolution, every report is a problem_events row, idempotent on (source, external_id). Recurrence survives everything, including deleting the ticket.
The decision. The return value carries investigate: true or false. The producer starts the investigation workflow only when told to. The flow itself never dedupes and never captures a task.

The Todoist task is created by a projector, from the problem. It is a projection, never the identity. The rule I wrote into the docs, because I knew I'd be tempted to break it: do not look a problem up by its task title or fingerprint.

Three things I learned after it shipped

The design above was the easy half. The interesting part is what production did to it once real alerts started arriving.

A blip earns no task

The hub worked. It also created a lot of tasks. Of the 60 problems that earned one, 15 were over inside fifteen minutes and 8 inside five — created, auto-clarified, and auto-completed with no human ever acting on them. Perfectly deduplicated noise is still noise.

The fix is a per-class settle window. A problem younger than its class's window stays in the hub, the digest and the chat channel — it is fully visible — but it earns no Todoist task. One that resolves inside the window never earns a task at all.

Two details matter more than the feature:

It is a clock, not a count. The obvious implementation is "three occurrences before it counts." That is wrong, because a five-minute flap occurs five times. Only elapsed time since first occurrence distinguishes a blip from a condition.

Only the task waits, and only for two producers. The investigation still starts on the first occurrence, so the diagnosis and the chat card are exactly as fast as before — what's deferred is the chore, not the work. And the window applies only to the two sources that re-check on a scale of seconds: the monitoring stack and the swarm heartbeat. Everything else — a money reconciliation, a stale feed, an agent's own question — is projected on sight, because it comes from a sweep that runs every half hour or slower and has already judged the thing worth reporting. A wait of minutes cannot observe a blip there; it can only delay the chore.

The honest caveat, which is in the docs because I will forget it: that per-class number has two jobs. It is also the verification delay before an investigation spends model tokens and takes its one automatic restart. Setting it to zero everywhere buys back the instant task at the price of every verification delay, so a blip that would have self-healed during the wait now costs a billed investigation and a forced restart. That is not a return to the old behaviour. It is a different, worse behaviour that looks like the old one.

The same failure on many things is one problem

Six social posts wedged in the same queue used to be six problems and six tasks. Each one was correctly deduplicated. The dedupe just wasn't the thing that was wrong.

Every five minutes a sweep looks for three or more live, ungrouped problems sharing a class and a subject kind. If it finds a cluster it spends exactly one model call asking whether they're one condition. Only a yes folds them into a group; from then on the next stuck post joins the group instead of opening a seventh task.

The first run in production — by which time five of the six were still live — is the shape to expect:

stuck_post:post                grouped=true   n=5
  "All posts are stuck in the same queue, indicating a single queue
   drainage issue rather than individual post failures."

swarmoverlayblackhole:service  grouped=false  n=3
  "Different hosts, different overlay networks, and different endpoint
   counts suggest independent network partitioning issues requiring
   separate investigation."

Both judgements together cost $0.0009. The second one is the reason I trust the first: the model declined to group three superficially identical alerts because the evidence said they were independent.

But the judge is not the safety mechanism. Four rules live in code, where a model cannot reason its way past them: never group across classes; never group hand-written tasks (their problems carry a person's sessions and pull request links, and folding two would move one task's history onto another); never group a money finding; never group on the count alone. A "no" is cached for a day, or until the cluster grows, so the sweep does not re-price the same question every tick — most ticks make no model call at all.

That's the pattern I'd reuse anywhere: let the model answer the judgement question, and let code own every invariant.

The monitor that cannot see the thing that broke

On 2026-09-11 the proxy in front of AEGIS stopped passing requests. Every inbound webhook was dropped for three and a half hours. Nothing alerted.

Nothing could. The container healthcheck runs inside the container, so it proved the API process was alive — which it was. The alerting stack could not tell AEGIS, because being told was precisely the capability that had broken. This is the failure mode every inbound monitor shares and almost nobody tests: you cannot be informed that you are uninformable.

The fix is an outbound canary on the heartbeat that already runs every two minutes: from inside the worker, fetch AEGIS's own public URL. Three details make it work rather than merely exist:

Reachable means any answer under 500 from the host you asked. A redirect that lands somewhere else fails. An identity proxy would otherwise send the probe to its own login page, which answers 200 forever whether or not the origin is alive.
Aim it at a path the proxy will not challenge, and pin a status code the proxy cannot invent. Mine is a bare /api/webhooks/ping that answers 204. Pinning 204 also catches a proxy serving its own 404 for a route it has quietly lost. The obvious choice — a POST-only webhook path, expecting 405 on a GET — does not work here, because the admin SPA's catch-all claims every unmatched /api/ GET before the framework can say 405. It answers 404, which is exactly what a routeless proxy says, so it asserts nothing.
Two consecutive failures, not one. A single dropped request is what a rolling update of your own service looks like, and this alert escalates.

The rule underneath all of it

Every fix above is the same sentence in a different costume:

The record of what is true and the notification about it are different objects, and the notification must never be allowed to define the record.

A ticket deleted by a human is not evidence that a node came back. An occurrence counter that resets when a task is recreated is not evidence that a problem is new. A task completed by someone tidying their inbox is not evidence that a bill was paid — that one is the money post.

Once the problem has its own row, all three of those become trivially correct, and most of the machinery you built to compensate for their absence can be deleted. The hub let me drop two dedupe tables, delete the copy-pasted dedupe SQL from three files, and remove an entire ad-hoc ledger that had been living inside a settings row.

That's the actual return on this kind of work. Not the feature. The deletions.

AEGIS is open source and MIT-licensed — the hub is in services/hub.py, and the design document with all the measurements above is in the repo: github.com/hikmahtech/aegis. If you want the wider tour, it's at /aegis.

The other two posts in this series: a ledger is not a database table and a pile of documents is not knowledge.