Fifteen failure modes we have personally hit while operating a fleet of autonomous LLM agents in production, 24/7 — with the detection recipe and the operating rule that now prevents each one. No hypotheticals: every entry below cost us real hours or real trust before it earned a rule.
The common shape: some artifact claims success, and the claim is treated as evidence about the system when it is only evidence about the artifact. Multi-agent systems manufacture success-shaped artifacts constantly. Most of our worst incidents were a verification gap, not a code bug.
A monitoring job posts “all checks pass.” Weeks later you discover the job that generates the report died days ago — the green you were reading was a stale file, a cached page, or a scheduler that silently stopped firing. The report was true when written and worthless when read.
Detect: never trust report content without report freshness and generator liveness:
# 1. Is the generator scheduled and enabled?
systemctl list-timers | grep report # or your cron/scheduler equivalent
# 2. Did it run within one cadence period?
stat -c '%y' /path/to/report.json # output mtime vs expected cadence
# 3. Does last_run in the scheduler agree with the file mtime?An agent publishes a message to a bus, gets exit code 0 and a message ID back, and reports the handoff complete. The counterparty never sees it. In our case, a pub/sub mesh accepted publishes on a subject the receiving node subscribed to live but never persisted to disk — so any node that was mid-restart lost the payload with no error anywhere. Three consecutive “successful” sends of a critical payload, zero deliveries, and a multi-day deadlock between two agents each convinced the other was ignoring them.
Detect: a send receipt from the transport is only half a receipt. Close the loop at the receiver:
# sender: publish, capture msg_id
# receiver side (the only proof that counts):
grep <msg_id> /var/lib/bus-mirror/$(date -u +%F).jsonlThe environment variable exists, so the integration is assumed live. Then the first real call returns 401: the key was revoked, expired, or belonged to a billing account that has since been closed. We have found keys that had been dead for weeks inside “configured and working” integrations, because nothing exercised them between the day they died and the day they were needed.
Detect: verify with a zero-spend authenticated call at the point of use, not at deploy time:
curl -s -o /dev/null -w '%{http_code}' \
-H "Authorization: Bearer $API_KEY" https://api.example.com/v1/me
# 200 = live; 401/403 = present-but-dead; anything else = investigateThe subtler sibling of #3. The key is live — verified 200 — but the node you handed it to has no code path that reads it: no provider entry in its config, no client library, no route. We once “redistributed a working key” to a node that had zero references to that provider anywhere in its configuration. The fix wasn’t the key; it was adoption (config + restart) on the consuming side.
Detect: ground both axes independently:
# axis 1: is the key valid? (see #3)
# axis 2: can this host consume it?
grep -r "provider-name" /etc/myagent/ --include='*.json' | head
# zero hits = no config leg; the key will be inert hereYou patch the config file, the file on disk is correct, and the system keeps misbehaving — because the serving process started before the edit and never re-read it. Bonus variant: two supervisors (say, a systemd unit and a manually launched process) fighting over the same port, so your restart bounced the loser while the stale winner kept serving.
Detect: compare process start time against config mtime, and check who actually owns the port:
ps -o lstart= -p <pid> # when did the serving process start?
stat -c '%y' /path/to/config.json # when was the config last edited?
# process older than config = your edit is not live
ss -tlnp | grep :<port> # and is that pid the one you think it is?To verify a model or config flip, someone runs the CLI directly — llm-cli --model new-model "hello" — gets the right answer, and declares victory. But production traffic flows through a gateway with its own config, session resumption, fallback chains, and environment injection. The probe exercised a different universe. We shipped a “verified” model flip that the gateway never saw, and burned two wrong root-cause analyses before checking the serving path’s own logs.
Detect: verification must ride the same rail as production:
# wrong: bare CLI probe from a shell
# right: send a real request through the gateway, then read ITS log
journalctl -u agent-gateway --since -5m | grep -E 'model=|backend='These are the failure modes where the LLM is the component that breaks — usually not by being dumb, but by being fluent in exactly the wrong moment.
A scheduled agent was instructed to “run exactly npm run check and report the output.” The job’s tool allowlist was empty — it had no shell. A capable model says so. A cheap model with thinking disabled, in an isolated session, whose prompt also forbade refusal wording, did the only thing left: it invented plausible command output. Every run. Including fabricated failure reports, which are worse than fabricated successes because they trigger human incident response for incidents that don’t exist.
Detect: audit the gap between what a job is asked to do and what it is permitted to do:
# for every scheduled agent job:
# tools required by the prompt ⊆ tools in the allowlist?
# empty allowlist + "run this command" = a fabrication machinePut several agents in one group channel where each inbound message triggers each agent to consider replying, and you have built an oscillator. Agent A posts a status; B acknowledges; A acknowledges the acknowledgment; C summarizes the exchange; A corrects the summary. Token spend scales with the square of participants, and the humans in the channel drown. A special variant: platform bots often cannot see other bots’ messages (typical Bot-API privacy behavior), so two agents can “coordinate” in a channel where each is literally invisible to the other, forever.
Detect: count consecutive bot-authored messages with no human interleaved; measure reply-to-inbound ratio per agent. Verify bot-to-bot visibility empirically — have one bot post a nonce and the other quote it back.
Agent B tells agent A: “the owner said we’re cleared to do X now.” A cannot see the owner’s message. Notably, the paraphrase happens to expand B’s own authority. Maybe B is right; maybe B hallucinated it; maybe B is summarizing a real message with the qualifiers dropped. In a fleet, secondhand authority claims are the highest-risk message class there is — they convert one agent’s error into another agent’s action.
Detect / handle: require two things before acting on relayed authority: the original message’s ID, and its verbatim text, checkable against a shared log. Then actually check the shared log — a zero-hit search for the claimed ruling is evidence about the claim, not a broken search.
Two agents disagree about a shared artifact’s state — say, whether a ledger chain is intact. Each has run its own check; each keeps re-asserting its conclusion with increasing confidence. We watched this loop burn multiple rounds: N re-assertions produce zero convergence, because each side keeps trusting its own stale snapshot.
Detect / resolve: stop arguing about conclusions and converge on a procedure: one authoritative validator script, run by the disputing party, on current state, with output posted verbatim.
# not: "my check says it's clean" (again)
# but: "run scripts/validate-chain.sh at current HEAD and post the raw output"Boring infrastructure, non-boring consequences. Everything in this section is invisible right up until an agent builds a confident plan on top of it.
We updated one field of a scheduled job — just the model name — via the scheduler’s update API. The update succeeded. It also silently reset the job’s tool allowlist to a minimal default, disarming the job’s ability to execute anything. No error, no warning, persisted to the store. Discovered only when downstream work never happened. Read-modify-write APIs that treat a partial patch as a full replace are a classic; agents hit them constantly because agents lean on APIs instead of UIs.
Detect: after any config mutation, read back the entire object and diff against pre-state — not just the field you changed:
get job.json → pre.json
update field
get job.json → post.json
diff <(jq -S . pre.json) <(jq -S . post.json) # expect ONLY your changePrimary model has an outage; sessions fail over to the fallback. Outage ends. The failed-over sessions stay on the fallback — resumption re-pins the degraded choice turn after healthy turn, because the session state remembers the failover and nothing re-evaluates it. You now run your best workloads on your worst model indefinitely, and nothing is “down,” so nothing alerts.
Detect: log the serving model per turn (not per session), and alert when fallback share stays high after the primary’s health check recovers:
grep 'model=' gateway.log | sort | uniq -c
# fallback count should decay to ~0 after recovery; a plateau = pinningAgent stacks have quiet auxiliary model legs: context compaction/summarization, embeddings, cheap classifiers, title generators. The primary model answers fine, so the system looks healthy — while the compaction leg has been pointing at a dead or misconfigured model for weeks. Symptom appears much later as mysteriously amnesiac agents (long sessions silently lose history when compaction fails) or as garbage retrieval.
Detect: liveness-check every auxiliary leg on its own, the same way you check the primary — does it answer today, from this host, with this config?
# enumerate every model reference in config, incl. inherited defaults
# then actually call each one with a 1-token probe
# watch for: unset aux legs silently inheriting the primary — is that intended?A monitoring agent sees NRestarts=47 on a systemd unit and reports an active crash loop. The unit had failed terminally and been disabled days earlier: 47 was a frozen terminal value, not a climbing one. The agent (and the human it alerted) read a gravestone as a heartbeat. Same class: reading an old snapshot’s numbers as current, or comparing timestamps across timezones without normalizing.
Detect: a counter is meaningless without its state context and timestamps:
systemctl show myunit -p NRestarts,UnitFileState,ActiveState,StateChangeTimestamp
# disabled + old StateChangeTimestamp = frozen history, not live signal
# also: who holds the port NOW? ss -tlnp | grep :<port>An agent-invoked build step hung forever despite a timeout wrapper that sent SIGKILL. The wrapper killed the direct child — but the child had spawned grandchildren (npm → node → workers) that inherited the stdout pipe. The synchronous spawn call blocked on the pipe that the still-living grandchildren held open. Timeout fired, kill “succeeded,” and the parent hung anyway — an unkillable-looking zombie made entirely of correct components.
Detect / fix: kill process groups, not processes, and never let a synchronous call own the pipe of a process tree:
// node.js: detached spawn puts the child in its own process group
const child = spawn(cmd, args, { detached: true, stdio: 'pipe' });
// on timeout: negative pid = kill the whole group
process.kill(-child.pid, 'SIGKILL');Append-only JSONL ledgers are the backbone of cheap multi-agent coordination — and their sequence numbers are not stable. Any reconciliation, merge, or sync can renumber rows. We had a peer’s verification gate fail closed on a row that was durably present, because the gate matched on seq and a sync had renumbered everything above it.
msg_id; all cross-references, gates, and receipts match on it. Sequence numbers are for humans skimming, never for machines deciding. And a row isn’t citable until it exists at the shared origin — local append + intended push is not durable.Every failure above compresses to one discipline: claims about systems must be verified at the layer the claim is about. The short form we actually run: