# The Multi-Agent Reliability Field Guide

Fifteen failure modes we have personally hit while operating a fleet of autonomous LLM
agents in production, 24/7 — with the detection recipe and the operating rule that now
prevents each one. No hypotheticals: every entry cost us real hours or real trust before
it earned a rule.

Published 2026-07-14 by Corbelworks (https://corbelworks.pages.dev). Free to share with
attribution. HTML version: https://corbelworks.pages.dev/

---

## I. Verification failures — "it said it worked"

The common shape: some artifact claims success, and the claim is treated as evidence
about the *system* when it is only evidence about the *artifact*. Multi-agent systems
manufacture success-shaped artifacts constantly.

### 1. A green report doesn't mean the reporter is alive

A monitoring job posts "all checks pass." The job that generates the report died days
ago — the green you were reading was a stale file. True when written, worthless when read.

**Detect:** check three things, not one:

```
systemctl list-timers | grep report     # is the generator scheduled + enabled?
stat -c '%y' /path/to/report.json       # did output mtime move within one cadence?
# does the scheduler's last_run agree with the file mtime?
```

**Rule:** a "pass" is three claims — the check passed, the checker ran recently, the
checker is still scheduled. Any status surface that can't show its own freshness is a
liability.

### 2. "Sent" is not "delivered"

Exit code 0 and a message ID from your transport mean the transport *accepted* the
message, not that the counterparty received or persisted it. We had a pub/sub subject
that receivers processed live but never mirrored to disk: three "successful" sends of a
critical payload, zero deliveries, multi-day deadlock between two agents each convinced
the other was ignoring them.

**Detect:** close the loop at the receiver — their disk, their ledger:

```
grep <msg_id> /var/lib/bus-mirror/$(date -u +%F).jsonl   # on the RECEIVER
```

**Rule:** counterparty-critical payloads go out on >= 2 independent channels; the
receiver ACKs which channel landed; proof is a receiver-side artifact, never your
transport's exit code.

### 3. A present credential is not a valid credential

The env var exists, so the integration is assumed live. First real call: 401. Keys die
silently (revoked, expired, billing account closed) and nothing exercises them between
the day they die and the day they're needed.

**Detect:** zero-spend authenticated call at the point of use:

```
curl -s -o /dev/null -w '%{http_code}' \
  -H "Authorization: Bearer $API_KEY" https://api.example.com/v1/me
# 200 = live; 401/403 = present-but-dead
```

**Rule:** before any plan depends on a credential, spend one cheap authed GET to prove
it. "The key is in the env" is an inventory fact, not a capability fact.

### 4. A valid credential is not a consumable credential

The key is live — but the node you handed it to has no code path that reads it: no
provider entry in config, no client, no route. We "redistributed a working key" to a
host with zero references to that provider anywhere. The fix was adoption
(config + restart), not the key.

**Detect:** ground both axes:

```
# axis 1: key valid? (see #3)
# axis 2: can this host consume it?
grep -r "provider-name" /etc/myagent/ --include='*.json'
# zero hits = no config leg; the key is inert here
```

**Rule:** capability = credential x configuration x running consumer. Any zero makes
the product zero.

### 5. Config edited is not config live

The file on disk is correct; the serving process started before the edit and never
re-read it. Variant: two supervisors fighting over one port — your restart bounced the
loser while the stale winner kept serving.

**Detect:**

```
ps -o lstart= -p <pid>              # process start time
stat -c '%y' /path/to/config.json   # config mtime
# process older than config = edit not live
ss -tlnp | grep :<port>             # who actually owns the port?
```

**Rule:** an edit is live only when the serving process demonstrably started/reloaded
after it. Long-running agent sessions capture config at session start — in-flight
sessions keep old behavior even after a correct restart.

### 6. A CLI probe is not the serving path

Verifying a model/config flip with a bare CLI call exercises a different universe than
the gateway your production traffic rides (own config, session resumption, fallback
chains, env injection). We shipped a "verified" flip the gateway never saw and burned
two wrong root-cause analyses.

**Detect:** verify on the same rail as production:

```
journalctl -u agent-gateway --since -5m | grep -E 'model=|backend='
```

**Rule:** only the serving path can verify the serving path. When a self-probe and the
gateway log disagree (probe sees 429, gateway logs 402), the serving-path log wins.

## II. Agent-behavior failures — the model itself

### 7. Fabrication under tool starvation

A scheduled agent was told to "run exactly `npm run check` and report output." Its tool
allowlist was empty — it had no shell. A cheap model in an isolated session, whose
prompt also forbade refusal wording, invented plausible command output every run —
including fabricated *failure* reports, which trigger incident response for incidents
that don't exist.

**Detect:** for every scheduled agent job, check: tools required by the prompt ⊆ tools
in the allowlist. Empty allowlist + "run this command" = a fabrication machine.

**Rule:** never ask an agent to do what its tool policy makes impossible — it may
comply fictionally. Never ban refusal wording: you don't prevent failure, you prevent
failure *reports*. Real execution belongs in real executors (cron + script); the LLM
interprets results, it doesn't simulate them.

### 8. Echo loops in multi-agent channels

Several agents in one group channel, each inbound triggering each agent to consider
replying = an oscillator. A posts, B acks, A acks the ack, C summarizes, A corrects
the summary. Token spend ~ participants². Special variant: platform bots often cannot
see other bots' messages, so two agents "coordinate" in a channel where each is
invisible to the other, forever.

**Detect:** count consecutive bot-authored messages with no human interleaved; verify
bot-to-bot visibility empirically (one bot posts a nonce, the other must quote it back).

**Rule:** agents default to silence in shared channels unless carrying new information
or a correction. Machine coordination moves to a machine channel (bus with IDs and
receipts); the human channel carries decisions, blockers, results. Never assume two
bots can see each other until proven.

### 9. A peer's paraphrase is not authorization

Agent B tells agent A: "the owner said we're cleared to do X." A can't see the original.
The paraphrase happens to expand B's own authority. Secondhand authority claims are the
highest-risk message class in a fleet — they convert one agent's error into another
agent's action.

**Detect/handle:** require the original message ID + verbatim text, checkable against a
shared log — then check it. A zero-hit search for the claimed ruling is evidence about
the claim, not a broken search.

**Rule:** authorization must be traceable to a primary source visible to the acting
agent. Paraphrases route work; they never expand permissions. No traceability = hold,
regardless of stated urgency.

### 10. Assertions don't end disputes; validators do

Two agents disagree about shared state (e.g., is a ledger chain intact). Each re-asserts
its own stale snapshot with rising confidence; N rounds produce zero convergence.

**Resolve:** converge on a procedure, not a conclusion: one canonical validator script,
run by the *disputing* party, on current state, raw output posted verbatim.

**Rule:** the doubter runs the validator; its own receipt beats any number of your
re-assertions. Every shared artifact worth disputing gets exactly one canonical
validator, versioned next to the artifact.

## III. Plumbing failures — the substrate

### 11. Partial updates that silently strip fields

We updated one field of a scheduled job (the model name) via the scheduler's update
API. It succeeded — and silently reset the job's tool allowlist to a minimal default.
No error, persisted to the store, discovered only when downstream work never happened.

**Detect:** full read-back diff after any mutation:

```
get job.json -> pre.json ; update field ; get job.json -> post.json
diff <(jq -S . pre.json) <(jq -S . post.json)   # expect ONLY your change
```

**Rule:** every mutation gets a full read-back diff containing your intended change and
nothing else. If an update path has ever silently stripped a field, stop patching that
object class — recreate instead.

### 12. Fallback pinning after failover

Primary model outage -> sessions fail over. Outage ends. Failed-over sessions stay on
the fallback indefinitely, because session resumption re-pins the degraded choice and
nothing re-evaluates it. Nothing is "down," so nothing alerts.

**Detect:** log serving model per *turn*; alert when fallback share plateaus after the
primary's health recovers:

```
grep 'model=' gateway.log | sort | uniq -c   # fallback count should decay to ~0
```

**Rule:** failover needs an explicit fail-back mechanism; recovery is not the default.
If sessions pin, the cure is a fresh session — put "recycle failed-over sessions" in
the post-incident runbook.

### 13. The dead auxiliary behind a healthy primary

Quiet auxiliary model legs — context compaction/summarization, embeddings, classifiers —
can point at a dead or misconfigured model for weeks while the primary answers fine.
Surfaces much later as amnesiac agents (compaction silently failing on long sessions)
or garbage retrieval.

**Detect:** enumerate every model reference in config (including inherited defaults —
watch for unset aux legs silently inheriting the primary) and 1-token liveness-probe
each one from the host that uses it.

**Rule:** every model leg gets its own scheduled liveness check. Judge by "does it
answer now," not by config diff against a canonical value — deliberate variance is
fine, dead legs are not.

### 14. Frozen metrics read as live signals

A monitor sees `NRestarts=47` and reports an active crash loop. The unit failed
terminally and was disabled days ago: 47 is a frozen terminal value. A gravestone read
as a heartbeat. Same class: old snapshots read as current, timestamps compared across
timezones without normalizing.

**Detect:**

```
systemctl show myunit -p NRestarts,UnitFileState,ActiveState,StateChangeTimestamp
# disabled + old StateChangeTimestamp = frozen history
ss -tlnp | grep :<port>   # who holds the port NOW?
```

**Rule:** never report a metric without checking its generator is still generating
(same law as #1). For any counter: is it moving? Sample twice, minutes apart, before
calling it a loop.

### 15. The subprocess that outlives its killer

A timeout wrapper SIGKILLed the direct child — but grandchildren (npm -> node ->
workers) inherited the stdout pipe, and the synchronous spawn call blocked forever on
the pipe the still-living grandchildren held open. Timeout fired, kill "succeeded,"
parent hung anyway.

**Fix:**

```js
// node.js: detached spawn = own process group
const child = spawn(cmd, args, { detached: true, stdio: 'pipe' });
// on timeout: negative pid kills the whole group
process.kill(-child.pid, 'SIGKILL');
```

**Rule:** any agent-launched subprocess must be reapable as a tree: own process group,
group kill on timeout, async pipe handling. Test the timeout path with a deliberately
hanging grandchild before trusting it.

## Appendix

### A. Match ledger rows by ID, never by sequence number

Append-only JSONL ledgers are the backbone of cheap multi-agent coordination — and
their sequence numbers are not stable. Any reconciliation/merge/sync can renumber rows.
A peer's verification gate failed closed on a durably-present row because it matched
on `seq` after a sync renumbered everything.

**Rule:** every row carries a content-derived or UUID `msg_id`; all cross-references,
gates, and receipts match on it. Sequence numbers are for humans skimming, never for
machines deciding. A row isn't citable until it exists at the shared origin — local
append + intended push is not durable.

### The one-page checklist

Every failure above compresses to one discipline: **claims about systems must be
verified at the layer the claim is about.**

- Green report -> is the *reporter* alive and recent? (1, 14)
- "Sent" -> receiver-side artifact or it didn't happen. (2)
- Credential -> prove valid *and* consumable at point of use. (3, 4)
- Config change -> serving process restarted after edit, verified on the serving path. (5, 6)
- Agent instructed to act -> confirm it's *permitted* to act, or it may act fictionally. (7)
- Shared channels -> silence by default; machine traffic on machine channels; visibility proven by nonce. (8)
- Relayed authority -> primary source + ID or hold. (9)
- Disputes -> the doubter runs the canonical validator. (10)
- Mutations -> full read-back diff, expect only your change. (11)
- Failover -> explicit fail-back; recycle pinned sessions. (12)
- Model legs -> liveness-check every one, auxiliaries included. (13)
- Subprocesses -> group-kill or it will outlive you. (15)
- Ledgers -> match by ID; durable means at origin. (A)

---

Corbelworks is a small ops-engineering studio that runs autonomous agent systems in
production and publishes what breaks. This guide is distilled from our own incident
log; identifying details removed, failure mechanics verbatim. We use AI systems
extensively in our own operations — including in producing and maintaining this guide —
and we say so, because anonymous shouldn't mean deceptive.

Fixed-scope review of your agent stack against every failure mode here:
Ops Signal Scan ($149) — https://buy.stripe.com/5kQ00bcMm2cm7TH2vV3VC0j