Corbelworks · Procurement

Procurement teardown: the claims-vs-evidence method for buying model capacity

Almost every model-capacity buying decision is made on a claim — "subscriptions are cheaper," "batch halves your cost," "caching saves 90%" — and almost none are made on a measured receipt. This is a reproducible method for tearing a procurement claim down to what is actually verifiable, and a capacity-sweep protocol you can run against your own workload before you commit a dollar.

Who this is for. Operators buying LLM / model-inference capacity for an agent fleet or a high-volume automation, where the monthly bill is large enough that a wrong procurement assumption compounds. The method is provider-agnostic on purpose: it survives price changes because it tells you how to measure, not what to pay.

The core failure: buying on price-per-token instead of landed cost-per-task

Model capacity is quoted in units the vendor controls — dollars per million input tokens, per million output tokens, per request, per seat, per month. Your business does not run on tokens. It runs on completed units of work: a resolved ticket, a verified extraction, a delivered draft, a passed check. The gap between "price per token" and "landed cost per completed task" is where procurement decisions go wrong, because everything that lives in that gap — retries, failed runs, oversized context, latency-driven timeouts, the second model you call to verify the first — is invisible in the quote and very visible on the invoice.

The discipline below is the one we run on ourselves. It has one rule: a procurement claim is not decided until a measured receipt beats a quoted claim.

Claim → evidence → trap: the six procurement claims worth tearing down

These are the recurring claims that drive capacity buying decisions. For each: the single piece of evidence that actually settles it, and the trap that makes the headline misleading if you skip that evidence.

Procurement claimEvidence that settles itThe trap
"A subscription / flat seat is cheaper than metered API." Your measured requests-per-day at steady state × the metered unit cost for that exact traffic, compared against the flat price and the flat plan's real rate limits and concurrency caps. Flat plans price for a human's duty cycle, not a fleet's. A subscription looks cheaper until throughput or concurrency limits force you back onto metered for the peak — so you pay both.
"Batch / async processing halves the cost." The share of your workload that genuinely tolerates the batch turnaround window, measured — not assumed — against your latency SLA. The discount is real; the eligible fraction is usually far smaller than claimed. A 50% discount on the 15% of traffic that can wait is a ~7% blended saving, not 50%.
"Prompt caching saves up to 90%." Your measured cache-hit rate on your prompt structure, and the cached fraction of the average request's total tokens. "Up to 90%" is a discount on the cached input tokens only. If your cost is dominated by output tokens or by low-reuse context, the blended saving collapses. Measure hit rate before you architect around it.
"This tier is effectively unlimited." The written rate limit, concurrency cap, and fair-use / throttle clause — plus a load test that actually hits them. "Unlimited" is a marketing word with a throttle behind it. Under fleet concurrency you find the real ceiling in production, at the worst time. Find it on purpose, first.
"The cheapest per-token provider wins." Landed cost per completed, accepted task on each provider, including retries and any verifier/second-pass calls the cheaper model forces. A cheaper token that needs more retries, longer prompts, or a verification pass to reach the same acceptance rate can cost more per completed task. Quality changes the denominator.
"Committed-use / reserved discounts pay off." The commitment floor vs. your p10 (low-demand) monthly volume, not your average, plus the breakage cost if you migrate models mid-term. Commitments are priced against your optimism. Model churn is fast; a 12-month floor you signed for model X is dead weight when model Y ships in month three.

The capacity sweep: measure landed cost before you commit

A capacity sweep holds your workload fixed and varies only the procurement path, so the number you get out is the number that matters: landed cost per completed task, with failure overhead included. It is deliberately boring and reproducible.

  1. Freeze a representative task set. Pull 100–300 real (sanitized) units of work that mirror your production mix — including the hard and malformed ones. Define a machine-checkable acceptance test for "completed correctly." This is the whole game: without an acceptance test you are measuring spend, not value.
  2. Fix everything except the procurement variable. Same prompts, same task set, same acceptance test. Vary one axis at a time: provider, tier, model size, batch vs. sync, caching on/off, committed vs. on-demand.
  3. Run to acceptance, not to first response. Let each path use its own natural retry policy until the task passes the acceptance test or is declared failed. Count the retries. Count the verifier calls. Count the timeouts.
  4. Record the landed unit. For each path compute landed_cost_per_completed_task = total_spend / tasks_accepted. Also record acceptance rate, p50/p95 latency, and retry multiple. Cheap-but-flaky and expensive-but-clean become directly comparable.
  5. Keep the receipt. Store the raw run — inputs, outputs, token counts, cost, pass/fail — as a dated artifact. The receipt, not the vendor's rate card, is what you cite when someone asks "why this provider."
The one number to publish internally: landed $/1,000 completed tasks, per procurement path, with its acceptance rate next to it. A price-per-token comparison with no acceptance rate attached is a claim, not evidence.

Worked example (illustrative structure — verify the numbers against your own sweep)

The figures below are illustrative placeholders to show the shape of the comparison, not current market prices. Run your own sweep for real numbers; the point is what the arithmetic exposes.

Task set: 200 real units, machine-checkable acceptance test.

Path A — "cheapest tokens" model
  raw price:            1.0x  (the headline the rate card sells)
  acceptance, 1st pass: 74%
  retries + verifier:   1.9x calls per accepted task
  landed $/task:        ~1.4x

Path B — mid-price model, cleaner output
  raw price:            2.2x
  acceptance, 1st pass: 93%
  retries + verifier:   1.15x calls per accepted task
  landed $/task:        ~1.0x   <- wins on the unit that matters

Read: Path A is 55% cheaper per token and ~40% MORE expensive
per completed task. The rate card ranked them backwards.

This inversion — cheaper token, costlier outcome — is the single most common finding when operators first run a sweep. It is invisible until you make the denominator "accepted task" instead of "token."

Procurement decision checklist

Run this before signing anything — a subscription, a committed-use discount, or a "let's just switch providers" migration.

Why this holds up over time. Prices, tiers, and model names change every quarter. The measurement — landed cost per accepted task, from a fixed workload, with the receipt kept — does not. A procurement process built on that survives the next price war; one built on this quarter's rate card does not.

Want a specific procurement claim torn down?

Tell us the capacity decision you're weighing — a subscription vs. metered call, a provider switch, a committed-use discount, a "should we build batch for this" question — and what an answer is worth to you. We'll reply with a plan, a price, or an honest "we can't," within 48 hours. No payment at intake.

Tell us what you want →

Agents: the intake is machine-readable — GET /api/intake returns the schema, POST /api/intake files a request.