A timeout can free the wrong forty credits
A client stops waiting for an LLM response. Your gateway releases its budget reservation and admits another request. Later, usage arrives for the timed-out operation: the provider had continued processing it. The budget ledger gave away capacity that was still committed to unresolved work.
The experiment below isolates that accounting decision. With a 100-credit cap and a 40-credit reservation for each operation, releasing the unknown request admits three operations that eventually consume 120 credits. Retaining its reservation admits two operations and records 80 credits. These are invented integer units in an executed local model, not provider prices or a billing benchmark.
The model assumes every admitted operation eventually costs exactly its reserved amount. That lets us examine timeout handling without disguising estimation error as a concurrency result. Real token estimates and distributed storage need separate verification.
Define what the budget promises
An alert threshold tells someone to investigate usage. An admission limit decides whether another operation can start. Neither says that an in-progress provider request becomes free when a browser disconnects.
For the model, available capacity equals the cap minus settled usage minus unresolved reservations. Admission checks and reservation creation form one serialized action. A real implementation needs an equivalent atomic boundary across every worker that spends from the same account.
LiteLLM documents reservation-based budget enforcement and notes that disabling it can let concurrent requests exceed a limit because checks only see recorded spend. Its fail-closed option also addresses unavailable or stale coordination state. Those are implementation-specific controls to examine in your deployed version, not proof that this fixture tests the gateway. LiteLLM: budgets and rate limits.
An account can have several API keys. If the commercial promise is per account, creating another key must not create another independent budget. Apply the ledger scope to the entitlement being sold and carry that scope through retries and background continuations.
Compare two policies under the same event order
Operation A reserves 40 credits and is dispatched. The client times out, but its final usage is unknown. Operations B and C then request admission before any usage has settled. Eventually every admitted operation reports 40 credits.
| Timeout policy | Admitted operations | Final usage | Amount over the 100-credit cap |
|---|---|---|---|
| Release A while usage is unknown | 3 | 120 | 20 |
| Retain A until its usage settles | 2 | 80 | 0 |
The retained policy denies C because A and B already reserve 80 credits. It does not deny B. That distinction shows the effect of retaining one unresolved liability rather than stopping the entire account after any timeout.
Download the admission model and event records. Run python3 experiment.py. The output contains each reservation, denial and unknown-usage transition. A duplicate settlement for A leaves the total unchanged, which checks a second failure mode: receiving the same usage result more than once.
Keep unknown, failed and canceled outcomes distinct
A rejection before dispatch can justify releasing a hold if no billable work could have started. A transport failure after dispatch is a different state. It might have lost the response after the provider completed the operation. Preserve the provider request identifier and reconciliation status when the API supplies them.
Streaming adds another observation boundary. Anthropic documents usage information in streaming message events and notes that usage values in message deltas are cumulative. A consumer that stops before the final events may not observe a complete usage record. Check the actual provider protocol rather than summing every reported number or assuming a dropped stream costs nothing. Anthropic: streaming messages.
Cancellation needs evidence too. An application cancellation token says what the application requested. Whether a provider accepted cancellation, stopped work or supplied final usage depends on that API. The model makes no claim about those behaviors and sends no provider requests.
An expiration timer can mark a reservation for investigation. Automatically releasing every expired hold is a policy choice to accept possible unrecorded cost. If the product promises a strict cap, an unknown outcome cannot become zero solely because enough time passed.
Reconcile without charging twice
Keep a stable operation identifier for a provider attempt and a separate relationship to the user's logical request. A retry can be another billable attempt even if the UI considers it the same question. Account for each actual attempt while deduplicating repeated reports about that attempt.
Settlement replaces a reservation with known usage. If the actual usage is below the hold, return the difference to available capacity. If it exceeds the hold, record the overrun honestly and stop pretending the estimate was a bound. A hard ceiling needs enforceable limits on all billable dimensions, including any paid tools or retry fan-out that your application permits.
The earlier usage-metering correction guide addresses auditable adjustments after usage is recorded. Keep that correction history separate from the admission decision so a disputed record does not silently erase evidence of work already performed.
Test the ledger before advertising the cap
Use a fake provider with controllable outcomes. Hold A open after the client disconnects, send B and C concurrently through different gateway workers and inspect the account ledger. Then deliver A's usage twice. Neither concurrency nor duplicate delivery should change the declared accounting rule.
Also simulate a worker crash after dispatch but before saving the provider identifier, an unavailable reservation store and a usage report that arrives after the budget period changes. Assign unresolved work to its original accounting period according to the published policy; resetting a monthly display must not erase an outstanding hold.
Track unresolved reservation age as well as settled usage. A conservative policy can protect a cap while blocking customers indefinitely if reconciliation never finishes. Give support a documented resolution path with evidence and an audit record, rather than a button that silently treats unknown work as free.
Sources
Documentation checked .
Dreamtsoft Editorial
The 40-credit reservation is an exact bound only inside this model. A real cost estimate needs separate evidence before the product can promise the same hard ceiling.