Eight callers reach the recovery gate
Eight callers arrived while a dependency was being tested for recovery. With a guard that only checked whether the circuit was open, all eight entered. Replacing that guard with an atomic reservation limited the same burst to two admitted calls and six immediate rejections. Both variants eventually reached CLOSED, so their final state alone concealed the difference.
These are observations from a controlled Python policy lab, not a production incident. The lab also exposed a separate accounting fault: a successful call admitted during an older trial helped close a newer trial. Adding a generation check prevented that result from counting toward the new recovery decision.
A circuit breaker stops admitting calls after its failure policy trips. OPEN rejects normal work. HALF_OPEN admits a limited recovery sample. CLOSED resumes normal admission. The important recovery contract is more specific than the state name: who owns permission to call, which trial owns the result and what happens to work that has not finished.
The Microsoft circuit breaker pattern describes restricted recovery traffic and treats retry as a separate concern. Our lab makes those recovery decisions observable with pending calls and deliberately ordered completions.
Reserve permission before starting the call
Permission must be reserved in the same critical section as the state check. Reading HALF_OPEN and incrementing a counter later lets callers make decisions from the same available capacity. Hold the lock only while updating local state. The remote operation runs outside it.
The lab grants at most two tickets in one recovery generation and requires two successful results from that generation before closing. A failed probe immediately reopens the circuit. Those are explicit experimental rules, not universal library defaults.
This is a sample budget rather than a replenishing concurrency semaphore: a completed success contributes evidence but does not return another sample slot. An unused reservation can be returned before invocation. A bulkhead answers a different question by limiting active resource use, including work admitted in CLOSED.
Resilience4j's circuit breaker documentation distinguishes permittedNumberOfCallsInHalfOpenState from slidingWindowSize. The sliding window stores observations. It does not cap concurrent calls in CLOSED. Its half-open decision uses configured failure and slow-call rates, so the lab's any-failure rule should not be copied as a description of Resilience4j.
Two admitted probes also need not mean recovery after the first success. In our mixed-outcome case, the first success left the breaker HALF_OPEN. The second probe failed and moved it to OPEN. Inspect the completion rule as well as the admission limit.
A late success belongs to its original trial
Call old-slow and a failing peer entered the first trial. The peer failed, reopening the circuit. After the cooldown, current-a entered the next trial. Then old-slow returned successfully, followed by current-a.
The unfenced policy counted both successes against the new trial and reached CLOSED. Only one belonged there. With generation checking enabled, the older result was ignored for the state decision and the breaker remained HALF_OPEN after current-a. A second current-generation success was needed to close it.
The ticket carries its admission generation. Completion processing compares that value with the current generation while holding the state lock. A mismatch can still be recorded as an operation outcome, but it contributes no evidence to the new trial. State transitions and manual resets must invalidate the appropriate older tickets.
This mechanism is present in maintained code too: the reviewed Sony gobreaker implementation returns a generation from beforeRequest and checks it in afterRequest before updating outcomes. That is source inspection as of November 16, 2026, not an execution test of gobreaker or a claim about every released version.
Local completion fencing also differs from preventing stale workers from writing business data. A generation comparison inside a breaker cannot enforce a database write condition. The background-job fencing experiment covers that storage boundary.
The nine cases and their checkpoints
The downloadable lab uses standard-library asyncio tasks. An event barrier holds admitted calls pending until all eight burst callers have attempted admission. Other cases use explicit completion order and a virtual clock. No wall-clock sleeps or network service are involved.
| Case | Controlled input | Observed checkpoint |
|---|---|---|
| State-only burst | Eight callers before any completion | Eight admitted, peak pending eight |
| Bounded burst | Same burst, two-ticket sample | Two admitted, six rejected, peak pending two |
| Mixed outcomes | One current success, then one failure | HALF_OPEN after success; OPEN after failure |
| Late success, unfenced | Old success, then one current success | CLOSED using evidence from two trials |
| Late success, fenced | Identical completion order | HALF_OPEN; second current success later closes |
| Expired trial | Two old calls pending, then two new calls | Peak pending four; two old results ignored |
| Unused reservation | Return a ticket before invoking work | Replacement admitted; earlier third caller rejected |
| Duplicate completion | Settle the same successful ticket twice | Duplicate ignored; still HALF_OPEN until peer succeeds |
| Independent instances | Four breakers, two tickets each | Eight calls pending across the model fleet |
The chart compares selected pending-call counts, not latency, throughput or failure rates. Python 3.14.7 executed all nine cases, with assertions for the stated checkpoints. A second run produced byte-identical JSON. The model starts OPEN and isolates recovery behavior. It does not implement closed-state failure sampling, a distributed coordinator or real transport cancellation.
Download experiment.py and results.json. Python 3.11 or later is required. Put the script in an empty directory, run it and compare the output with the supplied JSON. Its runtime version field will reflect your Python installation.
python3 experiment.py > local-results.jsonFor an application acceptance test, replace the modeled operation with a controlled test server that can hold responses open. Keep the admission and completion checkpoints. This article has not run those cases against a production client, Java library or Go library.
An expired trial does not stop old work
The expired-trial case held two calls pending, expired their recovery window and then admitted two calls in a newer generation after the cooldown. Four modeled calls were pending together. Old completions were fenced correctly, yet the two-ticket rule never promised a cross-generation resource ceiling.
A maximum half-open wait bounds how long the breaker waits for a decision. A per-call deadline bounds how long a caller waits for its operation. Neither statement proves that an underlying transport stopped consuming resources. Cancellation needs a defined propagation path, and a bulkhead needs to retain occupancy until the relevant resource actually ends.
Resilience4j documents maxWaitDurationInHalfOpenState separately from waitDurationInOpenState: the former bounds a half-open wait, while the latter controls the open interval before recovery is attempted. Its documented zero half-open maximum means waiting without that timeout. Check the contract of the exact library version deployed before relying on these settings.
Its automaticTransitionFromOpenToHalfOpenEnabled setting controls whether a later caller is needed to trigger the state change. Changing state does not itself establish that a probe succeeded. Our model makes the transition on admission after the cooldown, so low traffic can leave it OPEN until another caller arrives.
If a probe hangs, record the trial age and the still-pending operation separately. Reopening can unblock the state machine while the earlier work remains outstanding. Our fixture deliberately leaves that work running. It provides no evidence that any real HTTP cancellation succeeds.
Give each reservation one terminal outcome
Returning an unused permit is safe only under the contract that the dependency was never invoked. The lab reserves two tickets, rejects a third caller, returns one unused ticket and admits a replacement. Both actual probe outcomes must still be reported. Silently dropping a reservation can strand the trial without enough evidence to close.
The current Resilience4j permission interface documents tryAcquirePermission, outcome reporting through onSuccess or onError, and releasePermission for acquired permission that was not used. Prefer the library's supported decorator or execution wrapper. Manual permission handling adds cancellation and exception paths that need their own tests.
After invocation, a caller abandoning its wait does not establish that the backend operation never happened. Decide which outcome the breaker observes, whether cancellation reaches the transport and when resource occupancy ends. Keep these choices consistent with the application's request deadline and retry budget.
Double settlement is another independent fault. In the duplicate-completion case, the first successful callback counted once and its duplicate counted zero times. The breaker stayed HALF_OPEN with one peer pending. Settling a ticket once protects both the sample count and active-call accounting from duplicated callback paths.
One process cannot promise a fleet-wide probe cap
Four independent breaker instances with two permissions each admitted eight pending calls in the model fleet. A per-instance registry cannot promise a two-call ceiling across replicas. Restarts and synchronized cooldowns can create additional recovery bursts unless the design accounts for them.
Reuse a breaker at a deliberate failure-domain boundary, such as one dependency operation or unhealthy shard. A single broad breaker may reject healthy operations. A new breaker per request loses accumulated state. Per-tenant instances can multiply probe traffic and registry size. Choose bounded keys from the actual isolation need rather than treating every identifier as a new circuit.
Retry placement changes the unit being admitted. A breaker around an entire retry operation may see one logical completion while that operation makes several dependency attempts. A breaker checked for each attempt can reject a later retry after opening. Specify the order, stop retrying local open-circuit rejections and preserve the end-to-end deadline.
Polly's strategy documentation likewise separates breaker fault monitoring from retry execution. Its settings and recovery behavior are not interchangeable with our two-success model. A general concurrency ceiling belongs in a separate resource control, as the connection-pool saturation cases demonstrate for database connections.
Release a breaker only after these checks
Collect the state transition reason alongside allowed and rejected admission counts. Observe actual dependency calls and active resources separately from breaker outcomes. Use trial identifiers in bounded diagnostic logs rather than creating a metric label for every generation. A fallback response must not be mistaken for a successful dependency probe.
- Hold every recovery response open, send more callers than the intended sample budget and verify the dependency invocation count before releasing any response. A final CLOSED state cannot establish the admission cap.
- Fail one probe, enter a new trial and finish an older successful probe. Verify that only current-trial results contribute to the next recovery decision, including after a manual reset.
- Exercise unused-permission cancellation, cancellation after invocation and duplicate completion. Assert the terminal accounting for each path rather than treating every cancellation as an unused ticket.
- Expire the recovery window while work is pending. Measure the old and new resource occupancy together, then repeat across the actual number of client instances.
- Verify failure classification and retry composition with the deployed client. Decide explicitly how validation errors, caller cancellation, throttling and dependency failures affect the sample. A local circuit rejection is not a new backend failure.
A small recovery sample gives limited evidence about an entire dependency. Low traffic can delay a decision, and synchronized replicas can probe together even when every local cap works. Choose cooldown and sample size from the dependency's recovery behavior. Jitter can spread attempts but cannot enforce a fleet-wide ceiling. The acceptance evidence should show the admitted call count, the generation of each completion and the resources still occupied after the trial ends.
Sources
Documentation checked .

Continue the conversation
Comments (0)