Resilience and recovery

Circuit Breaker Half Open

A recovery state does not reserve permission. Follow nine controlled cases through probe admission, late results and work that remains pending after a trial expires.

Executed policy lab
Nine deterministic asyncio cases, virtual time, Python 3.14.7
Same eight-call burst
State-only guard admitted 8; atomic two-ticket budget admitted 2
Separate stale-result check
Older success ignored; one current success left the new trial HALF_OPEN
Model boundary
No network, latency benchmark or executed production library
A gate attendant holds two admission tokens as two couriers enter a workshop. Others wait outside, while a blue-tagged return parcel rests on a separate desk.
Story illustration of bounded recovery admission and an older result kept separate from the new trial.

Eight callers reach the recovery gate

Eight callers arrived while a dependency was being tested for recovery. With a guard that only checked whether the circuit was open, all eight entered. Replacing that guard with an atomic reservation limited the same burst to two admitted calls and six immediate rejections. Both variants eventually reached CLOSED, so their final state alone concealed the difference.

These are observations from a controlled Python policy lab, not a production incident. The lab also exposed a separate accounting fault: a successful call admitted during an older trial helped close a newer trial. Adding a generation check prevented that result from counting toward the new recovery decision.

A circuit breaker stops admitting calls after its failure policy trips. OPEN rejects normal work. HALF_OPEN admits a limited recovery sample. CLOSED resumes normal admission. The important recovery contract is more specific than the state name: who owns permission to call, which trial owns the result and what happens to work that has not finished.

The Microsoft circuit breaker pattern describes restricted recovery traffic and treats retry as a separate concern. Our lab makes those recovery decisions observable with pending calls and deliberately ordered completions.

Reserve permission before starting the call

Permission must be reserved in the same critical section as the state check. Reading HALF_OPEN and incrementing a counter later lets callers make decisions from the same available capacity. Hold the lock only while updating local state. The remote operation runs outside it.

The lab grants at most two tickets in one recovery generation and requires two successful results from that generation before closing. A failed probe immediately reopens the circuit. Those are explicit experimental rules, not universal library defaults.

This is a sample budget rather than a replenishing concurrency semaphore: a completed success contributes evidence but does not return another sample slot. An unused reservation can be returned before invocation. A bulkhead answers a different question by limiting active resource use, including work admitted in CLOSED.

Resilience4j's circuit breaker documentation distinguishes permittedNumberOfCallsInHalfOpenState from slidingWindowSize. The sliding window stores observations. It does not cap concurrent calls in CLOSED. Its half-open decision uses configured failure and slow-call rates, so the lab's any-failure rule should not be copied as a description of Resilience4j.

Two admitted probes also need not mean recovery after the first success. In our mixed-outcome case, the first success left the breaker HALF_OPEN. The second probe failed and moved it to OPEN. Inspect the completion rule as well as the admission limit.

A late success belongs to its original trial

Call old-slow and a failing peer entered the first trial. The peer failed, reopening the circuit. After the cooldown, current-a entered the next trial. Then old-slow returned successfully, followed by current-a.

The unfenced policy counted both successes against the new trial and reached CLOSED. Only one belonged there. With generation checking enabled, the older result was ignored for the state decision and the breaker remained HALF_OPEN after current-a. A second current-generation success was needed to close it.

A failed first trial reopens the breaker. In the next trial, the old success is ignored by generation. Two current successes are required to close.
Executed fenced sequence. A late operation can succeed without contributing evidence to a newer recovery trial. View full-size figure.

The ticket carries its admission generation. Completion processing compares that value with the current generation while holding the state lock. A mismatch can still be recorded as an operation outcome, but it contributes no evidence to the new trial. State transitions and manual resets must invalidate the appropriate older tickets.

This mechanism is present in maintained code too: the reviewed Sony gobreaker implementation returns a generation from beforeRequest and checks it in afterRequest before updating outcomes. That is source inspection as of November 16, 2026, not an execution test of gobreaker or a claim about every released version.

Local completion fencing also differs from preventing stale workers from writing business data. A generation comparison inside a breaker cannot enforce a database write condition. The background-job fencing experiment covers that storage boundary.

The nine cases and their checkpoints

The downloadable lab uses standard-library asyncio tasks. An event barrier holds admitted calls pending until all eight burst callers have attempted admission. Other cases use explicit completion order and a virtual clock. No wall-clock sleeps or network service are involved.

Nine executed policy cases: controlled input and the checkpoint that distinguishes each outcome
CaseControlled inputObserved checkpoint
State-only burstEight callers before any completionEight admitted, peak pending eight
Bounded burstSame burst, two-ticket sampleTwo admitted, six rejected, peak pending two
Mixed outcomesOne current success, then one failureHALF_OPEN after success; OPEN after failure
Late success, unfencedOld success, then one current successCLOSED using evidence from two trials
Late success, fencedIdentical completion orderHALF_OPEN; second current success later closes
Expired trialTwo old calls pending, then two new callsPeak pending four; two old results ignored
Unused reservationReturn a ticket before invoking workReplacement admitted; earlier third caller rejected
Duplicate completionSettle the same successful ticket twiceDuplicate ignored; still HALF_OPEN until peer succeeds
Independent instancesFour breakers, two tickets eachEight calls pending across the model fleet
Peak pending modeled calls: state-only burst8, bounded burst2, overlapping expired trials4 and four independent breakers8.
Actual counts from four selected model cases. Two tickets per generation or instance are not a cross-generation or fleet-wide resource cap. View full-size figure.

The chart compares selected pending-call counts, not latency, throughput or failure rates. Python 3.14.7 executed all nine cases, with assertions for the stated checkpoints. A second run produced byte-identical JSON. The model starts OPEN and isolates recovery behavior. It does not implement closed-state failure sampling, a distributed coordinator or real transport cancellation.

Download experiment.py and results.json. Python 3.11 or later is required. Put the script in an empty directory, run it and compare the output with the supplied JSON. Its runtime version field will reflect your Python installation.

Run the downloaded standard-library Python policy lab
python3 experiment.py > local-results.json

For an application acceptance test, replace the modeled operation with a controlled test server that can hold responses open. Keep the admission and completion checkpoints. This article has not run those cases against a production client, Java library or Go library.

An expired trial does not stop old work

The expired-trial case held two calls pending, expired their recovery window and then admitted two calls in a newer generation after the cooldown. Four modeled calls were pending together. Old completions were fenced correctly, yet the two-ticket rule never promised a cross-generation resource ceiling.

A maximum half-open wait bounds how long the breaker waits for a decision. A per-call deadline bounds how long a caller waits for its operation. Neither statement proves that an underlying transport stopped consuming resources. Cancellation needs a defined propagation path, and a bulkhead needs to retain occupancy until the relevant resource actually ends.

Resilience4j documents maxWaitDurationInHalfOpenState separately from waitDurationInOpenState: the former bounds a half-open wait, while the latter controls the open interval before recovery is attempted. Its documented zero half-open maximum means waiting without that timeout. Check the contract of the exact library version deployed before relying on these settings.

Its automaticTransitionFromOpenToHalfOpenEnabled setting controls whether a later caller is needed to trigger the state change. Changing state does not itself establish that a probe succeeded. Our model makes the transition on admission after the cooldown, so low traffic can leave it OPEN until another caller arrives.

If a probe hangs, record the trial age and the still-pending operation separately. Reopening can unblock the state machine while the earlier work remains outstanding. Our fixture deliberately leaves that work running. It provides no evidence that any real HTTP cancellation succeeds.

Give each reservation one terminal outcome

Returning an unused permit is safe only under the contract that the dependency was never invoked. The lab reserves two tickets, rejects a third caller, returns one unused ticket and admits a replacement. Both actual probe outcomes must still be reported. Silently dropping a reservation can strand the trial without enough evidence to close.

The current Resilience4j permission interface documents tryAcquirePermission, outcome reporting through onSuccess or onError, and releasePermission for acquired permission that was not used. Prefer the library's supported decorator or execution wrapper. Manual permission handling adds cancellation and exception paths that need their own tests.

After invocation, a caller abandoning its wait does not establish that the backend operation never happened. Decide which outcome the breaker observes, whether cancellation reaches the transport and when resource occupancy ends. Keep these choices consistent with the application's request deadline and retry budget.

Double settlement is another independent fault. In the duplicate-completion case, the first successful callback counted once and its duplicate counted zero times. The breaker stayed HALF_OPEN with one peer pending. Settling a ticket once protects both the sample count and active-call accounting from duplicated callback paths.

One process cannot promise a fleet-wide probe cap

Four independent breaker instances with two permissions each admitted eight pending calls in the model fleet. A per-instance registry cannot promise a two-call ceiling across replicas. Restarts and synchronized cooldowns can create additional recovery bursts unless the design accounts for them.

Reuse a breaker at a deliberate failure-domain boundary, such as one dependency operation or unhealthy shard. A single broad breaker may reject healthy operations. A new breaker per request loses accumulated state. Per-tenant instances can multiply probe traffic and registry size. Choose bounded keys from the actual isolation need rather than treating every identifier as a new circuit.

Retry placement changes the unit being admitted. A breaker around an entire retry operation may see one logical completion while that operation makes several dependency attempts. A breaker checked for each attempt can reject a later retry after opening. Specify the order, stop retrying local open-circuit rejections and preserve the end-to-end deadline.

Polly's strategy documentation likewise separates breaker fault monitoring from retry execution. Its settings and recovery behavior are not interchangeable with our two-success model. A general concurrency ceiling belongs in a separate resource control, as the connection-pool saturation cases demonstrate for database connections.

Release a breaker only after these checks

Collect the state transition reason alongside allowed and rejected admission counts. Observe actual dependency calls and active resources separately from breaker outcomes. Use trial identifiers in bounded diagnostic logs rather than creating a metric label for every generation. A fallback response must not be mistaken for a successful dependency probe.

  1. Hold every recovery response open, send more callers than the intended sample budget and verify the dependency invocation count before releasing any response. A final CLOSED state cannot establish the admission cap.
  2. Fail one probe, enter a new trial and finish an older successful probe. Verify that only current-trial results contribute to the next recovery decision, including after a manual reset.
  3. Exercise unused-permission cancellation, cancellation after invocation and duplicate completion. Assert the terminal accounting for each path rather than treating every cancellation as an unused ticket.
  4. Expire the recovery window while work is pending. Measure the old and new resource occupancy together, then repeat across the actual number of client instances.
  5. Verify failure classification and retry composition with the deployed client. Decide explicitly how validation errors, caller cancellation, throttling and dependency failures affect the sample. A local circuit rejection is not a new backend failure.

A small recovery sample gives limited evidence about an entire dependency. Low traffic can delay a decision, and synchronized replicas can probe together even when every local cap works. Choose cooldown and sample size from the dependency's recovery behavior. Jitter can spread attempts but cannot enforce a fleet-wide ceiling. The acceptance evidence should show the admitted call count, the generation of each completion and the resources still occupied after the trial ends.

Sources

Documentation checked .

  1. CircuitBreaker
  2. Circuit Breaker Pattern - Azure Architecture Center | Microsoft Learn
  3. Circuit breaker resilience strategy | Polly
  4. gobreaker.go
  5. Resilience4j manual permission contract

Continue the conversation

Comments (0)

    Leave a comment

    Your name and comment stay in this page and are cleared after the spam check.

    10–2,000 characters. Keep the discussion relevant to this article.

    Spam protection verification
    Spam protection loads when you begin the form.

    JavaScript is required to use this form and its spam protection.