Secret rotation: prove consumers switched credentials

A current secret is only one side of the handoff

Changing the value in a secret store does not establish that every application has started using it. The system that authenticates a credential, the store that distributes it and the processes that cache it can move at different times. Rotation needs a rule for crossing those boundaries.

Consider an API process and a background worker sharing a service credential. The API refreshes its cached value first. The worker still holds the previous version when the operator revokes it. A new API connection succeeds and the rotation dashboard looks healthy. The worker's next authentication fails.

The accompanying executable model makes that handoff inspectable. It compares early revocation with a gate that waits for both known consumers. Across 24 fresh-authentication probes, early revocation produces one rejection and the adoption gate produces none. These are deterministic model results, not a database availability test. The complete trace records every consumer version and accepted version set.

Decide which overlap the authority actually supports

A credential authority might accept several active credentials, replace a password in place or issue short-lived credentials that expire independently. The application cannot invent an overlap period that the authority does not support. Start the design review with the behavior of new authentication attempts during the transition.

AWS describes single-user and alternating-users rotation for database secrets. Single-user rotation has a window between changing the database password and updating its stored value. Alternating users can leave both identities valid after a rotation. AWS also notes that permissions changed on the original user must be reflected on its clone. Those are properties to inspect when choosing a mechanism, not a universal promise of uninterrupted operation.

Our model assumes an authority that can temporarily accept the labels old and new. They are version names, not credentials. The store publishes new only after the authority accepts it. An independently reachable store lets consumers refresh even if their service credential has stopped working. If either assumption is false, this handoff must be redesigned before using its results.

The decision is narrower than choosing a secret-management vendor. Specify who creates the new credential, which permissions it receives, how a fresh connection tests it and what confirms its propagation to every authentication endpoint. A successful read of the secret value proves distribution access. It does not prove that the target service will accept that value.

The revocation gate changes one failed connection

Both policies start with the API and worker using old. The authority then accepts new, the store points to it and the API refreshes. At step 3, the early policy removes old; the gated policy holds revocation because the worker has not adopted new. At step 4, the worker refreshes. Step 5 removes old in both runs.

The extra overlap is responsible for the difference in this fixture. No retries rescue the rejected probe, no existing session hides it and no propagation delay affects the authority. The result file records those limits alongside the counts.

A consumer's reported version needs a meaningful definition. For a connection pool, a process can load the new credential while continuing to use old authenticated sessions. Decide whether adoption means loading configuration, creating a successful new connection or replacing all sessions. These are separate observations. Never log secret material to demonstrate any of them.

Inventory the consumer that rarely runs

An adoption gate is only as complete as its inventory. A nightly exporter, a maintenance command or a dormant deployment can remain absent from ordinary request traffic. Waiting for acknowledgements from the API and worker says nothing about a third consumer that the coordinator does not know exists.

Record the runtime owner, refresh trigger and authentication test for each consumer. Include deployment versions that can still be started, scheduled jobs and emergency operating procedures. For a dormant consumer, the review needs either a controlled fresh-authentication check or an explicit decision that the old runtime is retired. An empty traffic chart does not make that decision.

The useful operational signal is version-scoped authentication success or failure with a safe identifier. Avoid usernames or labels that expose tenant information unnecessarily, and exclude secret values from application logs. OWASP's secrets-management guidance treats rotation, revocation and lifecycle metadata as related controls. The proposed worksheet here applies those concerns to a specific consumer handoff.

Use the handoff worksheet to expose unresolved consumers. Its rows are proposed acceptance checks and remain pending until a real system supplies evidence. The script does not establish an appropriate rotation frequency or assess a live credential's exposure.

Separate routine rotation from emergency revocation

A routine change can wait for controlled adoption if the old credential remains acceptable during that interval. A known compromise changes the decision: extending overlap also extends the usefulness of the exposed credential. The incident owner may choose immediate revocation and accept interruption while consumers recover through an independent path.

That choice belongs in the operating procedure before an incident. Identify which consumers can obtain replacement credentials without using the compromised credential, which operations can pause and who decides when a failed dependency must remain unavailable. Do not turn a routine availability gate into a rule that prevents urgent containment.

Rollback also needs a precise object. Moving a store pointer back to a revoked password does not restore authentication. Re-enabling an old credential can reintroduce the reason it was revoked. Treat credential authority state, application configuration and existing sessions as separate parts of a recovery decision.

For the two-consumer fixture, the final evidence is concrete: both consumers demonstrate a fresh authentication with new, the authority rejects old and each pending inventory row has an owner. In a real deployment, preserve that evidence without preserving the credential itself.

Sources

Documentation checked September 6, 2026.

1. AWS: rotation strategies
2. OWASP: Secrets Management Cheat Sheet