The index is clean, but the old answer still appears
Consider a document assistant that caches a finished answer after retrieval. A user loses access to a source document. The next vector search would exclude it, but an earlier cache lookup returns the finished answer before retrieval happens. The search permission check never runs.
This is a constructed failure scenario, not a report of a Dreamtsoft customer incident. The local fixture represents two cached answers and five changes to their sources. A cache that always serves the existing answer produces seven unsafe serves in ten reads. A guard that checks current source state allows three and denies seven. No retriever or language model is called.
The diagnostic question is whether every route that can supply an answer carries the same access decision. Index deletion is only one route. A response cache, saved conversation or precomputed summary can retain text derived from a document that is no longer available to the caller.
Follow the answer's dependencies
The example has document A at version 3 and document B at version 7. One answer depends only on A; another depends on both. Each cached answer records the source IDs and versions used to create it. Those dependencies let the application ask whether the answer remains eligible to serve.
| Source change | Answer from A | Answer from A and B |
|---|---|---|
| No change | Allow | Allow |
| User loses access to A | Deny | Deny |
| A is marked deleted | Deny | Deny |
| B advances to version 8 | Allow | Deny |
| A is absent from the authority | Deny | Deny |
The B-version case matters. Invalidating everything is safe for this fixture but wastes an answer whose only source is unchanged. Reusing everything serves an answer based on superseded B content. Dependency tracking makes the distinction explicit without pretending that a text hash alone proves authorization.
The guard fails closed when it cannot find a source. That is a chosen policy for this example. An unavailable authorization service should have an explicit product response, such as a temporary error, rather than being treated as permission to reuse whatever happens to be cached.
Check access before passages enter the prompt
Authorization still belongs on the retrieval path. A query filter should be constructed from trusted user and tenant context, not from group names supplied by the caller. Do not retrieve protected passages into an external model and rely on removing their citations later.
Microsoft's security-filter pattern stores access identities in searchable documents and uses the authenticated caller's identities to filter results. The article also distinguishes string-based filtering from a full authentication mechanism. The application remains responsible for constructing the appropriate filter. Microsoft: security filter pattern.
A valid retrieval filter does not automatically cover answer-cache hits. Put the eligibility check on that path too. If generation takes long enough for permissions to change before delivery, define a final authorization boundary. A response that has already been streamed to the user cannot be made unseen by a later revocation.
For retrieval quality and delivery planning, Pharos Production's RAG knowledge systems with permission-aware retrieval and deletion tests describes ingestion, permission filtering, citation verification and evaluation. Its freshness and governance section explicitly includes access revocation and deletion across the retrieval path. That is a relevant scope to request when evaluating a RAG implementation, alongside evidence for derived-answer caches.
Run the ten cache reads
Download the Python fixture and result record. Run python3 experiment.py in a writable folder. It uses local dictionaries to represent authoritative document state and writes its assertions beside the script.
The unchanged state allows both answers. Updating B allows only the answer from A. Revocation, deletion and missing A each deny both answers. An additional cross-user assertion confirms that a user outside A's allowed set is denied even when its version is unchanged.
These counts describe the declared ten-read fixture, not a measured leak rate. It assumes the guard sees current authority data. A stale authorization cache can make the guard approve an answer that should now be denied. Test that propagation boundary separately and specify which state is authoritative when systems disagree.
Deletion needs a record beyond the vector ID
Map a source document to all generated chunks and any persisted summaries. Removing a source from its origin does not prove that every downstream copy has disappeared. Keep enough identifiers to target the derived objects and verify the relevant serving paths.
Microsoft's search deletion guidance calls out orphaned documents when source records disappear before an indexer observes deletion. It also distinguishes document deletion from later physical storage reclamation. A storage-size graph is therefore not a sufficient test that an answer can no longer be served. Microsoft: delete search documents.
Handle late ingestion jobs as another path. An old task must not recreate a source that a newer deletion has retired. The existing stale search-index deletion example addresses event ordering. Here the additional concern is an answer derived from several sources and reused without a fresh retrieval.
Close the incident with a serving-path test
Prepare a synthetic document containing a distinctive harmless phrase. Generate an answer while the test user has access, then revoke access while deliberately leaving the answer cache populated. Exercise every cache mode your product supports. The response must not expose the phrase or its protected source metadata after the agreed enforcement boundary.
Repeat with deletion, a source version change and an authorization lookup failure. Test multi-source answers as well as single-source ones. Decide whether saved conversation history is a retained record or a recomputed view; that product decision changes what revocation promises.
Log the policy version and source identifiers used for the decision without putting protected passages into unrestricted logs. Record the last permitted response and the first enforced denial using the actual runtime. The local fixture supplies expected decisions, while the integration test establishes whether your connectors and caches deliver them in time.
Sources
Documentation checked .
Dreamtsoft Editorial
The multi-source answer is denied when B changes, while the answer based only on A remains eligible. This makes source dependency tracking testable without calling a language model.