When the Index Remembers What the Business Forgot: Auditing Retrieval Corpora for Decay
Photo: BalticServers.com, CC BY-SA 3.0, via Wikimedia Commons
Ask a team why its retrieval-augmented assistant gave a wrong answer, and the conversation tends to go immediately to the model. Temperature, chunk size, embedding version, reranker configuration. These are real variables. They are also, in most failures, not the cause.
The cause is usually upstream and much duller: the system retrieved a document that was accurate in 2021, or a document that was accurate but restricted, or a document that was superseded by a policy change nobody propagated. The retrieval layer did exactly what it was built to do. It faithfully surfaced the enterprise's accumulated memory, including the parts the enterprise would rather forget.
This is worth treating as an infrastructure problem, because it is one. A retrieval corpus is a data system with a lifecycle, a schema of sorts, ownership boundaries, and a physical footprint. It is usually governed as none of those things.
Rotation Is Not Retrieval
Most organizations have a retention policy, and most retention policies are honored in the breach. Documents get archived to cold storage on schedule and remain fully indexed for retrieval, because the archive tier and the retrieval pipeline are managed by different teams with different mandates. The storage team's job is to reduce cost. The AI team's job is to improve answer quality. Neither owns the intersection, and the intersection is where the rot lives.
Three classes of decay account for most of the damage.
Temporal rot. The content was correct and is now wrong. Pricing sheets, organizational charts, escalation matrices, product specifications, and regulatory summaries all decay at different rates, and none of them announce their expiration. A retrieval system has no native concept of supersession unless someone encodes it.
Structural rot. The document was never authoritative. Drafts, meeting notes, and abandoned proposals sit in the same index as ratified policy, and similarity search does not distinguish between them. A confident-sounding draft can outrank the actual policy simply because it uses the query's vocabulary more closely.
Ownership rot. The document has no current owner. Its author left, its team was reorganized, and it persists in the index as an orphan with full voting rights. Orphaned documents are the single most reliable source of confidently wrong answers in enterprise retrieval, and they are almost never tracked as a metric.
Build an Ownership Ledger Before You Tune Anything
Before adjusting chunking strategy, establish who is accountable for each document class. In practice this means a ledger with four fields per source system: the business owner, the technical owner, the authoritative source of record, and a review cadence.
Most enterprises cannot complete this exercise for more than a fraction of their corpus, and that finding is itself the deliverable. It tells you precisely where retrieval quality will fail first, and it gives you a defensible basis for excluding entire repositories from the index until ownership is established.
Exclusion is underused. A retrieval system that indexes sixty percent of the corpus with high confidence is generally more useful than one that indexes everything with unknown provenance.
Permission Drift Is a Security Problem Wearing a Quality Costume
Access-control drift deserves separate treatment because it fails in both directions, and the second direction is the dangerous one.
Over-restriction degrades quality: the assistant cannot retrieve the document that would answer the question, so it either declines or confabulates. This is annoying and visible.
Under-restriction is a genuine exposure: a document that was moved, re-shared, or inherited through a group membership change is now reachable by the retrieval layer and therefore by any user who asks a sufficiently well-phrased question. Retrieval systems inherit permission models from source systems that were designed for human browsing, not for semantic search across a consolidated index.
Audit this by sampling. Take several hundred indexed documents, resolve effective permissions at the document level, and compare them against the permissions the retrieval system believes apply. Discrepancies in the permissive direction should be treated as incidents, not backlog items.
Evaluations That Catch Rot Before Users Do
Standard retrieval evaluations measure whether the right document was returned. They do not measure whether the right document still deserves to exist. Add three test families.
Supersession tests. For a set of known policy changes, query the old state and the new state. A healthy system returns the current document, or both with clear dating. A rotted system returns the outdated one with full confidence.
Orphan probes. Query topics whose owning team has been dissolved. The system should either return nothing or flag low confidence. If it answers fluently, the index is speaking for a department that no longer exists.
Negative controls. Ask questions the corpus genuinely cannot answer. Measure refusal rate. A system that never refuses is not knowledgeable; it is uncalibrated, and it is almost certainly fabricating from stale material.
Run these on the same schedule as your model evaluations, because corpus rot and model updates compound. A model upgrade evaluated against a rotted corpus will produce misleading results in both directions.
Retention Is Also an Energy Decision
There is a physical dimension that rarely enters the discussion. Indexes are rebuilt, re-embedded, and re-ranked continuously. Stale content consumes compute on every rebuild and on many queries, and it does so invisibly because nobody attributes the cost to the document.
Attributing inference and re-indexing cost to source repositories changes behavior quickly. Teams that see a line item for maintaining an abandoned wiki tend to make decisions about it. That is not a coincidence; it is the same mechanism that makes chargeback work anywhere else. The corpus is infrastructure. Budget it like infrastructure, and the rot becomes someone's problem — which is the only condition under which it gets fixed.