3 min read
Why your RAG chatbot gives stale answers, and how to keep the index in sync
Most RAG pipelines only ever add to their vector index, so edited and deleted documents keep being served. The record-manager pattern that fixes it.
Short answer: most RAG systems only ever add to their vector index. When a source document is edited, the old chunks stay searchable next to the new ones. When a document is deleted, its chunks stay searchable forever. The fix is a record manager: a small database that remembers what was ingested, so every sync run can tell what is new, what changed, and what must be removed.
I built exactly this for a Pinecone-backed pipeline, and it is the part of RAG that tutorials usually skip.
What is going wrong?
A typical RAG ingestion script looks like this: read the documents, split them into chunks, embed the chunks, upsert them into the vector database. Run it again next week and it does the same thing.
That works on day one. Over time, three things break it:
| What happens at the source | What a naive pipeline does | What the chatbot then says |
|---|---|---|
| A document is edited | Adds the new chunks, keeps the old ones | Mixes the old answer and the new one |
| A document is deleted | Nothing, because it no longer sees it | Keeps quoting a policy that no longer exists |
| A document is unchanged | Re-embeds it anyway | Correct, but you paid to embed it again |
The model is not hallucinating in these cases. It is faithfully answering from what the index contains. The index is wrong.
Why can the vector database not handle this on its own?
A vector database stores vectors and metadata. It does not know what your source looks like now. To know that a document was deleted, something has to remember that it existed. That memory is state, and a plain ingestion script has none.
Adding vectors is easy. Knowing what to remove requires state. That one sentence is the whole reason the record manager exists.
How does a record manager work?
The record manager is an ordinary table, MySQL in my case, tracking what has been ingested and in what state. A useful minimum is one row per source item with its ID, a fingerprint of its content, and the IDs of the vectors it produced. Each sync run then does five things:
- Read the source on a schedule or trigger.
- Compare against the record manager and classify every item as new, changed, unchanged or removed.
- Embed only new and changed items. Unchanged items are skipped entirely.
- Update the vector database: upsert the new vectors, and delete the vectors that belong to changed or removed items.
- Update the record manager so the next run starts from an accurate baseline.
The vector store stops being an append-only pile and becomes a projection of the source.
How do you detect that a document changed?
The simplest reliable method is a hash of each item's content. If the stored hash and the new hash differ, the item changed. It is cheap, it does not depend on the source system having trustworthy "last modified" timestamps, and it catches edits that timestamps miss.
What does this save?
Two things, and both compound over time:
- Correctness. Deleted and outdated content stops being retrievable, which is usually the difference between a demo and something a support team will trust.
- Cost. Re-embedding everything on every run is simple and expensive. Change detection keeps each run proportional to what actually moved.
Is this only for Pinecone?
No. The pattern works with any vector database, including Pinecone, pgvector, Qdrant and Weaviate, and with any orchestration layer. I built mine as an n8n workflow, but it is equally at home in a Python job or an Airflow DAG. The record manager is what matters, not the tools around it.
Checklist for your own RAG pipeline
- Can you name every document currently in your index?
- If a document is deleted at the source today, when does it stop being retrievable?
- If a document is edited, are its old chunks removed?
- Does a sync run with no changes cost you zero embedding calls?
If any answer is "not sure", your chatbot is probably already serving stale answers somewhere. The workflow I built is open source on GitHub.
Frequently asked questions
- Why does my RAG chatbot return outdated information?
- Usually because the ingestion pipeline only adds to the vector index. When a source document is edited, its old chunks stay searchable next to the new ones, and when a document is deleted its chunks are never removed. The model is answering faithfully from an index that no longer matches the source.
- What is a record manager in a RAG pipeline?
- A small database table that tracks which source items have been ingested, a fingerprint of their content, and the vectors they produced. Each sync compares the source against it to classify items as new, changed, unchanged or removed, so the vector index can be updated and cleaned instead of only appended to.
- Do I need to re-embed all documents when some of them change?
- No. With change detection, such as comparing a content hash against the one stored in the record manager, only new and changed documents are embedded. Unchanged documents are skipped, so a sync with no changes costs no embedding calls.
- How do I remove deleted documents from a vector database like Pinecone?
- You need to know which vector IDs belong to the deleted document, which is exactly what a record manager stores. When a source item disappears, the sync deletes its vectors by ID and removes its row from the record manager.