AI Record Manager & Vector Database Sync
RAG / Vector Databases / Data Engineering
A workflow that synchronises external content with a Pinecone vector database while maintaining a record manager in MySQL, so embeddings reflect the current state of the source rather than drifting out of date.
- Year
- 2025
- Role
- Automation & data engineer
- Source
- Public repository
Technologies
- n8n
- Pinecone
- MySQL
- Embeddings
- RAG
- Vector Search
The problem
A RAG system is only as current as its index. Source content changes, gets deleted, gets re-added — and a vector store with no memory of what it already ingested will happily serve stale or duplicated answers.
The solution
A record manager in MySQL tracks what has been ingested and in what state, so each sync run knows what is new, what changed and what should be removed from Pinecone. The vector store becomes a projection of the source rather than an append-only pile.
Architecture
How it is put together.
Source read
External content is read on a schedule or trigger.
Change detection
The MySQL record manager is compared against the source to classify each item as new, changed, unchanged or removed.
Embedding
New and changed content is chunked and embedded.
Pinecone upsert
Vectors are upserted, and vectors for removed content are deleted.
Record update
The record manager is updated so the next run starts from an accurate baseline.
Key features
What it actually does.
- MySQL record manager as the source of ingestion truth
- New / changed / unchanged / removed classification per run
- Incremental embedding instead of full re-index
- Deletion handling so removed content stops being retrievable
Engineering challenges
The parts that were hard.
Deletes are the hard part
Adding vectors is easy; knowing what to remove requires state. The record manager exists precisely to answer that.
Avoiding wasteful re-embedding
Re-embedding everything is simple and expensive. Change detection keeps each run proportional to what actually moved.
Outcomes
What came out of it.
Qualitative outcomes only — no invented benchmarks or metrics.
An index that stays consistent with its source
Incremental syncs rather than full rebuilds
A reusable pattern for any RAG ingestion pipeline
Next project
AI Email Response & CRM Management
An AI agent reads incoming email, works out intent, and drives the CRM accordingly.
Let’s build
Want something like this for your business?
Tell me the problem and I will come back with an architecture, a scope and a timeline.