Bluefield
Data provenance for RAG
Bluefield gives RAG pipelines verifiable source provenance: each scraped chunk arrives with a signed attestation (content hash, source URL, fetch time) you store alongside the embedding. When your app cites a fact, you can prove which page it came from and that the page has not changed since.
What is data provenance for RAG?
Data provenance for RAG is a verifiable record of where each retrieved chunk originated and whether it is still current. Storing a signed attestation with every chunk lets you trace any generated answer back to a proven source.
- RAG pipeline
- Retrieval-augmented generation: embedding web chunks and retrieving them at inference time.
- Provenance manifest
- The signed record stored beside each chunk: source URL, content hash, fetch time.
- Re-verification
- Re-checking a source with watch mode so stale chunks are caught when the page changes.
Bluefield capabilities
| Capability | Detail |
|---|---|
| POST /scrape | Signed, LLM-ready markdown for each source page |
| POST /batch/scrape | Bulk ingestion with a signed manifest per URL |
| Store with the chunk | Keep the manifest next to the embedding for traceability |
| Watch mode | Get a fresh signed capture when a source page changes |
| npx bluefields verify | Prove a chunk’s source offline at audit time |
Frequently asked questions
- How do I attach provenance to a chunk?
- Scrape with Bluefield, store the returned `attestation` manifest in the same record as the embedding and text, and you have a verifiable link from every chunk back to its source page and fetch time.
- Why does my RAG app need provenance?
- When the model cites a fact, you may need to show where it came from and that it was current. Without provenance, a retrieved chunk is an unverifiable string; with it, you can prove the source.
- What happens when the source page changes?
- Put the URL in watch mode. When the page meaningfully changes, you get a webhook with a new signed capture, so you can re-embed and your provenance stays accurate.