Bluefield
LLM ready web scraping
Bluefield delivers LLM ready web data by returning clean markdown, structured JSON, and search results formatted for retrieval pipelines without HTML parsing on your side.
What is LLM ready web scraping?
LLM ready web scraping means page content is extracted, deduplicated, and formatted as plain text or JSON before it reaches your model or vector store.
- RAG pipeline
- Retrieval augmented generation that embeds live web chunks at inference time.
- Structured extraction
- Schema constrained JSON output via POST /scrape or POST /extract.
- Search plus scrape
- POST /search returns ranked results with optional full page content.
Bluefield capabilities
| Capability | Detail |
|---|---|
| Clean markdown | Boilerplate removed before return |
| POST /extract | Two credits per URL for bulk structured jobs |
| POST /search | One credit for the query plus per result scrapes |
| Watch mode | Monitor pages including AI answer pages for changes |
| llms.txt support | Scrape output ready for llms.txt style corpora |
Frequently asked questions
- What is the best endpoint for a RAG pipeline?
- POST /scrape with formats markdown for single pages or POST /batch/scrape for bulk ingestion.
- Can I extract a fixed JSON schema?
- Yes. Pass a schema on POST /scrape structured format or use POST /extract for bulk jobs.
- How does provenance help LLM pipelines?
- Every live fetch returns a signed attestation manifest you can store alongside chunks for audit trails. Cached responses and PDFs are unsigned in v1.