Glossary
Key terms
Definitions for the web scraping, AI, and data pipeline concepts that appear throughout the Bluefield docs.
- Adaptive polling
- An algorithm that adjusts how often a URL is checked based on how frequently it has changed in the past. Bluefield uses adaptive polling for watch mode subscriptions to reduce unnecessary requests on stable pages.
- Attestation
- A signed cryptographic record attached to live fetches (one-shot /scrape and watch mode). The manifest includes a hash of the content and a timestamp, and can be verified independently without trusting the scraping service.
- Headless browser
- A web browser that runs without a graphical interface. Used to render JavaScript-heavy pages before extracting content. Bluefield runs a managed headless browser fleet so you do not have to operate one yourself.
- Markdown
- A lightweight text markup format. Bluefield returns page content as clean markdown by default. Markdown is compact, easy to tokenize, and works well as input to LLMs and RAG pipelines.
- MCP server
- A server implementing the Model Context Protocol, which lets AI agent hosts discover and call external tools in a standard way. The Bluefield MCP server exposes scrape, map, crawl, and search as callable tools for any compatible agent.
- RAG pipeline
- Retrieval-Augmented Generation. A pattern where an LLM is given relevant documents at inference time rather than relying entirely on training data. Bluefield feeds live web content into RAG pipelines by scraping and cleaning pages on demand.
- Semantic diff
- A comparison that identifies meaningful changes in content rather than simple character-level differences. Bluefield uses a semantic diff to decide whether a page change is worth firing a webhook for.
- SSRF
- Server-Side Request Forgery. An attack where an attacker tricks a server into making HTTP requests to internal or private resources. Bluefield validates every URL against an SSRF allowlist before fetching.
- Structured extraction
- Using a large language model to pull specific fields from a page into a JSON schema you define. Bluefield performs structured extraction via the POST /extract endpoint, which accepts a URL and a JSON schema.
- Watch mode
- A Bluefield feature that monitors a URL on your behalf and sends a signed webhook to your server when the content changes meaningfully. Powered by adaptive polling and a semantic differ.
- Webhook
- An HTTP POST request sent from one server to another when a specific event occurs. Bluefield uses webhooks to notify you when a watched URL changes. Each webhook is signed with an HMAC secret so you can verify its authenticity.