doc-scraper
Concurrent Go crawler that converts documentation sites to clean Markdown for LLM ingestion (RAG, training data), with an MCP server and incremental crawling for AI-agent workflows.
- Go
- BadgerDB
- MCP
- Open Source
Overview
A resilient web crawler built in Go for structured documentation extraction, enabling scalable ingestion pipelines for AI/LLM systems. The tool converts documentation websites into clean Markdown format, ready for RAG applications or model training. It also runs as an MCP server so AI agents can trigger crawls and detect documentation changes, and supports incremental re-crawling to keep datasets fresh.
Why I Built This
Documentation is scattered across countless sites, each with different structures. When building LLM applications, you need clean, structured data. This tool automates the extraction process while preserving the semantic structure of the content.
Technical Approach
- Go for performance and concurrency (worker pools with per-host rate limiting)
- BadgerDB for crawl state, enabling resumable crawls after interruption
- MCP server mode exposing the crawler as a Model Context Protocol tool for Claude Code, Cursor, and other AI agents
- Incremental crawling backed by a SQLite crawl history that diffs added, removed, and changed pages between runs
- Auto framework detection (Docusaurus, MkDocs, Sphinx, GitBook, ReadTheDocs) with a readability fallback
- Respects
robots.txtand sitemaps, with exponential-backoff retries and graceful shutdown
Community Adoption
Adopted by the open-source community on GitHub. Developers use it to streamline knowledge integration into ML workflows, build custom RAG systems, and prepare training datasets.