GitHub radar
Crawl4AI: turn any website into clean text for your AI
Crawl4AI is a free, open-source library that gets websites ready for AI: instead of messy HTML, you get clean text built for LLMs and agents.
It's a crawler: it opens a page in a real browser, strips out menus, footers and other clutter, and hands back clean markdown text or structured JSON data.
When it helps
Useful when your agent needs solid text from any site — for market research, tracking competitors, or feeding documents to RAG (retrieval augmented generation). Skip it if you don't work with web pages or a simple API call already does the job.
Pros
- Free and runs locally, no subscription needed
- Strips navigation and footers automatically, keeps the substance
- Pulls structured data via selectors with no AI needed
- Can disguise itself as a human on bot-blocking sites
Cons
- Sites with strong bot protection need manual proxy setup
- LLM-based parsing needs a paid model key
How to set it up — step by step
- 1Open your AI agent (Claude Code or similar) inside your project folder.
- 2Send the agent the text from the block below — it installs crawl4ai and sets up the browser.
- 3Ask the agent to crawl one test page into markdown.
- 4Check the result — the agent should show clean text with no menus or footers.
- 5If you want LLM-based parsing, give the agent a model key when it asks.
Text for your agent
Copy this and send it to your agent — Claude Code, Codex, any of them:
Install crawl4ai from https://github.com/unclecode/crawl4ai — follow the README (pip install -U crawl4ai and crawl4ai-setup), run it on one test page to Markdown and show me the result; ask me for a model API key if you go for LLM-based extraction.
Runs on a regular laptop: just Python and some disk space for the headless browser, which the crawl4ai-setup command installs for you, no GPU needed. For LLM-based extraction you need a model API key (ask me for it). On pages with strong bot protection, without tuning your own proxies, the project's paid cloud is easier, but that is no longer local.
Open on GitHub▌ More finds