GitHub radar
Scrape any website with Python — even past bot protection
Scrapling is a Python library that fetches web pages and pulls out the data you need: text, prices, links. It bypasses bot checks and keeps your login on its own, so you don't have to solve that yourself.
A ready-made set of Python tools for scraping sites: plain requests, or a hidden browser when a site needs JavaScript or blocks bots.
When it helps
Useful when you need to regularly pull data from a site protected by Cloudflare or requiring a login. Skip it if the site already offers an open API for the data.
Pros
- Passes Cloudflare's "prove you're not a bot" checks on its own
- Keeps your login and cookies between requests automatically
- Teaches your AI agent the library's current commands
- Checks scraped text for malicious instructions aimed at AI
Cons
- Setting up Agent Skill and MCP server is two extra steps beyond install
- The browser part downloads extra files on first run
How to set it up — step by step
- 1Install Scrapling with pip install scrapling inside your Python project
- 2Open your AI agent (Claude Code, Codex, or similar) in your project folder
- 3Ask the agent to set up the Agent Skill and MCP server per the instructions on Scrapling's GitHub page
- 4Send the agent the text from the block below, stating your task and the site's URL
- 5Check that the agent returned exactly the data you asked it to collect
Text for your agent
Copy this and send it to your agent — Claude Code, Codex, any of them:
Install Scrapling (pip install scrapling) and set up its Agent Skill and MCP server following https://github.com/D4Vinci/Scrapling, then scrape [I'll describe the task] from [I'll give the URL] and show me the result.
Runs on a regular laptop — just Python; the headless browser downloads what it needs on first run, no graphics card required.
Open on GitHub▌ More finds