Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Author description: Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Popularity
Repository stars9,63778,654Highest
7-day net stars——
Fork8206,141Highest
30-day trendInsufficient historyInsufficient history
Browse by purposeAI & agents · Developer toolsRetrieval-augmented generation
TypeFrameworks & SDKsFrameworks & SDKs
SubcategoryModel SDKsModel SDKs
PlatformNot verifiedNot verified
Cost detailsCheck current official software, hosting, model and infrastructure pricing.Check current official software, hosting, model and infrastructure pricing.
Pricing source——
First collected2026-11-032026-11-03
MetricRepository stars, not unique users or individual-skill installsRepository stars, not unique users or individual-skill installs
LanguagePythonPython
About this projectCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.Author description: Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Measured at2026-11-02 19:01 UTC2026-11-02 18:59 UTC
License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited. Ranked by repository stars, one position per repository. Stars and forks are not user counts.