All projects
Repositories in this selection 16 · UTC 2026-11-04
CompareRankProjectStarFork30-day trend7-day net changeLast pushSign in to save
1—Uunclecode / crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtfPythonApache-2.0 · PermissiveInsufficient rank history86,7779,043——2026-11-02
2—SScrapeGraphAI / Scrapegraph-aiPython scraper based on AIPythonMIT · PermissiveInsufficient rank history32,4373,313——2026-10-06
3—Llexmount / moliAuthor description: Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in RustRustApache-2.0 · PermissiveInsufficient rank history22,222610——2026-11-02
4—Ccrawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · PermissiveInsufficient rank history12,2901,884——2026-02-10
5—Sssssssss-team / spider-flowThe new generation crawler platform defines the crawler process graphically and can complete the crawler without writing code.JavaMIT · PermissiveInsufficient rank history11,3652,167——2023-06-14
6—Aapify / crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.PythonApache-2.0 · PermissiveInsufficient rank history9,637820——2026-11-02
7—Aadithya-s-k / omniparseIngest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworksPythonGPL-3.0 · Share derivative sourceInsufficient rank history7,936675——2025-12-12
8—Ffirecrawl / firecrawl-mcp-serverOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.JavaScriptMIT · PermissiveInsufficient rank history7,754926——2026-11-03
9—BBruceDone / awesome-crawlerA collection of awesome web crawler,spider in different languagesUnknownMIT · PermissiveInsufficient rank history7,361752——2024-06-16
10—KKnockOutEZ / wigoloThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.TypeScriptNOASSERTION · Restricted / unknownInsufficient rank history5,524455——2026-11-02
11—Ddivinelabio / ArgusThe Ultimate Information Gathering ToolkitPythonMIT · PermissiveInsufficient rank history4,295561——2025-12-10
12—AAnakin-Inc / anakinOpen-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make upGoAGPL-3.0 · Share derivative sourceInsufficient rank history3,881164——2026-08-18
13—00xMassi / webclawFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.RustAGPL-3.0 · Share derivative sourceInsufficient rank history2,422240——2026-11-03
14—Ffirecrawl / firecrawl-app-examples🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.Jupyter NotebookUnknown · Restricted / unknownInsufficient rank history835209——2026-01-13
15—Aantchfx / antchAntch, a fast, powerful and extensible web crawling & scraping framework for GoGoMIT · PermissiveInsufficient rank history26540——2020-05-31
16—SSriram-PR / doc-scraperGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)GoApache-2.0 · PermissiveInsufficient rank history10823——2026-11-03
License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited.