All projects
Repositories in this selection 16 · UTC 2026-11-04
CompareRankProjectStarFork30-day trend7-day net changeLast pushSign in to save
1—DD4Vinci / ScraplingA web scraping framework supporting individual requests and full crawls.PythonBSD-3-Clause · PermissiveInsufficient rank history92,6029,757——2026-11-02
2—Sscrapy / scrapyScrapy, a fast high-level web crawling & scraping framework for Python.PythonBSD-3-Clause · PermissiveInsufficient rank history65,19012,095——2026-11-02
3—Aapify / crawleeNode.js library for web scraping and browser automation in JavaScript and TypeScript.TypeScriptApache-2.0 · PermissiveInsufficient rank history26,3171,740——2026-11-02
4—Ggocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · PermissiveInsufficient rank history25,5741,860——2026-10-16
5—Ggetmaxun / maxunTool for extracting websites into structured APIs, with search, monitoring, and automation.TypeScriptAGPL-3.0 · Share derivative sourceInsufficient rank history18,1331,564——2026-10-28
6—Ccodelucas / newspapernewspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:PythonMIT · PermissiveInsufficient rank history15,1822,103——2026-10-26
7—Aapify / crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.PythonApache-2.0 · PermissiveInsufficient rank history9,637820——2026-11-02
8—Llorien / awesome-web-scrapingList of libraries, tools and APIs for web scraping and data processing.JavaScriptNOASSERTION · Restricted / unknownInsufficient rank history8,207969——2026-10-27
9—Ggo-rod / rodA Chrome DevTools Protocol driver for web automation and scraping.GoMIT · PermissiveInsufficient rank history7,143489——2026-08-11
10—Yyujiosaka / headless-chrome-crawlerDistributed crawler powered by Headless ChromeJavaScriptMIT · PermissiveInsufficient rank history5,632403——2023-04-29
11—Eexa-labs / exa-mcp-serverExa MCP for web search and web crawling!TypeScriptMIT · PermissiveInsufficient rank history5,209408——2026-11-03
12—Hhakluke / hakrawlerSimple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web applicationGoGPL-3.0 · Share derivative sourceInsufficient rank history5,157539——2026-08-05
13—Aai-robots-txt / ai.robots.txtA list of AI agents and robots to block.PythonMIT · PermissiveInsufficient rank history4,245197——2026-11-02
14—Hhardkoded / puppeteer-sharpHeadless Chrome .NET APIC#MIT · PermissiveInsufficient rank history3,934483——2026-10-29
15—Eedoardottt / cariddiTake a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and moreGoGPL-3.0 · Share derivative sourceInsufficient rank history3,811344——2026-10-30
16—Aantchfx / antchAntch, a fast, powerful and extensible web crawling & scraping framework for GoGoMIT · PermissiveInsufficient rank history26540——2020-05-31
License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited.