All projects
Repositories in this selection 53 · UTC 2026-11-04
CompareRankProjectStarFork30-day trend7-day net changeLast pushSign in to save
1—DD4Vinci / ScraplingA web scraping framework supporting individual requests and full crawls.PythonBSD-3-Clause · PermissiveInsufficient rank history92,6029,757——2026-11-02
2—Uunclecode / crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtfPythonApache-2.0 · PermissiveInsufficient rank history86,7779,043——2026-11-02
3—Sscrapy / scrapyScrapy, a fast high-level web crawling & scraping framework for Python.PythonBSD-3-Clause · PermissiveInsufficient rank history65,19012,095——2026-11-02
4—NNaiboWang / EasySpiderA visual no-code/code-free web crawler/spider: a visual browser automated testing/data collection/web crawler software that can graphically design and execute crawler tasks without code. Alias: ServiceWrapper is an intelligent service packaging system for Web applications.JavaScriptAGPL-3.0 · Share derivative sourceInsufficient rank history44,7655,393——2026-11-02
5—SScrapeGraphAI / Scrapegraph-aiPython scraper based on AIPythonMIT · PermissiveInsufficient rank history32,4373,313——2026-10-06
6—Iiawia002 / lux👾 Fast and simple video download library and CLI tool written in GoGoMIT · PermissiveInsufficient rank history31,8183,329——2026-03-29
7—Aapify / crawleeNode.js library for web scraping and browser automation in JavaScript and TypeScript.TypeScriptApache-2.0 · PermissiveInsufficient rank history26,3171,740——2026-11-02
8—Ggocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · PermissiveInsufficient rank history25,5741,860——2026-10-16
9—Jjhao104 / proxy_poolPython ProxyPool for web spiderPythonMIT · PermissiveInsufficient rank history23,7795,417——2026-06-15
10—EEvil0ctal / Douyin_TikTok_Download_API🚀 Self-hosted Douyin, TikTok, Bilibili data collection and watermark-free video download API, supports AI agent calling through MCP and offline voice-to-text, and one-click deployment with Docker. | Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.PythonApache-2.0 · PermissiveInsufficient rank history21,1122,938——2026-11-02
11—Ggetmaxun / maxunTool for extracting websites into structured APIs, with search, monitoring, and automation.TypeScriptAGPL-3.0 · Share derivative sourceInsufficient rank history18,1331,564——2026-10-28
12—Pprojectdiscovery / katanaA next-generation crawling and spidering framework.GoMIT · PermissiveInsufficient rank history17,7861,213——2026-11-02
13—Ccodelucas / newspapernewspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:PythonMIT · PermissiveInsufficient rank history15,1822,103——2026-10-26
14—Sshengqiangzhang / examples-of-web-crawlersSome very interesting python crawler examples, which are friendly to novices, mainly crawl Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ and other websites. (Some interesting examples of python crawlers that are friendly to beginners. )HTMLMIT · PermissiveInsufficient rank history14,7093,751——2025-06-28
15—Ss0md3v / PhotonA web crawler designed for open-source intelligence collection.PythonGPL-3.0 · Share derivative sourceInsufficient rank history13,3171,691——2026-10-17
16—Ccrawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · PermissiveInsufficient rank history12,2901,884——2026-02-10
17—Ccode4craft / webmagicA scalable web crawler framework for Java.JavaApache-2.0 · PermissiveInsufficient rank history11,6714,085——2025-12-20
18—Sssssssss-team / spider-flowThe new generation crawler platform defines the crawler process graphically and can complete the crawler without writing code.JavaMIT · PermissiveInsufficient rank history11,3652,167——2023-06-14
19—Iinjetlee / PythonPython script. Simulate login to Zhihu, crawler, operate excel, WeChat public account, remote bootPythonUnknown · Restricted / unknownInsufficient rank history10,8944,244——2026-05-09
20—Gguyueyingmu / avbookAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video DatabasePHPUnknown · Restricted / unknownInsufficient rank history10,1001,992——2024-06-01
License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited.