All projects
Repositories in this selection 28 · UTC 2026-11-03
CompareRankProjectStarFork30-day trend7-day net changeLast pushSign in to save
1—NNaiboWang / EasySpiderA visual no-code/code-free web crawler/spider: a visual browser automated testing/data collection/web crawler software that can graphically design and execute crawler tasks without code. Alias: ServiceWrapper is an intelligent service packaging system for Web applications.JavaScriptAGPL-3.0 · Share derivative sourceInsufficient rank history44,7655,393——2026-11-02
2—Ggocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · PermissiveInsufficient rank history25,5741,860——2026-10-16
3—Jjhao104 / proxy_poolPython ProxyPool for web spiderPythonMIT · PermissiveInsufficient rank history23,7795,417——2026-06-15
4—Sshengqiangzhang / examples-of-web-crawlersSome very interesting python crawler examples, which are friendly to novices, mainly crawl Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ and other websites. (Some interesting examples of python crawlers that are friendly to beginners. )HTMLMIT · PermissiveInsufficient rank history14,7093,751——2025-06-28
5—Ss0md3v / PhotonA web crawler designed for open-source intelligence collection.PythonGPL-3.0 · Share derivative sourceInsufficient rank history13,3171,691——2026-10-17
6—Ccrawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · PermissiveInsufficient rank history12,2901,884——2026-02-10
7—Sssssssss-team / spider-flowThe new generation crawler platform defines the crawler process graphically and can complete the crawler without writing code.JavaMIT · PermissiveInsufficient rank history11,3652,167——2023-06-14
8—Iihmily / DouyinLiveRecorderLive broadcast recording software that can be looped and recorded by multiple people, supports live recording on Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, shopee and other 40+ platformsPythonMIT · PermissiveInsufficient rank history11,1591,497——2026-04-01
9—Gguyueyingmu / avbookAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video DatabasePHPUnknown · Restricted / unknownInsufficient rank history10,1001,992——2024-06-01
10—Kkangvcar / InfoSpiderINFO-SPIDER is a crawler toolbox that integrates many data sources. It is designed to help users get back their data safely and quickly. The tool code is open source and the process is transparent. Supported data sources include GitHub, QQ mailbox, NetEase mailbox, Alibaba mailbox, Sina mailbox, Hotmail mailbox, Outlook mailbox, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ group, generated friend circle album, browser browsing history, 12306, Blog Park, CSDN blog, open source Chinese blog, and Jianshu.PythonGPL-3.0 · Share derivative sourceInsufficient rank history8,2681,471——2026-04-21
11—Llorien / awesome-web-scrapingList of libraries, tools and APIs for web scraping and data processing.JavaScriptNOASSERTION · Restricted / unknownInsufficient rank history8,207969——2026-10-27
12—Aandeya / pholcusPholcus is a distributed high-concurrency crawler software written in pure golangGoApache-2.0 · PermissiveInsufficient rank history7,5841,664——2026-08-24
13—BBruceDone / awesome-crawlerA collection of awesome web crawler,spider in different languagesUnknownMIT · PermissiveInsufficient rank history7,361752——2024-06-16
14—Lluyishisi / Anti-Anti-SpiderMore and more websites have anti-crawler features, some use pictures to hide key data, some use anti-human verification codes, establish anti-anti-crawler code warehouses, and improve technology by fighting against websites with different characteristics (without malice). (Welcome to submit websites that are difficult to collect) (The project is suspended due to work reasons)PythonUnknown · Restricted / unknownInsufficient rank history7,2932,123——2021-10-17
15—Bbda-research / node-crawlerWeb Crawler/Spider for NodeJS + server-side jQuery ;-)TypeScriptMIT · PermissiveInsufficient rank history6,796862——2026-06-18
16—Wwechatsync / WechatsyncA browser tool for synchronizing articles across content platforms.TypeScriptGPL-3.0 · Share derivative sourceInsufficient rank history6,5531,105——2026-10-23
17—SSpiderClub / haipproxy:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and RedisPythonMIT · PermissiveInsufficient rank history5,525890——2022-12-26
18—DDedSecInside / TorBotDark Web OSINT ToolPythonNOASSERTION · Restricted / unknownInsufficient rank history5,158779——2026-10-28
19—Ttophubs / TopListToday's hot list, an aggregation website that obtains hot headlines from major popular websites, is written in Go language, and uses multiple coroutines to asynchronously and quickly capture information. Preview: https://mo.fishGoApache-2.0 · PermissiveInsufficient rank history4,732935——2023-02-23
20—BBoris-code / feapder🚀🚀🚀feapder is a user-friendly, powerful crawler framework | feapder is a simple-to-use, powerful Python crawler framework. It includes four crawler tools—AirSpider, Spider, TaskSpider, and BatchSpider—to meet different needs in various scenarios. It also supports resume crawling at breakpoints, monitoring and alerts, browser rendering, and duplicate removal of large amounts of data. Additionally, the powerful feaplat crawler management system provides convenient deployment and scheduling.PythonNOASSERTION · Restricted / unknownInsufficient rank history3,736549——2026-08-21
License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited.