All projects
Repositories in this selection 12 · UTC 2026-11-05
CompareRankProjectStarFork30-day trend7-day net changeLast pushSign in to save
1—Ccrawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · PermissiveInsufficient rank history12,2901,884——2026-02-10
2—Llining0806 / PythonSpiderNotesPython Getting Started Web Crawler Essence EditionPythonUnknown · Restricted / unknownInsufficient rank history7,4872,147——2021-06-21
3—Cchyroc / WechatSogouWeChat public account crawler interface based on Sogou WeChat searchPythonApache-2.0 · PermissiveInsufficient rank history6,4081,698——2026-10-23
4—DDropsDevopsOrg / ECommerceCrawlersPractical combat 🐍Various website and e-commerce data crawlers🕷. Including 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, recruitment websites, Xianyu, Ali tasks, blog garden, Weibo, Baidu Tieba, Douban Movies, Baotu.com, Panorama.com, Douban Music, a provincial Food and Drug Administration, Sohu News, Machine learning text collection, fofa asset collection, Autohome, National Bureau of Statistics, Baidu keyword collection number, Spider Pan-Directory, Toutiao, Douban Film Review, Ctrip, Xiaomi App Store, Anjuke, Tujia B&B❤️❤️❤️. WeChat crawler display project:PythonMIT · PermissiveInsufficient rank history5,7661,456——2024-05-22
5—Rrmax / scrapy-redisRedis-based components for Scrapy.PythonMIT · PermissiveInsufficient rank history5,6471,579——2026-10-29
6—SSpiderClub / haipproxy:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and RedisPythonMIT · PermissiveInsufficient rank history5,525890——2022-12-26
7—Nnghuyong / WeiboSpiderContinuously maintained Sina Weibo collection tool🚀🚀🚀PythonMIT · PermissiveInsufficient rank history4,130837——2026-06-30
8—BBoris-code / feapder🚀🚀🚀feapder is a user-friendly, powerful crawler framework | feapder is a simple-to-use, powerful Python crawler framework. It includes four crawler tools—AirSpider, Spider, TaskSpider, and BatchSpider—to meet different needs in various scenarios. It also supports resume crawling at breakpoints, monitoring and alerts, browser rendering, and duplicate removal of large amounts of data. Additionally, the powerful feaplat crawler management system provides convenient deployment and scheduling.PythonNOASSERTION · Restricted / unknownInsufficient rank history3,736549——2026-08-21
9—GGerapy / GerapyDistributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.jsPythonMIT · PermissiveInsufficient rank history3,515650——2026-07-04
10—Mmy8100 / scrapydwebA web app for Scrapyd cluster management, Scrapy log analysis & visualization, auto packaging, timer tasks, monitoring & alerts, and mobile UI. Documentation: point_right:PythonGPL-3.0 · Share derivative sourceInsufficient rank history3,410584——2025-02-19
11—Wwkunzhi / Python3-SpiderPython web scraping practices – simulating logins to various websites. Includes, but is not limited to: slider verification, Pinduoduo, Meituan, Baidu, Bilibili, Dazhongpinping, Taobao. If you like it, please start ❤️PythonUnknown · Restricted / unknownInsufficient rank history3,4061,031——2023-11-03
12—Ttb0hdan / domainsAuthor description: World’s single largest Internet domains datasetJavaScriptBSD-3-Clause · PermissiveInsufficient rank history1,177186——2026-05-03
License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited.