All projects
13 repositories · updated 2026-11-07
Loading…
1crawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · PermissiveInactiveInsufficient rank history12,296
2lining0806 / PythonSpiderNotesPython Getting Started Web Crawler Essence EditionPythonLicense not declaredInactiveInsufficient rank history7,485
3chyroc / WechatSogouWeChat public account crawler interface based on Sogou WeChat searchPythonApache-2.0 · PermissiveInsufficient rank history6,408
4DropsDevopsOrg / ECommerceCrawlersPractical combat 🐍Various website and e-commerce data crawlers🕷. Including 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, recruitment websites, Xianyu, Ali tasks, blog garden, Weibo, Baidu Tieba, Douban Movies, Baotu.com, Panorama.com, Douban Music, a provincial Food and Drug Administration, Sohu News, Machine learning text collection, fofa asset collection, Autohome, National Bureau of Statistics, Baidu keyword collection number, Spider Pan-Directory, Toutiao, Douban Film Review, Ctrip, Xiaomi App Store, Anjuke, Tujia B&B❤️❤️❤️. WeChat crawler display project:PythonMIT · PermissiveInactiveInsufficient rank history5,770
5rmax / scrapy-redisRedis-based components for Scrapy.PythonMIT · PermissiveInsufficient rank history5,647
6SpiderClub / haipproxy:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and RedisPythonMIT · PermissiveInactiveInsufficient rank history5,524
7nghuyong / WeiboSpiderContinuously maintained Sina Weibo collection tool🚀🚀🚀PythonMIT · PermissiveInsufficient rank history4,128
8Boris-code / feapder🚀🚀🚀feapder is a user-friendly, powerful crawler framework | feapder is a simple-to-use, powerful Python crawler framework. It includes four crawler tools—AirSpider, Spider, TaskSpider, and BatchSpider—to meet different needs in various scenarios. It also supports resume crawling at breakpoints, monitoring and alerts, browser rendering, and duplicate removal of large amounts of data. Additionally, the powerful feaplat crawler management system provides convenient deployment and scheduling.PythonLicense not declaredInsufficient rank history3,736
9Gerapy / GerapyDistributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.jsPythonMIT · PermissiveInsufficient rank history3,516
10my8100 / scrapydwebA web app for Scrapyd cluster management, Scrapy log analysis & visualization, auto packaging, timer tasks, monitoring & alerts, and mobile UI. Documentation: point_right:PythonGPL-3.0 · Share derivative sourceInactiveInsufficient rank history3,411
11wkunzhi / Python3-SpiderPython web scraping practices – simulating logins to various websites. Includes, but is not limited to: slider verification, Pinduoduo, Meituan, Baidu, Bilibili, Dazhongpinping, Taobao. If you like it, please start ❤️PythonLicense not declaredInactiveInsufficient rank history3,409
12RayXBuilds / Movie_RecommendHistorical learning projects: Spark movie recommendations, web scraping, and backend systems. For the latest updates, see cinematch-recommender.JavaMIT · PermissiveInsufficient rank history3,011
13tb0hdan / domainsAuthor description: World’s single largest Internet domains datasetJavaScriptBSD-3-Clause · PermissiveInactiveInsufficient rank history1,177
