GitHub stars · StarDay

1NaiboWang / EasySpiderA visual no-code/code-free web crawler/spider: a visual browser automated testing/data collection/web crawler software that can graphically design and execute crawler tasks without code. Alias: ServiceWrapper is an intelligent service packaging system for Web applications.JavaScriptAGPL-3.0 · Share derivative source44,768

2gocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · Permissive25,578

3jhao104 / proxy_poolPython ProxyPool for web spiderPythonMIT · Permissive23,776

4shengqiangzhang / examples-of-web-crawlersSome very interesting python crawler examples, which are friendly to novices, mainly crawl Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ and other websites. (Some interesting examples of python crawlers that are friendly to beginners. )HTMLMIT · Permissive14,709

5s0md3v / PhotonA web crawler designed for open-source intelligence collection.PythonGPL-3.0 · Share derivative source13,327

6crawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · Permissive12,296

7ssssssss-team / spider-flowThe new generation crawler platform defines the crawler process graphically and can complete the crawler without writing code.JavaMIT · Permissive11,359

8ihmily / DouyinLiveRecorderLive broadcast recording software that can be looped and recorded by multiple people, supports live recording on Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, shopee and other 40+ platformsPythonMIT · Permissive11,180

9guyueyingmu / avbookAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video DatabasePHPLicense not declared10,109

10kangvcar / InfoSpiderINFO-SPIDER is a crawler toolbox that integrates many data sources. It is designed to help users get back their data safely and quickly. The tool code is open source and the process is transparent. Supported data sources include GitHub, QQ mailbox, NetEase mailbox, Alibaba mailbox, Sina mailbox, Hotmail mailbox, Outlook mailbox, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ group, generated friend circle album, browser browsing history, 12306, Blog Park, CSDN blog, open source Chinese blog, and Jianshu.PythonGPL-3.0 · Share derivative source8,268

11lorien / awesome-web-scrapingList of libraries, tools and APIs for web scraping and data processing.JavaScriptLicense not declared8,213

12andeya / pholcusPholcus is a distributed high-concurrency crawler software written in pure golangGoApache-2.0 · Permissive7,584

13BruceDone / awesome-crawlerA collection of awesome web crawler,spider in different languagesMIT · Permissive7,362

14luyishisi / Anti-Anti-SpiderMore and more websites have anti-crawler features, some use pictures to hide key data, some use anti-human verification codes, establish anti-anti-crawler code warehouses, and improve technology by fighting against websites with different characteristics (without malice). (Welcome to submit websites that are difficult to collect) (The project is suspended due to work reasons)PythonLicense not declared7,293

15bda-research / node-crawlerWeb Crawler/Spider for NodeJS + server-side jQuery ;-)TypeScriptMIT · Permissive6,795

16wechatsync / WechatsyncA browser tool for synchronizing articles across content platforms.TypeScriptGPL-3.0 · Share derivative source6,575

17SpiderClub / haipproxy:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and RedisPythonMIT · Permissive5,524

18DedSecInside / TorBotDark Web OSINT ToolPythonLicense not declared5,165

19tophubs / TopListToday's hot list, an aggregation website that obtains hot headlines from major popular websites, is written in Go language, and uses multiple coroutines to asynchronously and quickly capture information. Preview: https://mo.fishGoApache-2.0 · Permissive4,733

20Boris-code / feapder🚀🚀🚀feapder is a user-friendly, powerful crawler framework | feapder is a simple-to-use, powerful Python crawler framework. It includes four crawler tools—AirSpider, Spider, TaskSpider, and BatchSpider—to meet different needs in various scenarios. It also supports resume crawling at breakpoints, monitoring and alerts, browser rendering, and duplicate removal of large amounts of data. Additionally, the powerful feaplat crawler management system provides convenient deployment and scheduling.PythonLicense not declared3,736