GitHub stars · StarDay

1D4Vinci / ScraplingA web scraping framework supporting individual requests and full crawls.PythonBSD-3-Clause · Permissive98,564

2unclecode / crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtfPythonApache-2.0 · Permissive86,973

3scrapy / scrapyScrapy, a fast high-level web crawling & scraping framework for Python.PythonBSD-3-Clause · Permissive65,241

4NaiboWang / EasySpiderA visual no-code/code-free web crawler/spider: a visual browser automated testing/data collection/web crawler software that can graphically design and execute crawler tasks without code. Alias: ServiceWrapper is an intelligent service packaging system for Web applications.JavaScriptAGPL-3.0 · Share derivative source44,770

5ScrapeGraphAI / Scrapegraph-aiPython scraper based on AIPythonMIT · Permissive32,566

6iawia002 / lux👾 Fast and simple video download library and CLI tool written in GoGoMIT · Permissive31,826

7apify / crawleeNode.js library for web scraping and browser automation in JavaScript and TypeScript.TypeScriptApache-2.0 · Permissive26,355

8gocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · Permissive25,578

9jhao104 / proxy_poolPython ProxyPool for web spiderPythonMIT · Permissive23,780

10Evil0ctal / Douyin_TikTok_Download_API🚀 Self-hosted Douyin, TikTok, Bilibili data collection and watermark-free video download API, supports AI agent calling through MCP and offline voice-to-text, and one-click deployment with Docker. | Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.PythonApache-2.0 · Permissive21,218

11getmaxun / maxunTool for extracting websites into structured APIs, with search, monitoring, and automation.TypeScriptAGPL-3.0 · Share derivative source18,159

12projectdiscovery / katanaA next-generation crawling and spidering framework.GoMIT · Permissive17,806

13codelucas / newspapernewspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:PythonMIT · Permissive15,181

14shengqiangzhang / examples-of-web-crawlersSome very interesting python crawler examples, which are friendly to novices, mainly crawl Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ and other websites. (Some interesting examples of python crawlers that are friendly to beginners. )HTMLMIT · Permissive14,709

15s0md3v / PhotonA web crawler designed for open-source intelligence collection.PythonGPL-3.0 · Share derivative source13,323

16crawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · Permissive12,295

17code4craft / webmagicA scalable web crawler framework for Java.JavaApache-2.0 · Permissive11,668

18ssssssss-team / spider-flowThe new generation crawler platform defines the crawler process graphically and can complete the crawler without writing code.JavaMIT · Permissive11,359

19injetlee / PythonPython script. Simulate login to Zhihu, crawler, operate excel, WeChat public account, remote bootPythonLicense not declared10,903

20guyueyingmu / avbookAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video DatabasePHPLicense not declared10,107