GitHub stars · StarDay

1unclecode / crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtfPythonApache-2.0 · Permissive86,973

2NaiboWang / EasySpiderA visual no-code/code-free web crawler/spider: a visual browser automated testing/data collection/web crawler software that can graphically design and execute crawler tasks without code. Alias: ServiceWrapper is an intelligent service packaging system for Web applications.JavaScriptAGPL-3.0 · Share derivative source44,770

3iawia002 / lux👾 Fast and simple video download library and CLI tool written in GoGoMIT · Permissive31,826

4cheeriojs / cheerioThe fast, flexible, and elegant library for parsing and manipulating HTML and XML.TypeScriptMIT · Permissive30,534

5gocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · Permissive25,578

6Evil0ctal / Douyin_TikTok_Download_API🚀 Self-hosted Douyin, TikTok, Bilibili data collection and watermark-free video download API, supports AI agent calling through MCP and offline voice-to-text, and one-click deployment with Docker. | Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.PythonApache-2.0 · Permissive21,218

7codelucas / newspapernewspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:PythonMIT · Permissive15,181

8pwxcoo / chinese-xinhua:orange_book: Chinese Xinhua Dictionary Database. Including idioms, idioms, words, and Chinese characters.PythonMIT · Permissive11,760

9NopeCHALLC / nopecha-extensionAutomated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.MIT · Permissive11,096

10guyueyingmu / avbookAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video DatabasePHPLicense not declared10,107

11apify / crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.PythonApache-2.0 · Permissive9,651

12alirezamika / autoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for PythonPythonMIT · Permissive8,074

13BruceDone / awesome-crawlerA collection of awesome web crawler,spider in different languagesMIT · Permissive7,361

14go-rod / rodA Chrome DevTools Protocol driver for web automation and scraping.GoMIT · Permissive7,145

15mishushakov / llm-scraperTurn any webpage into structured data using LLMsTypeScriptMIT · Permissive6,952

16AhmadIbrahiim / Website-downloader💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.jsHTMLMIT · Permissive6,072

17yujiosaka / headless-chrome-crawlerDistributed crawler powered by Headless ChromeJavaScriptMIT · Permissive5,631

18JustAnotherArchivist / snscrapeA social networking service scraper in PythonPythonGPL-3.0 · Share derivative source5,467

19niespodd / browser-fingerprintingAnalysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?JavaScriptLicense not declared5,157

20bjesus / pipetSwiss-army tool for scraping and extracting data from online assets, made for hackersGoMIT · Permissive4,773