GitHub stars · StarDay

1firecrawl / firecrawlTools for gathering web data for AI agents.TypeScriptAGPL-3.0 · Share derivative source199,330

2unclecode / crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtfPythonApache-2.0 · Permissive87,015

3scrapy / scrapyScrapy, a fast high-level web crawling & scraping framework for Python.PythonBSD-3-Clause · Permissive65,250

4dgtlmoon / changedetection.ioAuthor description: Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!PythonApache-2.0 · Permissive35,596

5CloakHQ / CloakBrowserStealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.PythonMIT · Permissive32,769

6ScrapeGraphAI / Scrapegraph-aiPython scraper based on AIPythonMIT · Permissive32,587

7browserbase / stagehandThe SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.TypeScriptMIT · Permissive25,967

8lexmount / moliAuthor description: Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in RustRustApache-2.0 · Permissive22,832

9Evil0ctal / Douyin_TikTok_Download_API🚀 Self-hosted Douyin, TikTok, Bilibili data collection and watermark-free video download API, supports AI agent calling through MCP and offline voice-to-text, and one-click deployment with Docker. | Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.PythonApache-2.0 · Permissive21,227

10MODSetter / SurfSenseAir-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9PythonApache-2.0 · Permissive16,796

11yusufkaraaslan / Skill_SeekersConvert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detectionPythonMIT · Permissive15,291

12seleniumbase / SeleniumBase📊 APIs for Web Scraping, Browser Automation, and E2E Testing. CDP Mode bypasses bot-detection and handles CAPTCHAs. Adds stealth to Playwright. Includes an MCP server.PythonMIT · Permissive13,228

13jo-inc / camofox-browserStealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.JavaScriptMIT · Permissive12,192

14jhy / jsoupjsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.JavaMIT · Permissive11,412

15NopeCHALLC / nopecha-extensionAutomated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.MIT · Permissive11,095

16pinchtab / pinchtabHigh-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.GoMIT · Permissive10,560

17apify / crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.PythonApache-2.0 · Permissive9,657

18mherrmann / heliumLighter web automation with PythonPythonMIT · Permissive8,320

19lorien / awesome-web-scrapingList of libraries, tools and APIs for web scraping and data processing.JavaScriptLicense not declared8,213

20alirezamika / autoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for PythonPythonMIT · Permissive8,074