GitHub stars · StarDay

1unclecode / crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtfPythonApache-2.0 · Permissive87,015

2ScrapeGraphAI / Scrapegraph-aiPython scraper based on AIPythonMIT · Permissive32,587

3lexmount / moliAuthor description: Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in RustRustApache-2.0 · Permissive22,832

4crawlab-team / crawlabDistributed web crawler admin platform for spiders management regardless of languages and frameworks. Distributed web crawler admin platform supports any languages and frameworks.GoBSD-3-Clause · Permissive12,296

5ssssssss-team / spider-flowThe new generation crawler platform defines the crawler process graphically and can complete the crawler without writing code.JavaMIT · Permissive11,359

6apify / crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.PythonApache-2.0 · Permissive9,657

7adithya-s-k / omniparseIngest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworksPythonGPL-3.0 · Share derivative source7,934

8firecrawl / firecrawl-mcp-serverOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.JavaScriptMIT · Permissive7,770

9BruceDone / awesome-crawlerA collection of awesome web crawler,spider in different languagesMIT · Permissive7,362

10KnockOutEZ / wigoloThe go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.TypeScriptLicense not declared5,532

11divinelabio / ArgusThe Ultimate Information Gathering ToolkitPythonMIT · Permissive4,306

12Anakin-Inc / anakinOpen-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make upGoAGPL-3.0 · Share derivative source3,862

130xMassi / webclawFast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.RustAGPL-3.0 · Share derivative source2,433

14firecrawl / firecrawl-app-examples🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.Jupyter NotebookLicense not declared838

15antchfx / antchAntch, a fast, powerful and extensible web crawling & scraping framework for GoGoMIT · Permissive265

16Sriram-PR / doc-scraperGo web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)GoApache-2.0 · Permissive108