All projects
16 repositories · updated 2026-11-07
Loading…
1D4Vinci / ScraplingA web scraping framework supporting individual requests and full crawls.PythonBSD-3-Clause · PermissiveInsufficient rank history98,870
2scrapy / scrapyScrapy, a fast high-level web crawling & scraping framework for Python.PythonBSD-3-Clause · PermissiveInsufficient rank history65,250
3apify / crawleeNode.js library for web scraping and browser automation in JavaScript and TypeScript.TypeScriptApache-2.0 · PermissiveInsufficient rank history26,364
4gocolly / collyElegant Scraper and Crawler Framework for GolangGoApache-2.0 · PermissiveInsufficient rank history25,578
5getmaxun / maxunTool for extracting websites into structured APIs, with search, monitoring, and automation.TypeScriptAGPL-3.0 · Share derivative sourceInsufficient rank history18,163
6codelucas / newspapernewspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:PythonMIT · PermissiveInsufficient rank history15,181
7apify / crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.PythonApache-2.0 · PermissiveInsufficient rank history9,651
8lorien / awesome-web-scrapingList of libraries, tools and APIs for web scraping and data processing.JavaScriptLicense not declaredInsufficient rank history8,213
9go-rod / rodA Chrome DevTools Protocol driver for web automation and scraping.GoMIT · PermissiveInsufficient rank history7,145
10yujiosaka / headless-chrome-crawlerDistributed crawler powered by Headless ChromeJavaScriptMIT · PermissiveInsufficient rank history5,631
11exa-labs / exa-mcp-serverExa MCP for web search and web crawling!TypeScriptMIT · PermissiveInsufficient rank history5,217
12hakluke / hakrawlerSimple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web applicationGoGPL-3.0 · Share derivative sourceInsufficient rank history5,158
13ai-robots-txt / ai.robots.txtA list of AI agents and robots to block.PythonMIT · PermissiveInsufficient rank history4,249
14hardkoded / puppeteer-sharpHeadless Chrome .NET APIC#MIT · PermissiveInsufficient rank history3,933
15edoardottt / cariddiTake a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and moreGoGPL-3.0 · Share derivative sourceInsufficient rank history3,806
16antchfx / antchAntch, a fast, powerful and extensible web crawling & scraping framework for GoGoMIT · PermissiveInsufficient rank history265
