ai-scraping

There are 5 repositories under ai-scraping topic.

• mendableai / firecrawl

  🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

  aicrawlerdatamarkdownscraperhtml-to-markdownllmragscrapingweb-crawlerai-scrapingwebscraping
  Language:TypeScript 18891

• itsOwen / CyberScraper-2077

  A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

  ai-scrapinggemini-apillmllm-scraperopenaiscraperweb-scraperwebscraping
  Language:Python 1376

• spider-rs / spider

  A web crawler and scraper for Rust

  ai-scrapingcrawlerheadless-chromeindexerllm-crawlerrustspiderweb-crawlerweb-scraperweb-scraping
  Language:Rust 1154

• mendableai / firecrawl-app-examples

  🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

  aiai-scrapingdataexampleshtml-to-markdownllmmarkdownragscraperstemplatesweb-crawler
  Language:TypeScript 43

• ArchiveBox / abx-dl

  ⬇️ A CLI tool to download all discovered content from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git srcs, and more...

  ai-scrapingarchiveboxchromeclicli-toolcrawlingcurldownloadergallery-dlheadlesshttp-clientinternet-archivingplaywrightpuppeteerscrapingwgetyoutube-dlyt-dlp
  Language:Python 22

• spider-rs / web-crawling-guides

  How to guides on web-crawling or scraping

  agentsai-agentsai-scrapingclean-markdowncrawlerfast-webcrawlerhtml-to-markdownllm-webcrawlerscraperweb-scraping
  8

• spider-rs / spider-clients

  Python, Javascript, and Rust libraries for the Spider Cloud API.

  aiai-agentsai-scrapingcrawlerhtml-to-markdownllm-webcrawlerscraperspidersupabaseweb-scraping
  Language:Python 6