scraping-websites

There are 53 repositories under scraping-websites topic.

• ferret

  MontFerret / ferret

  Declarative web scraping

  cdpchromeclicrawlercrawlingdata-miningdslgogolanglibraryquery-languagescraperscrapingscraping-websitestool
  Language:Go 5885

• Anorov / cloudflare-scrape

  A Python module to bypass Cloudflare's anti-bot page.

  anti-bot-pagecloudflareprotected-pagescrapescraping-websites
  Language:Python 3485

• elixir-crawly / crawly

  Crawly, a high-level web crawling & scraping framework for Elixir.

  crawlercrawlingelixirerlangextract-datascraperscrapingscraping-websitesspider
  Language:Elixir 1055

• gildas-lormeau / single-file-cli

  CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

  archivingclicrawlerdenodockerfilenodejsscraping-websitessingle-fileweb-archivingweb-crawlerweb-scraperweb-scraping
  Language:JavaScript 1032

• AmmeySaini / Edu-Mail-Generator

  Generate Free Edu Mail(s) within minutes

  auto-install-webdrivereduedu-accountedu-generatoredumailinstall-webdrivermailpythonpython3scrapingscraping-websitesseleniumselenium-pythonstudent-mail
  Language:Python 854

• josephlimtech / linkedin-profile-scraper-api

  🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

  crawlercrawlingexpressjsjsonlinkedinlinkedin-botlinkedin-crawlerlinkedin-profilelinkedin-profile-scraperlinkedin-scraperlinkedin-scrapingnodejsprofile-datapuppeteerscraperscrapersscrapingscraping-websitesspiderwebsite-scraper
  Language:TypeScript 710

• Python-World / Python_and_the_Web

  Build Bots, Scrape a website or use an API to solve a problem.

  pythonscraping-websitesapifunbotshacktoberfest
  Language:Python 694

• slotix / dataflowkit

  Extract structured data from web sites. Web sites scraping.

  golanggolang-libraryextract-datachrome-fetcherscraping-websitescrawlingscraperscrapingcdpgoheadless
  Language:Go 692

• csbun / thal

  译文：Puppeteer 与 Chrome Headless —— 从入门到爬虫

  puppeteerheadless-chromescraping-websites
  Language:JavaScript 656

• KTZgraph / sarenka

  OSINT tool - gets data from services like shodan, censys etc. in one app

  djangoreactosint-pythonreconnaissanceosintdjango-rest-frameworkshodan-apicensys-apicwescraping-websitesreactjsreact-reduxreact-persistreact-thunkdockerpython3cvecve-searchcwe-searchcve-databases
  Language:Python 653
• PHPScraper

  spekulatius / PHPScraper

  A universal web-util for PHP.

  phpphp-spiderphp-crawlerscraping-websitesphp-scraperscraperscrapingweb-scraperweb-scrapingbeautifulsoupscrapyphp-spiderspuppeteerpyppeteerchromiumheadless-chrome
  Language:PHP 572

• avidLearnerInProgress / python-automation-scripts

  Simple yet powerful automation stuffs.

  codetopdfcomic-downloaderinstagram-scrapercricket-apimedium-downloaderscraping-websitescrawlerquoraquora-crawlerimagesselenium-webdriverbeautifulsoupcricinfoinstagrampdf-converterpdfimdb-webscrappingword-of-the-daynews-scrapercrime-data-scraper
  Language:Python 557
• quick-start-guide

  oxylabs / quick-start-guide

  Python quick start guides to get the most out of Oxylabs' Web Scraper API free trial.

  scraperweb-scraperoxylabsscraper-apiscraper-pythonscrapersscrapingscraping-websitesweb-scraping
  516
• tinking

  baptisteArno / tinking

  🧶 Extract data from any website without code, just clicks.

  scrapperpuppeteerscrapingharvestingscraping-websitesscrapping
  Language:TypeScript 425

• unixfox / pupflare

  A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

  cloudflare-bypasspuppeteerkoaproxydockercloudflareanti-bot-pageprotected-pagescrapescraping-websitescloudflare-scrapechromium
  Language:JavaScript 420

• lkuffo / web-scraping

  Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup

  beautifulsoupbeautifulsoup4lxml-etreescrapingscraping-pythonscraping-websitesscrapping-pythonscrapyscrapy-crawlerscrapy-spiderseleniumselenium-pythonselenium-webdriverweb-scrapingwebscraping
  Language:Python 378
• crawler

  crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlingphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scrapinghacktoberfestcrawlerweb-scraper
  Language:PHP 366

• Go-phie / gophie

  An Aggregator Engine for searching and downloading movies free - NO ADs!

  clidownload-moviesscraping-websitesweb-apino-adsdownloaderdownload-videosfreemoviesfree-softwareanime-downloadermovie-downloaderstreammovie-sitesgophie
  Language:Go 320

• kennethreitz / requests-html

  Pythonic HTML Parsing for Humans™

  requestshtmlscrapingscraping-websitesscraping-framework
  Language:Python 320

• driscoll42 / ebayMarketAnalyzer

  Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel

  ebaypythonscraping-websiteswebscraping
  Language:Python 235

• m92vyas / llm-reader

  Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.

  extract-datallmllm-agentscraperscrapingscraping-websiteswebscrapingai-agent-toolsai-agentsai-web-scraperfirecrawljinaairag
  Language:Python 228
• Kemono-and-Coomer-Downloader

  e43b / Kemono-and-Coomer-Downloader

  The Kemono and Coomer Downloader simplifies downloading posts from Kemono and Coomer websites, allowing users to download individual or multiple posts, including entire profiles. It offers advanced features like downloading attachments, videos, and automatically organizing files.

  coomercoomer-downloadercoomer-partycoomer-sukemonokemono-downloaderkemono-partykemono-suscrapingscraping-websites
  Language:Python 222
• SpyScrap

  RuthGnz / SpyScrap

  CLI and GUI for OSINT. Are you very exhibited on the Internet? Check it! Twitter, Tinder, Facebook, Google, Yandex, BOE. It uses facial recognition to provide more accurate results.

  osintsocial-networksfacial-recognitionfacebooktwittersocmintgooglereverse-image-searchscraping-websitesboeclidockertinder
  Language:Python 201

• RyuzakiH / CloudflareSolverRe

  Cloudflare Javascript & reCaptcha challenge (I'm Under Attack Mode or IUAM) solving / bypass .NET Standard library.

  cloudflaresolverbypassprotectioncloudflaresolverlibrarydelegatinghandlerrecaptchacaptchajavascriptjschallengeutilitiesddoscsharpscraperscraping-websites
  Language:C# 193

• DiegoCaraballo / Email-extractor

  The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

  spyderextractionstractoremail-extractoremailemail-marketingemailsscraperscrapyscrapingscraping-websitesscrapperscrappingscrapy-spiderscraperspython
  Language:Python 192

• hridaydutta123 / the-youtube-scraper

  Download YouTube video description and video comments without using the YouTube API.

  youtubeyoutube-scraperscraping-websitesdataset-generation
  Language:Python 173

• Bishalsarang / Leetcode-Questions-Scraper

  Scrape Algorithm Questions from leetcode and generate html and epub file

  scrapperleetcodeleetcode-downloaderleetcode-offlineleetcode-epubleetcode-pdfleetcode-questionsleetcode-scrapperleetcode-downloadleetcode-problems-solutionsleetcode-savescraping-websitesscraperscrapingdownload-leetcode
  Language:Python 161

• johnbumgarner / newspaper3_usage_overview

  This repository provides usage examples for the Python module Newspaper3k.

  newspapernewspaper3knewsscraping-websitespythondata-extractionbeautifulsouppython-requestsnlp-parsing
  Language:Python 148

• yousefkotp / Movies-and-Series-Scraper

  A console application to scrape a valid watching links for any movie or series with exact season and episode number, you can also download a whole season with one click.

  scraperseriesseries-scrapermovie-scrapperbeautifulsoupbeautifulsoup4moviemoviespythonpython-scraperpython-scrapingpython-scriptscrapingscraping-websites
  Language:Python 138
• html2rss

  html2rss / html2rss

  📰 Build RSS 2.0 feeds from websites (and JSON APIs) automatically or with a few CSS selectors.

  extractfeedfeed-configshtmlhtml2rssjsonpuppeteerrssrss-aggregatorrss-bridgerss-builderrss-feedrss-feed-scraperrss-generatorrubyscrapescraperscrapingscraping-websitesyahoo-pipes
  Language:Ruby 134

• alash3al / scraply

  Scraply a simple dom scraper to fetch information from any html based website

  domscraperscraping-websitesscrapersgolangservercrawlercrawlingscrapy
  Language:Go 130
• spidergram

  autogram-is / spidergram

  Structural analysis tools for complex web sites

  cmscontent-auditdesign-patternsinformation-architecturescraping-websites
  Language:TypeScript 130

• pavlovtech / WebReaper

  Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

  crawlerparserwebcrawlerwebscrapingdataminingscraperparsingscrapingscraping-apiscraping-datascraping-toolscraping-webscraping-websites
  Language:C# 130

• fedecalendino / nintendeals

  Library with a set of tools for scraping information about Nintendo games and its prices across all regions (NA, EU and JP).

  redditreddit-botnintendonintendo-switchdealsmetacriticscrapingscraping-websitesnintendo-eshopgames
  Language:Python 129

• voliveirajr / seleniumcrawler

  An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

  seleniumselenium-webdriverscraperscrapingscraping-websitesscrapperasp-netpythonscrapywebcrawlerwebcrawling
  Language:Python 128

• fernandod1 / Instagram-to-discord

  Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!

  instagram-scraperinstagraminstagram-photosinstagram-downloaderdiscorddiscord-botdiscordbotdiscordappinstagram-botmonitoring-scriptsmonitormonitorsscraperscrapingscrapperscraping-websitesscraping-pythonpythonpython3webhook-discord
  Language:Python 127