scraping-websites

There are 47 repositories under scraping-websites topic.

• ferret

  MontFerret / ferret

  Declarative web scraping

  golangquery-languagedata-miningscrapingscraping-websitesdslcdpcrawlingscrapercrawlergochromeclitoollibraryhacktoberfest
  Language:Go 5661

• Anorov / cloudflare-scrape

  A Python module to bypass Cloudflare's anti-bot page.

  cloudflareanti-bot-pageprotected-pagescrapescraping-websites
  Language:Python 3327

• elixir-crawly / crawly

  Crawly, a high-level web crawling & scraping framework for Elixir.

  elixirerlangscraperscrapingscraping-websitesextract-dataspidercrawlercrawling
  Language:Elixir 940

• AmmeySaini / Edu-Mail-Generator

  Generate Free Edu Mail(s) within minutes

  seleniumpythonedumailscrapingscraping-websitesinstall-webdriverpython3edumailstudent-mailedu-accountedu-generatorselenium-pythonauto-install-webdriver
  Language:Python 797

• Python-World / Python_and_the_Web

  Build Bots, Scrape a website or use an API to solve a problem.

  pythonscraping-websitesapifunbotshacktoberfest
  Language:Python 678

• csbun / thal

  译文：Puppeteer 与 Chrome Headless —— 从入门到爬虫

  puppeteerheadless-chromescraping-websites
  Language:JavaScript 656

• slotix / dataflowkit

  Extract structured data from web sites. Web sites scraping.

  golanggolang-libraryextract-datachrome-fetcherscraping-websitescrawlingscraperscrapingcdpgoheadless
  Language:Go 651

• KTZgraph / sarenka

  OSINT tool - gets data from services like shodan, censys etc. in one app

  djangoreactosint-pythonreconnaissanceosintdjango-rest-frameworkshodan-apicensys-apicwescraping-websitesreactjsreact-reduxreact-persistreact-thunkdockerpython3cvecve-searchcwe-searchcve-databases
  Language:Python 621

• avidLearnerInProgress / python-automation-scripts

  Simple yet powerful automation stuffs.

  codetopdfcomic-downloaderinstagram-scrapercricket-apimedium-downloaderscraping-websitescrawlerquoraquora-crawlerimagesselenium-webdriverbeautifulsoupcricinfoinstagrampdf-converterpdfimdb-webscrappingword-of-the-daynews-scrapercrime-data-scraper
  Language:Python 528

• josephlimtech / linkedin-profile-scraper-api

  🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

  puppeteernodejsscraperscrapingscraping-websiteswebsite-scraperjsonlinkedin-profile-scraperlinkedinscraperscrawlercrawlingspiderexpressjslinkedin-scraperlinkedin-scrapinglinkedin-botlinkedin-crawlerprofile-datalinkedin-profile
  Language:TypeScript 511
• PHPScraper

  spekulatius / PHPScraper

  A universal web-util for PHP.

  phpphp-spiderphp-crawlerscraping-websitesphp-scraperscraperscrapingweb-scraperweb-scrapingbeautifulsoupscrapyphp-spiderspuppeteerpyppeteerchromiumheadless-chrome
  Language:PHP 505
• tinking

  baptisteArno / tinking

  🧶 Extract data from any website without code, just clicks.

  scrapperpuppeteerscrapingharvestingscraping-websitesscrapping
  Language:TypeScript 416

• unixfox / pupflare

  A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

  cloudflare-bypasspuppeteerkoaproxydockercloudflareanti-bot-pageprotected-pagescrapescraping-websitescloudflare-scrapechromium
  Language:JavaScript 341

• lkuffo / web-scraping

  Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup

  scrapingscraping-pythonscraping-websiteswebscrapingscrapping-pythonselenium-webdriverselenium-pythonlxml-etreescrapyscrapy-crawlerscrapy-spiderbeautifulsoup4beautifulsoupseleniumweb-scraping
  Language:Python 320

• crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlingphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scrapinghacktoberfestcrawlerweb-scraper
  Language:PHP 311

• Go-phie / gophie

  An Aggregator Engine for searching and downloading movies free - NO ADs!

  clidownload-moviesscraping-websitesweb-apino-adsdownloaderdownload-videosfreemoviesfree-softwareanime-downloadermovie-downloaderstreammovie-sitesgophie
  Language:Go 297

• kennethreitz / requests-html

  Pythonic HTML Parsing for Humans™

  requestshtmlscrapingscraping-websitesscraping-framework
  Language:Python 283

• driscoll42 / ebayMarketAnalyzer

  Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel

  ebayscraping-websitespythonwebscraping
  Language:Python 191

• DiegoCaraballo / Email-extractor

  The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

  spyderextractionstractoremail-extractoremailemail-marketingemailsscraperscrapyscrapingscraping-websitesscrapperscrappingscrapy-spiderscraperspython
  Language:Python 176

• RyuzakiH / CloudflareSolverRe

  Cloudflare Javascript & reCaptcha challenge (I'm Under Attack Mode or IUAM) solving / bypass .NET Standard library.

  cloudflaresolverbypassprotectioncloudflaresolverlibrarydelegatinghandlerrecaptchacaptchajavascriptjschallengeutilitiesddoscsharpscraperscraping-websites
  Language:C# 176
• SpyScrap

  RuthGnz / SpyScrap

  CLI and GUI for OSINT. Are you very exhibited on the Internet? Check it! Twitter, Tinder, Facebook, Google, Yandex, BOE. It uses facial recognition to provide more accurate results.

  osintsocial-networksfacial-recognitionfacebooktwittersocmintgooglereverse-image-searchscraping-websitesboeclidockertinder
  Language:Python 155

• Bishalsarang / Leetcode-Questions-Scraper

  Scrape Algorithm Questions from leetcode and generate html and epub file

  scrapperleetcodeleetcode-downloaderleetcode-offlineleetcode-epubleetcode-pdfleetcode-questionsleetcode-scrapperleetcode-downloadleetcode-problems-solutionsleetcode-savescraping-websitesscraperscrapingdownload-leetcode
  Language:Python 136

• johnbumgarner / newspaper3_usage_overview

  This repository provides usage examples for the Python module Newspaper3k.

  newspapernewspaper3knewsscraping-websitespythondata-extractionbeautifulsouppython-requestsnlp-parsing
  Language:Python 134

• hridaydutta123 / the-youtube-scraper

  Download YouTube video description and video comments without using the YouTube API.

  youtubeyoutube-scraperscraping-websitesdataset-generation
  Language:Python 132

• fedecalendino / nintendeals

  Library with a set of tools for scraping information about Nintendo games and its prices across all regions (NA, EU and JP).

  redditreddit-botnintendonintendo-switchdealsmetacriticscrapingscraping-websitesnintendo-eshopgames
  Language:Python 127

• voliveirajr / seleniumcrawler

  An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

  seleniumselenium-webdriverscraperscrapingscraping-websitesscrapperasp-netpythonscrapywebcrawlerwebcrawling
  Language:Python 127

• alash3al / scraply

  Scraply a simple dom scraper to fetch information from any html based website

  domscraperscraping-websitesscrapersgolangservercrawlercrawlingscrapy
  Language:Go 126

• fernandod1 / Instagram-to-discord

  Monitor instagram user account and automatically post new images to discord channel via a webhook. Working 2022!

  instagram-scraperinstagraminstagram-photosinstagram-downloaderdiscorddiscord-botdiscordbotdiscordappinstagram-botmonitoring-scriptsmonitormonitorsscraperscrapingscrapperscraping-websitesscraping-pythonpythonpython3webhook-discord
  Language:Python 123

• JusticeRage / ApkTrack

  ApkTrack is an Android app which checks if updates for installed APKs are available.

  android-appapkscraping-websitesandroid
  Language:Java 122

• yousefkotp / Movies-and-Series-Scraper

  A console application to scrape a valid watching links for any movie or series with exact season and episode number, you can also download a whole season with one click.

  scraperseriesseries-scrapermovie-scrapperbeautifulsoupbeautifulsoup4moviemoviespythonpython-scraperpython-scrapingpython-scriptscrapingscraping-websites
  Language:Python 119
• html2rss

  html2rss / html2rss

  📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors.

  rubyhtmlscraperrssfeedrss-generatorextracthtml2rssrss-feed-scraperfeed-configsrss-feedscrapingscraping-websitesscrapejsonrss-builderyahoo-pipesrss-aggregatorrss-bridgeatom-feed
  Language:Ruby 112

• Cartmanishere / zippyshare-scraper

  A module to get direct downloadable links from zippyshare download page.

  scraping-websiteszippysharedownload-linksmultithreadingzippyshare-scraper
  Language:Python 107
• spidergram

  autogram-is / spidergram

  Structural analysis tools for complex web sites

  cmscontent-auditdesign-patternsinformation-architecturescraping-websites
  Language:TypeScript 104

• pavlovtech / WebReaper

  Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

  crawlerparserwebcrawlerwebscrapingdataminingscraperparsingscrapingscraping-apiscraping-datascraping-toolscraping-webscraping-websites
  Language:C# 98

• Zhihan1996 / TradeTheEvent

  Implementation of "Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading." In Findings of ACL2021

  event-driven-tradingstock-predictionstock-price-predictiondatasetevent-detectionnewstradecorporate-eventaclstock-marketstock-analysisstock-tradingscraperscraping-websitesseleniumbert
  Language:Python 95

• alpdias / instagram-bot

  🤖 Python bot to view stories, like and comment on Instagram

  botpythoninstagramgeckodriverwebdriverwebscrapingscraping-websiteslikecommentsbot-python-instagrambot-pythonbot-instagrampython-instagramseleniumselenium-webdriverstoriesbot-instagram-pythonautomationautomation-pythonpt-br
  Language:Python 81