scrapers

There are 11 repositories under scrapers topic.

• emcf / thepipe

  Extract markdown and images from PDFs, URLs, docs, slides, and more, ready for multimodal LLMs. ⚡

  multimodalpdfvision-transformerlarge-language-modelswebgpt-4scrapersgpt-4o
  Language:Python 849

• openstates / openstates-scrapers

  source for Open States scrapers

  governmenthacktoberfestpythonscrapersstatesunited-states
  Language:Python 845

• alltheplaces / alltheplaces

  A set of spiders and scrapers to extract location information from places that post their location on the internet.

  geojsonhacktoberfestpythonscrapersscrapyspider
  Language:Python 593

• josephlimtech / linkedin-profile-scraper-api

  🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

  puppeteernodejsscraperscrapingscraping-websiteswebsite-scraperjsonlinkedin-profile-scraperlinkedinscraperscrawlercrawlingspiderexpressjslinkedin-scraperlinkedin-scrapinglinkedin-botlinkedin-crawlerprofile-datalinkedin-profile
  Language:TypeScript 515

• AlexMathew / scrapple

  A framework for creating semi-automatic web content extractors

  beautifulsoupcrawlercss-selectorextractorlxmlpythonscrapersscrapingscrapyselectorselector-expressiontutorialweb-scraperweb-scrapingxpath-expression
  Language:Python 496

• needmorecowbell / sniff-paste

  Pastebin OSINT Harvester

  osintscrapers
  Language:Python 315

• amoudgl / short-jokes-dataset

  Python scripts for building 'Short Jokes' dataset, featured on Kaggle

  beautiful-soupdatasethumorjokesonelinerspythonscrapersweb-scraping
  Language:Python 267

• DiegoCaraballo / Email-extractor

  The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

  spyderextractionstractoremail-extractoremailemail-marketingemailsscraperscrapyscrapingscraping-websitesscrapperscrappingscrapy-spiderscraperspython
  Language:Python 177

• alash3al / scraply

  Scraply a simple dom scraper to fetch information from any html based website

  crawlercrawlingdomgolangscraperscrapersscraping-websitesscrapyserver
  Language:Go 125

• RealEstateWebTools / property_web_scraper

  Web based UI to make scraping data from real estate websites super simple

  scrapersreal-estatemlsrailsretsruby
  Language:JavaScript 87

• philshem / gmaps_popular_times_scraper

  Scraper for Google Maps "Popular Times" for place entries

  scrapergoogle-mapspython3scrapers
  Language:Python 81

• SilentDemonSD / FZBypassBot

  A Elegant Fast Multi Threaded Bypass Bot for Bigger Deeds. Try Now !!

  bypasslink-bypasserlink-shortenertelegram-botbypasscaptchabypassingscrapescrapersscraping-websites
  Language:Python 79

• OpenByteDev / SourceScraper

  Simple library which helps you to retrieve the source of various video streaming sites.

  nodejsextractorsource-extractionscraperscrapingscraping-toolnpm-packagescrapers
  Language:TypeScript 66

• shashwatah / instahunter

  CLI OSINT app that can fetch data from Instagram's Web API without authentication.

  pythoncliinstagraminstagram-scraperosintosint-toolscrapers
  Language:Python 56

• scrapehero-code / amazon-review-scraper

  A basic python 3 based web scraper for extracting reviews from Amazon. Built using Selectorlib and requests

  web-scrapingamazon-reviewsscrapersselectorlib
  Language:Python 55

• xtream1101 / scraperx

  Library for scraping websites or apis at any scale

  scraperspython-librarywebscraperpythonframeworklibrary
  Language:Python 53

• xbmc / repo-scrapers

  Official source repository for Kodi scraper add-ons

  kodiaddonscrapershacktoberfest
  52

• omardelarosa / pitchfork-npm

  An Unofficial Pitchfork Music API client for Node.js

  clijavascriptnodepitchforkpitchfork-clientscraperscrapers
  Language:JavaScript 49

• kadnan / airflow-scraping

  Using Apache Airflow to schedule web scrapers

  airflowapache-airflowpythonscrapersscheduled-tasks
  Language:Python 41

• flulemon / sneakpeek

  Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

  crawlercrawler-pythoncrawlerscrawlingcrawling-enginecrawling-frameworkpythonpython3scraperscraper-apiscraper-enginescrapersscrapingscraping-frameworkvuewebsite-crawler
  Language:Python 37

• lvxhnat / pyetfdb-scraper

  Scrape ETF data for free - the unofficial python wrapper for ETFDB

  etfetf-investmentsetfsfinancepythonscrapescraperscraperstrading
  Language:Python 36

• sup-Udh / Scraper-io

  📈 A visual & terminal based web scraper

  aiscrapersatomcollaborategithub
  Language:TypeScript 36

• asimpson / nodejs-web-scraper-cookbook

  📝 Resources for web scraping with node.js

  cheerioweb-scrapersscrapersnodejs
  30

• AgentANAKIN / Google-Web-Scraper

  This Python code scrapes Google search results then applies sentiment analysis, generates text summaries, and ranks keywords.

  googlescraperscrapingscraping-websitesscrapescraperssearch-enginesentiment-analysissentiment-classificationsentimentkeyword-extractionkeywords-extractionkeyword-extractorkeywordspythonpython3python-3lexrankluhnlsa
  Language:Python 25
• Web-Scraping-With-Selenium

  oxylabs / Web-Scraping-With-Selenium

  In this guide on how to web scrape with Selenium, we will be using Python 3. The code should work with any version of Python above 3.6

  pythonscraperscraperswebscrapingdataextractordataextractiongithub-python
  Language:Python 25

• recommend-games / board-game-scraper

  Board game data scraper

  board-gamesboardgamegeekbggbgg-ratingscrapersspiderscrapyboard-gameboardgameboardgamesdatasetdatasetsdata-setdata-setstabletop-gamesscraperboardgamegeek-datasetscraped-datapythonscraping
  Language:Python 25

• yuis-ice / jseval

  Evaluate JavaScript on a URL through headless Chrome browser.

  command-lineheadless-browserweb-browserbrowser-automationpupeteerheadless-browserscmdlinecommandline-interfacecli-utilitiesevalevaluatorscrapersdatascrapingscrappingdata-scrapingwebscrappingweb-crawlingscrapperwebsite-scraperweb-scrapping
  Language:JavaScript 25

• sallamy2580 / python-web-scrapping

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  newsletterweb-scraperweb-scrappingbloombergdata-scrapingfinancial-datafuturesfutures-historical-datanews-scrapernews-websitesreutersscrapersweb-scrapersdata-scraperfinancial-timesoptions-datapython-web-scraperscrapingwall-street-jowallstreetbets
  Language:Python 24

• crawlbase / proxycrawl-node

  ProxyCrawl Node library for scraping and crawling

  crawlercrawlingproxycrawlproxycrawl-apiscraperscrapersscrapingscraping-apiscraping-websites
  Language:JavaScript 22

• crawlbase / proxycrawl-php

  ProxyCrawl PHP library for scraping and crawling websites

  crawlercrawlingproxycrawlproxycrawl-apiscraperscrapersscrapingscraping-frameworkscraping-websites
  Language:PHP 19

• robertciotoiu / mobile-de-car-data-collector

  Crawl, scrape and persist Mobile.de car listings data in a smart & responsible way

  javajsoupscraperscrapingsitemapresponsible-scrapingcrawlercrawlingdockerjava19kubernetesparsingmavencrawlersscrapers
  Language:Java 16
• Craper

  rtunazzz / Craper

  Collection of product scrapers for various websites.

  scraperscraperssneakerbotsneakerbotssneakermonitorsneakers
  Language:Python 16

• FoxInFlame / matomari

  Unofficial MAL API.

  malmyanimelistanimescrapersapiscraper
  Language:PHP 12

• jamesbower / Reddit-Scraper

  Reddit Scraper is a Python script that utilizies the PRAW Python library to interact with Reddit's API.

  big-dataosintprawpythonredditreddit-apiscrapersscraping
  Language:Python 12

• Seng-Jik / QwQ

  Anime Wallpaper Downloader.

  gelbooru-clientdanbooru-downloaderanime-wallpapersbooru-clientscrapershentai-downloader
  Language:F# 12

• muhamed-didovic / ccdown

  Scraper and downloader for codecourse.com

  nodejsscrapercodecoursedownloadercodecourse-downloadcodecourse-downloaderscraperscodecourse-dl
  Language:JavaScript 8