web-scraper

There are 62 repositories under web-scraper topic.

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  web-crawlercrawlerweb-scraperspidernode-crawlerscraperawesome
  6077

• php-curl-class / php-curl-class

  PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

  phpcurlclassapiapi-clientclientframeworkhttphttp-clienthttp-proxyjsonphp-curlphp-curl-libraryproxyrequestsrestfulweb-scraperweb-scrapingweb-servicexml
  Language:PHP 3228

• arpit-omprakash / 100ProjectsOfCode

  A list of practical knowledge-building projects.

  programmingprojectspythonweb-scrapermusic-playersearch-enginejavajavascriptccpp11csharpprocessing
  2928

• anaskhan96 / soup

  Web Scraper in Go, similar to BeautifulSoup

  beautifulsoupgogolanghtml-nodeweb-scraperwebscraperwebscraping
  Language:Go 2125
• lightnovel-crawler

  dipu-bd / lightnovel-crawler

  Generate and download e-books from online sources.

  lightnoveltermuxweb-scraperconsole-apppythonlightnovel-crawlerdiscordtelegramkindle-books
  Language:Python 1285

• juancarlospaco / faster-than-requests

  Faster requests on Python 3

  pythonpython3python-libraryhttp-requestspython-requestsweb-scrapingdownload-filespeedcurlcythonurllibfaster-than-requestsopen-dataurllib3requests3scrapyndjsonhigh-performancerequests-toolbeltweb-scraper
  Language:Nim 1074

• tholian-network / stealth

  :rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy

  web-browserweb-scraperweb-proxyweb-filterprivacy-protectionanonymitybrowser-automation
  Language:JavaScript 990
• monkey-dl

  Oshan96 / monkey-dl

  Bulk download your favourite anime episodes from your favourite anime websites

  anime-downloaderanimeanime-searchanime-fansanime-scraperweb-scraper9animeanimeultimaanimepahe4animeanimepahe-downloaderffmpeghls-downloadermonkey-dl
  Language:Python 811

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  spiderrubyspider-linkscrawlerwebscraperweb-scrapingweb-spiderweb-crawlerweb-scraper
  Language:Ruby 792

• k0rnh0li0 / onlyfans-dl

  OnlyFans content downloader

  onlyfansmedia-downloaderpythonweb-scraper
  Language:Python 750
• web-scraping

  je-suis-tm / web-scraping

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  srapingscrapperfuturesfutures-historical-datareuterswall-street-journalbloombergpython-web-scrapernews-websitesfinancial-timesweb-scrapingnews-scraperweb-scrapernewsletterweb-scrapersoptions-datafinancial-datadata-scrapingdata-scraperwallstreetbets
  Language:Python 672
• google-maps-scraper

  gosom / google-maps-scraper

  scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

  golanggoogle-maps-scrapingweb-scraperweb-scrapingdistributed-scraperdistributed-scrapinggoogle-maps
  Language:Go 609

• cassidoo / scrapers

  A list of scrapers from around the web.

  listscrape-websitesscraperweb-scraper
  562
• PHPScraper

  spekulatius / PHPScraper

  A universal web-util for PHP.

  phpphp-spiderphp-crawlerscraping-websitesphp-scraperscraperscrapingweb-scraperweb-scrapingbeautifulsoupscrapyphp-spiderspuppeteerpyppeteerchromiumheadless-chrome
  Language:PHP 496

• AlexMathew / scrapple

  A framework for creating semi-automatic web content extractors

  pythoncss-selectorxpath-expressionweb-scraperweb-scrapingscrapersscrapingscrapyselectorextractorcrawlerselector-expressiontutoriallxmlbeautifulsoup
  Language:Python 494

• gildas-lormeau / single-file-cli

  CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

  clidenonodejssingle-fileweb-archivingweb-scraperweb-scraping
  Language:JavaScript 475

• austinoboyle / scrape-linkedin-selenium

  `scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

  linkedinpythonscrapescraperscrapingseleniumselenium-webdriverweb-scraperweb-scraping
  Language:HTML 436

• jaebradley / basketball_reference_web_scraper

  NBA Stats API via Basketball Reference

  basketball-referencepythonnbaweb-scrapingweb-scraper
  Language:HTML 403
• social-media-profile-scrapers

  shaikhsajid1111 / social-media-profile-scrapers

  Fetch user's data across social media

  web-scrapingweb-scraperrequestpythonselenium-pythonfacebook-scrapertwitter-scraperpinterestreddit-scrapermedium-scrapertiktok-scraperquora-scraperinstagram-scraperpinterest-scrapperscrapping-pythonsocial-media
  Language:Python 402

• crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlingphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scrapinghacktoberfestcrawlerweb-scraper
  Language:PHP 299

• PhantomInsights / summarizer

  A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

  nlpprawpython3reddit-botspacyweb-scraperwordcloud
  Language:Python 268

• epiqueras / getsy

  A simple browser/client-side web scraper.

  browserclient-sidescraperweb-scraper
  Language:TypeScript 239

• duyet / awesome-web-scraper

  A collection of awesome web scaper, crawler.

  web-crawlerweb-scraperslimerjsphantomjsgoutteawesomeawesome-listphpstoragescrapyspider
  236
• facebook_page_scraper

  shaikhsajid1111 / facebook_page_scraper

  Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

  facebook-scraperfacebook-pagefacebook-page-scraperfacebook-page-postweb-scrapingweb-scraperfacebookcsvpythonseleniumscraperfacebook-apissocial-mediafb-scrapperfbfacebook-page-post-scraperopen-sourcehacktoberfest
  Language:Python 194
• web-unblocker

  oxylabs / web-unblocker

  Free trial Web Unblocker - an AI-powered proxy solution that can bypass even the most sophisticated anti-bot systems

  unblockerweb-scraperweb-unblockerbypassbypasscaptchacaptchacaptcha-solvingweb-scraping-apicaptcha-breakingcaptcha-bypassamazon-captchaunblocker-websitewebiste-unblocker-githubwebsite-unlockerwebsite-unblockerunblocked-websitesrotate-captchaweb-proxyweb-proxy-server
  Language:Python 187
• google-news-scraper

  lewisdonovan / google-news-scraper

  Lightweight scraper for Google News

  google-newsgoogle-news-scrapernewsnews-scrapernews-articlesweb-scrapercrawlerweb-crawlernews-crawlergoogle-crawler
  Language:JavaScript 176

• wikimedia / html-metadata

  MetaData html scraper and parser for Node.js (supports Promises and callback style)

  javascriptnodejsnode-modulemetadata-extractionmetadata-extractorweb-scrapingweb-scraper
  Language:JavaScript 137

• passivebot / facebook-marketplace-scraper

  This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.

  databasefacebookfacebook-marketing-automationfacebook-marketplacepythonsqlite3web-automationweb-scraperweb-scrapingplaywrightplaywright-python
  Language:Python 136

• suntong / cascadia

  Go cascadia package command line CSS selector

  cascadiacss-selectorhtml-sourceextractcsv-tabletsvhtml-textweb-scraperweb-scrapingcommand-line-toolcommand-linecurl
  Language:Go 134
• Senpwai

  SenZmaKi / Senpwai

  A desktop app for tracking and batch downloading anime

  animeautomationpythonanime-downloaderseleniumbeatifulsoupweb-scraperpyqtpyqt6-desktop-applicationbatch-downloaderbulk-downloaderanime-scraperanime-trackertracker
  Language:Python 128

• wearrrrr / HaiKei

  HaiKei is an anime streaming website that uses the consumet API

  anime-scraperanime-streamingconsumetconsumet-apiweb-scraper
  Language:JavaScript 123

• areed1192 / python-sec

  A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.

  financepythonsecsecurities-and-exchange-commissionweb-scraper
  Language:Python 115

• Fytex / Instagram-Giveaways-Winner

  Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!

  instagraminstagram-botinstagram-scraperinstagram-scriptgiveawayswinningcheatingmacrosseleniumselenium-pythonselenium-webdriverpythonpython3web-scrapercommentspostsspammentionshacktoberfesthacktoberfest2020
  Language:Python 114

• paulpierre / markdown-crawler

  A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

  html-to-markdownhtml-to-markdown-converterhtml2mdllmllmopsmarkdownmarkdown-crawlermarkdown-parsermarkdown-scrapermd-crawlerragweb-scraper
  Language:Python 114

• gan-of-culture / get-sauce

  A command line program to download Hentai videos and images from multiple websites

  hentai-scraperhentainsfwweb-scraper
  Language:Go 111
• AnimeEZ

  dhvitOP / AnimeEZ

  AnimeEZ - An Anime Streaming website without any ads for free (Demo - https://animeez.live) BTW ITS MADE IN HTML

  animeanime-scraperanime-streamingno-adsstreaminganime-streaming-for-freegogoanimehtmlnodejshlsweb-scraperanime-websitescrapergogoanime-apijavascript9animekissanimeanimeez
  Language:CSS 102