web-scraper

There are 65 repositories under web-scraper topic.

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  web-crawlercrawlerweb-scraperspidernode-crawlerscraperawesome
  6142

• php-curl-class / php-curl-class

  PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

  phpcurlclassapiapi-clientclientframeworkhttphttp-clienthttp-proxyjsonphp-curlphp-curl-libraryproxyrequestsrestfulweb-scraperweb-scrapingweb-servicexml
  Language:PHP 3230

• arpit-omprakash / 100ProjectsOfCode

  A list of practical knowledge-building projects.

  programmingprojectspythonweb-scrapermusic-playersearch-enginejavajavascriptccpp11csharpprocessing
  2955

• anaskhan96 / soup

  Web Scraper in Go, similar to BeautifulSoup

  golanggowebscraperwebscrapingbeautifulsoupweb-scraperhtml-node
  Language:Go 2132
• lightnovel-crawler

  dipu-bd / lightnovel-crawler

  Generate and download e-books from online sources.

  console-appdiscordkindle-bookslightnovellightnovel-crawlerpythontelegramtermuxweb-scraper
  Language:Python 1303

• juancarlospaco / faster-than-requests

  Faster requests on Python 3

  pythonpython3python-libraryhttp-requestspython-requestsweb-scrapingdownload-filespeedcurlcythonurllibfaster-than-requestsopen-dataurllib3requests3scrapyndjsonhigh-performancerequests-toolbeltweb-scraper
  Language:Nim 1081

• tholian-network / stealth

  :rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy

  web-browserweb-scraperweb-proxyweb-filterprivacy-protectionanonymitybrowser-automation
  Language:JavaScript 996
• monkey-dl

  Oshan96 / monkey-dl

  Bulk download your favourite anime episodes from your favourite anime websites

  anime-downloaderanimeanime-searchanime-fansanime-scraperweb-scraper9animeanimeultimaanimepahe4animeanimepahe-downloaderffmpeghls-downloadermonkey-dl
  Language:Python 810

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  spiderrubyspider-linkscrawlerwebscraperweb-scrapingweb-spiderweb-crawlerweb-scraper
  Language:Ruby 793

• k0rnh0li0 / onlyfans-dl

  OnlyFans content downloader

  media-downloaderonlyfanspythonweb-scraper
  Language:Python 758
• web-scraping

  je-suis-tm / web-scraping

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  srapingscrapperfuturesfutures-historical-datareuterswall-street-journalbloombergpython-web-scrapernews-websitesfinancial-timesweb-scrapingnews-scraperweb-scrapernewsletterweb-scrapersoptions-datafinancial-datadata-scrapingdata-scraperwallstreetbets
  Language:Python 685
• google-maps-scraper

  gosom / google-maps-scraper

  scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

  golanggoogle-maps-scrapingweb-scraperweb-scrapingdistributed-scraperdistributed-scrapinggoogle-maps
  Language:Go 648

• cassidoo / scrapers

  A list of scrapers from around the web.

  listscrape-websitesscraperweb-scraper
  567
• PHPScraper

  spekulatius / PHPScraper

  A universal web-util for PHP.

  phpphp-spiderphp-crawlerscraping-websitesphp-scraperscraperscrapingweb-scraperweb-scrapingbeautifulsoupscrapyphp-spiderspuppeteerpyppeteerchromiumheadless-chrome
  Language:PHP 496

• gildas-lormeau / single-file-cli

  CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

  clidenonodejssingle-fileweb-archivingweb-scraperweb-scraping
  Language:JavaScript 495

• AlexMathew / scrapple

  A framework for creating semi-automatic web content extractors

  pythoncss-selectorxpath-expressionweb-scraperweb-scrapingscrapersscrapingscrapyselectorextractorcrawlerselector-expressiontutoriallxmlbeautifulsoup
  Language:Python 494

• austinoboyle / scrape-linkedin-selenium

  `scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

  seleniumselenium-webdriverlinkedinscrapingweb-scraperweb-scrapingpythonscrapescraper
  Language:HTML 444
• social-media-profile-scrapers

  shaikhsajid1111 / social-media-profile-scrapers

  Fetch user's data across social media

  web-scrapingweb-scraperrequestpythonselenium-pythonfacebook-scrapertwitter-scraperpinterestreddit-scrapermedium-scrapertiktok-scraperquora-scraperinstagram-scraperpinterest-scrapperscrapping-pythonsocial-media
  Language:Python 410

• jaebradley / basketball_reference_web_scraper

  NBA Stats API via Basketball Reference

  basketball-referencenbapythonweb-scraperweb-scraping
  Language:Python 407

• crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlercrawlinghacktoberfestphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scraperweb-scraping
  Language:PHP 300

• PhantomInsights / summarizer

  A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

  nlpprawweb-scraperreddit-botpython3wordcloudspacy
  Language:Python 268

• epiqueras / getsy

  A simple browser/client-side web scraper.

  browserclient-sidescraperweb-scraper
  Language:TypeScript 239

• duyet / awesome-web-scraper

  A collection of awesome web scaper, crawler.

  web-crawlerweb-scraperslimerjsphantomjsgoutteawesomeawesome-listphpstoragescrapyspider
  238
• facebook_page_scraper

  shaikhsajid1111 / facebook_page_scraper

  Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

  facebook-scraperfacebook-pagefacebook-page-scraperfacebook-page-postweb-scrapingweb-scraperfacebookcsvpythonseleniumscraperfacebook-apissocial-mediafb-scrapperfbfacebook-page-post-scraperopen-sourcehacktoberfest
  Language:Python 199
• web-unblocker

  oxylabs / web-unblocker

  Free trial Web Unblocker - an AI-powered proxy solution that can bypass even the most sophisticated anti-bot systems

  unblockerweb-scraperweb-unblockerbypassbypasscaptchacaptchacaptcha-solvingweb-scraping-apicaptcha-breakingcaptcha-bypassamazon-captchaunblocker-websitewebiste-unblocker-githubwebsite-unlockerwebsite-unblockerunblocked-websitesrotate-captchaweb-proxyweb-proxy-serverunblocker-websites
  Language:Python 190
• google-news-scraper

  lewisdonovan / google-news-scraper

  Lightweight scraper for Google News

  google-newsgoogle-news-scrapernewsnews-scrapernews-articlesweb-scrapercrawlerweb-crawlernews-crawlergoogle-crawler
  Language:JavaScript 181

• paulpierre / markdown-crawler

  A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

  html-to-markdownhtml-to-markdown-converterhtml2mdllmllmopsmarkdownmarkdown-crawlermarkdown-parsermarkdown-scrapermd-crawlerragweb-scraper
  Language:Python 170

• passivebot / facebook-marketplace-scraper

  This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.

  databasefacebookfacebook-marketing-automationfacebook-marketplacepythonsqlite3web-automationweb-scraperweb-scrapingplaywrightplaywright-python
  Language:Python 153

• wikimedia / html-metadata

  MetaData html scraper and parser for Node.js (supports Promises and callback style)

  javascriptnodejsnode-modulemetadata-extractionmetadata-extractorweb-scrapingweb-scraper
  Language:JavaScript 138

• suntong / cascadia

  Go cascadia package command line CSS selector

  cascadiacss-selectorhtml-sourceextractcsv-tabletsvhtml-textweb-scraperweb-scrapingcommand-line-toolcommand-linecurl
  Language:Go 134
• Senpwai

  SenZmaKi / Senpwai

  A desktop app for tracking and batch downloading anime

  animeautomationpythonanime-downloaderseleniumbeatifulsoupweb-scraperpyqtpyqt6-desktop-applicationbatch-downloaderbulk-downloaderanime-scraperanime-trackertracker
  Language:Python 132

• wearrrrr / HaiKei

  HaiKei is an anime streaming website that uses the consumet API

  anime-scraperanime-streamingconsumetconsumet-apiweb-scraper
  Language:JavaScript 126

• areed1192 / python-sec

  A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.

  financepythonsecsecurities-and-exchange-commissionweb-scraper
  Language:Python 116

• Fytex / Instagram-Giveaways-Winner

  Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!

  instagraminstagram-botinstagram-scraperinstagram-scriptgiveawayswinningcheatingmacrosseleniumselenium-pythonselenium-webdriverpythonpython3web-scrapercommentspostsspammentionshacktoberfesthacktoberfest2020
  Language:Python 114

• gan-of-culture / get-sauce

  A command line program to download Hentai videos and images from multiple websites

  hentai-scraperhentainsfwweb-scraper
  Language:Go 112
• AnimeEZ

  dhvitOP / AnimeEZ

  AnimeEZ - An Anime Streaming website without any ads for free (Demo - https://animeez.live) BTW ITS MADE IN HTML

  animeanime-scraperanime-streamingno-adsstreaminganime-streaming-for-freegogoanimehtmlnodejshlsweb-scraperanime-websitescrapergogoanime-apijavascript9animekissanimeanimeez
  Language:CSS 102