web-scraper

There are 67 repositories under web-scraper topic.

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  awesomecrawlernode-crawlerscraperspiderweb-crawlerweb-scraper
  6281

• php-curl-class / php-curl-class

  PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

  apiapi-clientclassclientcurlframeworkhttphttp-clienthttp-proxyjsonphpphp-curlphp-curl-libraryproxyrequestsrestfulweb-scraperweb-scrapingweb-servicexml
  Language:PHP 3239

• arpit-omprakash / 100ProjectsOfCode

  A list of practical knowledge-building projects.

  ccpp11csharpjavajavascriptmusic-playerprocessingprogrammingprojectspythonsearch-engineweb-scraper
  3043

• anaskhan96 / soup

  Web Scraper in Go, similar to BeautifulSoup

  golanggowebscraperwebscrapingbeautifulsoupweb-scraperhtml-node
  Language:Go 2148
• lightnovel-crawler

  dipu-bd / lightnovel-crawler

  Generate and download e-books from online sources.

  console-appdiscordkindle-bookslightnovellightnovel-crawlerpythontelegramtermuxweb-scraper
  Language:Python 1365

• juancarlospaco / faster-than-requests

  Faster requests on Python 3

  curlcythondownload-filefaster-than-requestshigh-performancehttp-requestsndjsonopen-datapythonpython-librarypython-requestspython3requests-toolbeltrequests3scrapyspeedurlliburllib3web-scraperweb-scraping
  Language:Nim 1093

• tholian-network / stealth

  :rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy

  anonymitybrowser-automationprivacy-protectionweb-browserweb-filterweb-proxyweb-scraper
  Language:JavaScript 1016
• monkey-dl

  Oshan96 / monkey-dl

  Bulk download your favourite anime episodes from your favourite anime websites

  anime-downloaderanimeanime-searchanime-fansanime-scraperweb-scraper9animeanimeultimaanimepahe4animeanimepahe-downloaderffmpeghls-downloadermonkey-dl
  Language:Python 810

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  spiderrubyspider-linkscrawlerwebscraperweb-scrapingweb-spiderweb-crawlerweb-scraper
  Language:Ruby 798

• k0rnh0li0 / onlyfans-dl

  OnlyFans content downloader

  media-downloaderonlyfanspythonweb-scraper
  Language:Python 770
• google-maps-scraper

  gosom / google-maps-scraper

  scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

  distributed-scraperdistributed-scrapinggolanggoogle-mapsgoogle-maps-scrapingweb-scraperweb-scraping
  Language:Go 753
• web-scraping

  je-suis-tm / web-scraping

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  srapingscrapperfuturesfutures-historical-datareuterswall-street-journalbloombergpython-web-scrapernews-websitesfinancial-timesweb-scrapingnews-scraperweb-scrapernewsletterweb-scrapersoptions-datafinancial-datadata-scrapingdata-scraperwallstreetbets
  Language:Python 711

• cassidoo / scrapers

  A list of scrapers from around the web.

  listscrape-websitesscraperweb-scraper
  585

• gildas-lormeau / single-file-cli

  CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

  clidenonodejssingle-fileweb-archivingweb-scraperweb-scraping
  Language:JavaScript 554
• PHPScraper

  spekulatius / PHPScraper

  A universal web-util for PHP.

  phpphp-spiderphp-crawlerscraping-websitesphp-scraperscraperscrapingweb-scraperweb-scrapingbeautifulsoupscrapyphp-spiderspuppeteerpyppeteerchromiumheadless-chrome
  Language:PHP 511

• AlexMathew / scrapple

  A framework for creating semi-automatic web content extractors

  beautifulsoupcrawlercss-selectorextractorlxmlpythonscrapersscrapingscrapyselectorselector-expressiontutorialweb-scraperweb-scrapingxpath-expression
  Language:Python 496

• austinoboyle / scrape-linkedin-selenium

  `scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

  seleniumselenium-webdriverlinkedinscrapingweb-scraperweb-scrapingpythonscrapescraper
  Language:HTML 449
• social-media-profile-scrapers

  shaikhsajid1111 / social-media-profile-scrapers

  Fetch user's data across social media

  web-scrapingweb-scraperrequestpythonselenium-pythonfacebook-scrapertwitter-scraperpinterestreddit-scrapermedium-scrapertiktok-scraperquora-scraperinstagram-scraperpinterest-scrapperscrapping-pythonsocial-media
  Language:Python 426

• jaebradley / basketball_reference_web_scraper

  NBA Stats API via Basketball Reference

  basketball-referencenbapythonweb-scraperweb-scraping
  Language:Python 418

• crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlercrawlinghacktoberfestphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scraperweb-scraping
  Language:PHP 312
• web-unblocker

  oxylabs / web-unblocker

  Free trial Web Unblocker - an AI-powered proxy solution that can bypass even the most sophisticated anti-bot systems.

  unblockerweb-scraperweb-unblockerbypassbypasscaptchacaptchacaptcha-solvingweb-scraping-apicaptcha-breakingcaptcha-bypassamazon-captchaunblocker-websitewebiste-unblocker-githubwebsite-unlockerwebsite-unblockerunblocked-websitesrotate-captchaweb-proxyweb-proxy-serverunblocker-websites
  Language:Python 295

• PhantomInsights / summarizer

  A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

  nlpprawweb-scraperreddit-botpython3wordcloudspacy
  Language:Python 267

• duyet / awesome-web-scraper

  A collection of awesome web scaper, crawler.

  awesomeawesome-listgouttephantomjsphpscrapyslimerjsspiderstorageweb-crawlerweb-scraper
  242

• paulpierre / markdown-crawler

  A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

  html-to-markdownhtml-to-markdown-converterhtml2mdllmllmopsmarkdownmarkdown-parserragweb-scrapermarkdown-crawlermarkdown-scrapermd-crawler
  Language:Python 242

• epiqueras / getsy

  A simple browser/client-side web scraper.

  browserclient-sidescraperweb-scraper
  Language:TypeScript 240
• facebook_page_scraper

  shaikhsajid1111 / facebook_page_scraper

  Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

  facebook-scraperfacebook-pagefacebook-page-scraperfacebook-page-postweb-scrapingweb-scraperfacebookcsvpythonseleniumscraperfacebook-apissocial-mediafb-scrapperfbfacebook-page-post-scraperopen-sourcehacktoberfest
  Language:Python 211
• how-to-scrape-google-scholar

  oxylabs / how-to-scrape-google-scholar

  A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.

  google-scholargoogle-scholar-scraperpythonpython-scraperscraper-apiweb-scraperweb-scrapinggoogle-scholar-scrapper
  Language:Python 208
• google-news-scraper

  lewisdonovan / google-news-scraper

  Lightweight scraper for Google News

  google-newsgoogle-news-scrapernewsnews-scrapernews-articlesweb-scrapercrawlerweb-crawlernews-crawlergoogle-crawler
  Language:JavaScript 205

• passivebot / facebook-marketplace-scraper

  This repository contains a script to scrape Facebook Marketplace data using Playwright, BeautifulSoup and Streamlit.

  databasefacebookfacebook-marketing-automationfacebook-marketplacepythonsqlite3web-automationweb-scraperweb-scrapingplaywrightplaywright-python
  Language:Python 192
• Senpwai

  SenZmaKi / Senpwai

  A desktop app for tracking and batch downloading anime

  animeanime-downloaderanime-scraperanime-trackerautomationbatch-downloaderbeautifulsoupbulk-downloadercross-platformpyqtpyqt6-desktop-applicationpythontrackerweb-scraper
  Language:Python 159

• wikimedia / html-metadata

  MetaData html scraper and parser for Node.js (supports Promises and callback style)

  javascriptmetadata-extractionmetadata-extractornode-modulenodejsweb-scraperweb-scraping
  Language:JavaScript 138

• suntong / cascadia

  Go cascadia package command line CSS selector

  cascadiacss-selectorhtml-sourceextractcsv-tabletsvhtml-textweb-scraperweb-scrapingcommand-line-toolcommand-linecurl
  Language:Go 135

• wearrrrr / HaiKei

  HaiKei is an anime streaming website that uses the consumet API

  anime-scraperanime-streamingconsumetconsumet-apiweb-scraper
  Language:JavaScript 123

• areed1192 / python-sec

  A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.

  pythonsecsecurities-and-exchange-commissionfinanceweb-scraper
  Language:Python 116

• Fytex / Instagram-Giveaways-Winner

  Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!

  instagraminstagram-botinstagram-scraperinstagram-scriptgiveawayswinningcheatingmacrosseleniumselenium-pythonselenium-webdriverpythonpython3web-scrapercommentspostsspammentionshacktoberfesthacktoberfest2020
  Language:Python 116

• gan-of-culture / get-sauce

  A command line program to download Hentai videos and images from multiple websites

  hentai-scraperhentainsfwweb-scraper
  Language:Go 116