scrape

There are 19 repositories under scrape topic.

• twintproject / twint

  An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

  elasticsearchkibanaosintpythonscrapescrape-followersscrape-followingscrape-likestweeptweetstwinttwitter
  Language:Python 15537
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  scrapingscraperscrapewebscrapingcrawlerweb-scrapingaiartificial-intelligencepythonwebautomationautomationmachine-learning
  Language:Python 5931

• Anorov / cloudflare-scrape

  A Python module to bypass Cloudflare's anti-bot page.

  cloudflareanti-bot-pageprotected-pagescrapescraping-websites
  Language:Python 3290
• metascraper

  microlinkhq / metascraper

  Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

  metadatascrapeparse
  Language:HTML 2230

• trevorhobenshield / twitter-api-client

  Implementation of X/Twitter v1, v2, and GraphQL APIs

  apiautomationclientscrapetwitterasyncbotsearchtwitter-apixx-apitwitter-bottwitter-scraperx-botx-scraper
  Language:Python 1327

• Altimis / Scweet

  A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers, user info, images...

  selenium-webdriverscraperscrapingtwittertweetspythonfollowingfollowerstwitter-scraperscrapescrape-followersscrape-followingscrape-likesscrape-imagesscrape-tweetssave-imagedowload-images
  Language:Python 966
• twikit

  d60 / twikit

  Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

  api-wrapperfreetwittertwitter-apiwrapperxpythonpython3botclienttwitter-bottwitter-clientsearchscrapertwitter-scraperscrapescrapingtwitter-scraper-2023twitter-internal-apix-api
  Language:Python 599

• markowanga / stweet

  Advanced python library to scrap Twitter (tweets, users) from unofficial API

  apicrawlpythonscrapscrap-tweetscrapescraperscrappersearchsearchrunnertweettweetstwinttwittertwitter-apiunofficialuserusers
  Language:Python 568

• glebarez / cero

  Scrape domain names from SSL certificates of arbitrary hosts

  reconwebsecurityssltlsdomain-namesscrape
  Language:Go 502

• austinoboyle / scrape-linkedin-selenium

  `scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

  seleniumselenium-webdriverlinkedinscrapingweb-scraperweb-scrapingpythonscrapescraper
  Language:HTML 436
• instamancer

  ScriptSmith / instamancer

  Scrape Instagram's API with Puppeteer

  puppeteerinstagraminstagram-apiinstagram-scraperdata-miningscrape
  Language:TypeScript 391

• Anonyfox / elixir-scrape

  Scrape any website, article or RSS/Atom Feed with ease!

  scrapeelixirreadabilityrsshtmlinformation-retrievaldata-sciencescrapingfeed
  Language:Elixir 326

• unixfox / pupflare

  A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

  anti-bot-pagechromiumcloudflarecloudflare-bypasscloudflare-scrapedockerkoaprotected-pageproxypuppeteerscrapescraping-websites
  Language:JavaScript 314

• ultralytics / google-images-download

  Google/Bing Images Web Downloader

  google-image-searchgoogle-images-downloadergoogle-image-downloaderbing-images-downloaderimagesdownloadscrapescraper
  Language:Python 287

• yaroslaff / nudecrawler

  Crawl telegra.ph searching for nudes!

  nudenudity-detectioncrawlcrawlerfindnsfwnsfw-recognitionnudesonlyfanspythonpython3searchspidertitstelegra-phscrapescraperscrapingweb-scrapingwebscraping
  Language:Python 260

• andrewstuart / goq

  A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library

  golangunmarshallgoqueryselectorshtmlunmarshalingunmarshallerselectorstructscrapedecoderhtml-unmarshaling
  Language:Go 251

• JaredLGillespie / proxyscrape

  Python library for retrieving free proxies (HTTP, HTTPS, SOCKS4, SOCKS5).

  pythonproxyscrapescraperpython3
  Language:Python 231

• danieldotnl / ha-multiscrape

  Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

  scraperrestsensorscrapingscrapehacshome-assistant-customhome-assistant
  Language:Python 218

• evyatarmeged / Humanoid

  Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

  botanti-botanti-bot-pagebypasshumanoidbypass-wafscrapescrapingweb-scraping
  Language:JavaScript 197

• rocketlaunchr / google-search

  scrape google search results

  googlegogolangsearchapiscrape
  Language:Go 164

• meetyan / raise

  A simple (and unofficial) GitHub Trending client that lives in your menubar.

  electrongithubgithub-trendingpm2reactscrapeweb-scrapingwebpackjavascriptjsnodejs
  Language:JavaScript 155

• Jimut123 / jimutmap

  API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.

  scrapingsatelliteimageshighresolutionapigeosatellite-datamultithreadingenormousdatasetmlscrapebeautifulsoup4fake-headerimagejimutmapareal-imagedeep-learning-datasetsegmentation-mask
  Language:Python 135

• luengwaiban / instagram-python-scraper

  A instagram scraper wrote in python. Similar to instagram-php-scraper.Usages are in example.py. Enjoy it!

  pythoninstagramscrapescraperclawerinsig
  Language:Python 126
• html2rss

  html2rss / html2rss

  📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors.

  atom-feedextractfeedfeed-configshtmlhtml2rssjsonrssrss-aggregatorrss-bridgerss-builderrss-feedrss-feed-scraperrss-generatorrubyscrapescraperscrapingscraping-websitesyahoo-pipes
  Language:Ruby 111

• badoux / goscraper

  Golang pkg to quickly return a preview of a webpage (title/description/images)

  gogolangscraperscrapepreviewwebpagecrawlerenginescrapperimagedescriptiontitleiconpageurlwebsite
  Language:Go 104

• fefit / visdom

  A library use jQuery like API for html parsing & node selecting & node mutation, suitable for web scraping and html confusion.

  rustscrapehtml-parsejquerydom-manipulationconfusecss-selector
  Language:Rust 102

• DrKain / scrape-youtube

  A lightning fast package to scrape YouTube search results

  youtubescrapenpmsearchplaylistvideomovienodejsdiscordbotmaintainer-wanted
  Language:JavaScript 101

• serp-spider / core

  :spider: The PHP SERP Spider - A search engine scraper

  serpscrapscrapescraping
  Language:PHP 88

• Pringleman83 / SportsBook

  A sports data scraping and analysis tool

  pythonsportdatascrapeanalysismenujsonleaguecomparisoncomparepredictpredictionfootballfootball-datasoccer
  Language:Python 80

• warifp / Shopee-Scrape

  Shopee Scrape is a tool that functions to collect data - the data needed, such as finding data from photos, prices, names, store locations and others.

  shopeeshopee-apiscraperscrapescraped-datascraper-enginescrape-websitesscrape-imagesmarketplaceindonesiaphpphp-librarycurlcurl-librarycurlphpcurl-functions
  Language:PHP 77
• ransomware.live

  JMousqueton / ransomware.live

  🏴‍☠️💰 Another Ransomware gang tracker

  parseransomwarescraperansomnegotiationpythonscreenshotvictimencyclopediactithreat-intelligencethreatintel
  Language:Python 75

• sanjit-sinha / Bhadoo-Index-Scraper

  script to scrape direct download links (ddls) from google drive index.

  bhadoo-cloud-drivebhadoo-drivebhadoo-drive-indexgoogle-drive-indexindexscrapescraperbypass
  Language:Python 75

• ndgigliotti / shopify-spy

  Extract structured data from Shopify websites.

  spidercrawlerscrapyscrapershopifyecommercedropshippingscrapingscrapedatadata-sciencedata-acquisition
  Language:Python 68

• lapwat / reCatchable

  Turn a site into a book. Download a whole website and upload it to your reMarkable.

  remarkable-tabletremarkablescrapeebookepubremarkable-tabletsscraper
  Language:JavaScript 65

• SilentDemonSD / FZBypassBot

  A Elegant Fast Multi Threaded Bypass Bot for Bigger Deeds. Try Now !!

  bypasslink-bypasserlink-shortenertelegram-botbypasscaptchabypassingscrapescrapersscraping-websites
  Language:Python 64

• drudge / n8n-nodes-puppeteer

  n8n node for requesting webpages using Puppeteer

  browserchromiumn8nn8n-nodespuppeteerscrapescreenshotstealth-modepdfproxy-server
  Language:TypeScript 55