scrape

There are 20 repositories under scrape topic.

• twintproject / twint

  An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

  elasticsearchkibanaosintpythonscrapescrape-followersscrape-followingscrape-likestweeptweetstwinttwitter
  Language:Python 15660
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  scrapingscraperscrapewebscrapingcrawlerweb-scrapingaiartificial-intelligencepythonwebautomationautomationmachine-learning
  Language:Python 6082

• Anorov / cloudflare-scrape

  A Python module to bypass Cloudflare's anti-bot page.

  anti-bot-pagecloudflareprotected-pagescrapescraping-websites
  Language:Python 3334
• metascraper

  microlinkhq / metascraper

  Get unified metadata from websites using Open Graph, Microdata, RDFa, Twitter Cards, JSON-LD, HTML, and more.

  metadataparsescrape
  Language:HTML 2270

• trevorhobenshield / twitter-api-client

  Implementation of X/Twitter v1, v2, and GraphQL APIs

  apiasyncautomationbotclientscrapesearchtwittertwitter-apitwitter-bottwitter-scraperxx-apix-botx-scraper
  Language:Python 1476

• Altimis / Scweet

  A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers, user info, images...

  dowload-imagesfollowersfollowingpythonsave-imagescrapescrape-followersscrape-followingscrape-imagesscrape-likesscrape-tweetsscraperscrapingselenium-webdrivertweetstwittertwitter-scraper
  Language:Python 1006
• twikit

  d60 / twikit

  Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

  botclientpythonpython-web-scraperpython3scrapescraperscrapingsearchtweepytwittertwitter-apitwitter-bottwitter-clienttwitter-internal-apitwitter-scraperwrapperxx-api
  Language:Python 983

• markowanga / stweet

  Advanced python library to scrap Twitter (tweets, users) from unofficial API

  twitterapipythontweetsunofficialsearchcrawlscraperscrapperscrapetwintscrap-tweettwitter-apiuseruserssearchrunnertweetscrap
  Language:Python 578

• glebarez / cero

  Scrape domain names from SSL certificates of arbitrary hosts

  domain-namesreconscrapessltlswebsecurity
  Language:Go 523

• austinoboyle / scrape-linkedin-selenium

  `scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

  seleniumselenium-webdriverlinkedinscrapingweb-scraperweb-scrapingpythonscrapescraper
  Language:HTML 449
• instamancer

  ScriptSmith / instamancer

  Scrape Instagram's API with Puppeteer

  data-mininginstagraminstagram-apiinstagram-scraperpuppeteerscrape
  Language:TypeScript 398

• unixfox / pupflare

  A webpage proxy that request through Chromium (puppeteer) - can be used to bypass Cloudflare anti bot / anti ddos on any application (like curl)

  anti-bot-pagechromiumcloudflarecloudflare-bypasscloudflare-scrapedockerkoaprotected-pageproxypuppeteerscrapescraping-websites
  Language:JavaScript 345

• Anonyfox / elixir-scrape

  Scrape any website, article or RSS/Atom Feed with ease!

  scrapeelixirreadabilityrsshtmlinformation-retrievaldata-sciencescrapingfeed
  Language:Elixir 326

• ultralytics / google-images-download

  Google/Bing Images Web Downloader

  google-image-searchgoogle-images-downloadergoogle-image-downloaderbing-images-downloaderimagesdownloadscrapescraper
  Language:Python 297

• yaroslaff / nudecrawler

  Crawl telegra.ph searching for nudes!

  nudenudity-detectioncrawlcrawlerfindnsfwnsfw-recognitionnudesonlyfanspythonpython3searchspidertitstelegra-phscrapescraperscrapingweb-scrapingwebscraping
  Language:Python 276

• andrewstuart / goq

  A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library

  golangunmarshallgoqueryselectorshtmlunmarshalingunmarshallerselectorstructscrapedecoderhtml-unmarshaling
  Language:Go 257

• danieldotnl / ha-multiscrape

  Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.

  scraperrestsensorscrapingscrapehacshome-assistant-customhome-assistant
  Language:Python 253

• JaredLGillespie / proxyscrape

  Python library for retrieving free proxies (HTTP, HTTPS, SOCKS4, SOCKS5).

  proxypythonpython3scrapescraper
  Language:Python 237

• evyatarmeged / Humanoid

  Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

  botanti-botanti-bot-pagebypasshumanoidbypass-wafscrapescrapingweb-scraping
  Language:JavaScript 205

• rocketlaunchr / google-search

  scrape google search results

  apigogolanggooglescrapesearch
  Language:Go 167

• meetyan / raise

  A simple (and unofficial) GitHub Trending client that lives in your menubar.

  electrongithubgithub-trendingpm2reactscrapeweb-scrapingwebpackjavascriptjsnodejs
  Language:JavaScript 155

• essamamdani / search-result-scraper-markdown

  This project provides a powerful web scraping tool that fetches search results and converts them into Markdown format using FastAPI, SearXNG, and Browserless. It includes the capability to use proxies for web scraping and handles HTML content conversion to Markdown efficiently.

  browserlessfastapihttpxmarkdownmetasearchproxypythonrequestsscrapesearxngjinareaderaiexafirecrawlgroqopenaiexaai
  Language:Python 150

• Jimut123 / jimutmap

  API to get enormous amount of high resolution satellite images from satellites.pro quickly through multi-threading! create map your own map dataset. Bringing data to Humans.

  scrapingsatelliteimageshighresolutionapigeosatellite-datamultithreadingenormousdatasetmlscrapebeautifulsoup4fake-headerimagejimutmapareal-imagedeep-learning-datasetsegmentation-mask
  Language:Python 143

• luengwaiban / instagram-python-scraper

  A instagram scraper wrote in python. Similar to instagram-php-scraper.Usages are in example.py. Enjoy it!

  pythoninstagramscrapescraperclawerinsig
  Language:Python 127
• html2rss

  html2rss / html2rss

  📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors.

  rubyhtmlscraperrssfeedrss-generatorextracthtml2rssrss-feed-scraperfeed-configsrss-feedscrapingscraping-websitesscrapejsonrss-builderyahoo-pipesrss-aggregatorrss-bridgeatom-feed
  Language:Ruby 112

• fefit / visdom

  A library use jQuery like API for html parsing & node selecting & node mutation, suitable for web scraping and html confusion.

  rustscrapehtml-parsejquerydom-manipulationconfusecss-selector
  Language:Rust 107

• badoux / goscraper

  Golang pkg to quickly return a preview of a webpage (title/description/images)

  gogolangscraperscrapepreviewwebpagecrawlerenginescrapperimagedescriptiontitleiconpageurlwebsite
  Language:Go 105

• DrKain / scrape-youtube

  A lightning fast package to scrape YouTube search results

  youtubescrapenpmsearchplaylistvideomovienodejsdiscordbotmaintainer-wanted
  Language:JavaScript 105
• ransomware.live

  JMousqueton / ransomware.live

  🏴‍☠️💰 Another Ransomware gang tracker

  ctiencyclopedianegotiationparsepythonransomransomwarescrapescreenshotthreat-intelligencethreatintelvictim
  Language:Python 104

• serp-spider / core

  :spider: The PHP SERP Spider - A search engine scraper

  scrapscrapescrapingserp
  Language:PHP 90

• warifp / Shopee-Scrape

  Shopee Scrape is a tool that functions to collect data - the data needed, such as finding data from photos, prices, names, store locations and others.

  shopeeshopee-apiscraperscrapescraped-datascraper-enginescrape-websitesscrape-imagesmarketplaceindonesiaphpphp-librarycurlcurl-librarycurlphpcurl-functions
  Language:PHP 83

• Pringleman83 / SportsBook

  A sports data scraping and analysis tool

  pythonsportdatascrapeanalysismenujsonleaguecomparisoncomparepredictpredictionfootballfootball-datasoccer
  Language:Python 80

• sanjit-sinha / Bhadoo-Index-Scraper

  script to scrape direct download links (ddls) from google drive index.

  bhadoo-cloud-drivebhadoo-drivebhadoo-drive-indexgoogle-drive-indexindexscrapescraperbypass
  Language:Python 80

• SilentDemonSD / FZBypassBot

  A Elegant Fast Multi Threaded Bypass Bot for Bigger Deeds. Try Now !!

  bypasslink-bypasserlink-shortenertelegram-botbypasscaptchabypassingscrapescrapersscraping-websites
  Language:Python 78

• ndgigliotti / shopify-spy

  Extract structured data from Shopify websites.

  crawlerdatadata-acquisitiondata-sciencedropshippingecommercescrapescraperscrapingscrapyshopifyspider
  Language:Python 76

• tegridydev / auto-md

  convert files / GitHub repos into LLM-ready markdown.md files

  aiai-toolconvertgithubllmllm-toolsmdpythonpython-convertpython-scriptscrape
  Language:Python 67