website-scraper

There are 10 repositories under website-scraper topic.

• website-scraper / node-website-scraper

  Download website to local directory (including all css, images, js, etc.)

  javascriptwebsite-scraperscrapernodejshacktoberfest
  Language:JavaScript 1565

• imthaghost / goclone

  Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

  cloningcrawlergogolangwebsite-clonerwebsite-scraper
  Language:Go 1372

• josephlimtech / linkedin-profile-scraper-api

  🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

  crawlercrawlingexpressjsjsonlinkedinlinkedin-botlinkedin-crawlerlinkedin-profilelinkedin-profile-scraperlinkedin-scraperlinkedin-scrapingnodejsprofile-datapuppeteerscraperscrapersscrapingscraping-websitesspiderwebsite-scraper
  Language:TypeScript 564
• Uscrapper

  z0m31en7 / Uscrapper

  Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

  darkwebdarkweb-crawlerinformation-extractioninformation-gatheringosintosint-pythonosint-toolpythonreconnaissanceseleniumselenium-webscrapertorweb-scrapingwebcrawebcrawlerwebscrapingwebsite-scraperwebsites
  Language:Python 513

• website-scraper / website-scraper-puppeteer

  Plugin for website-scraper which returns html for dynamic websites using puppeteer

  chromechromiumhacktoberfestjavascriptnodejspuppeteerscraperwebsite-scraper
  Language:JavaScript 324
• Kooboo

  Kooboo / Kooboo

  CMS, WebSite, Application and Ecommerce Development Tool Using JavaScript

  cmsdevelopmentjavascriptkooboomagentoshopifytemplatesweb-application-platformwebsite-builderwebsite-developmentwebsite-scraperwordpress
  Language:C# 318
• html2rss-web

  html2rss / html2rss-web

  🕸 generates RSS feeds of any website a d servers to the web! Docker. Automatic scraping, use the built-in configs or create your own. Rolling release for speedy updates.

  html2rssrubydockerscraperrssfeedbuilderwebsite-scraperrss-feed-scraperhtml2rss-configsrss-feedrolling-releaserss-aggregatorfeed-configswebfeedsservesrodawebfeed
  Language:Ruby 91

• erlange / wbm-dl

  Wayback Machine Downloader. 🔥 Download your entire archived websites from the Internet Archive Wayback Machine.

  command-line-appcommand-line-parsercommand-line-toolconsoleconsole-appconsole-applicationcsharpinternetinternet-archiveinternet-wayback-machinewayback-machinewayback-machine-downloaderwebsite-scraper
  Language:C# 89

• OSINT-TECHNOLOGIES / dpulse

  DPULSE - Tool for complex approach to domain OSINT

  data-gatheringdomain-analysisinformation-gatheringinformation-securityinfosectoolsintelligenceintelligence-gatheringosintosint-toolweb-scrapinggoogle-dorkingwebscrapingwebsite-scrapercybersecuritycybersecurity-educationcybersecurity-toolosint-toolspentestpentest-toolpentesting
  Language:Python 82

• xarantolus / Collect

  A server to collect & archive websites that also supports video downloads

  self-hostedwebinterfacearchivewebsite-archivevideo-downloaderwebsite-scraperweb-archiving
  Language:TypeScript 79

• LexiestLeszek / scrapeGPT

  ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.

  crawlerhuggingfacelarge-language-modelsllmollamaproxyragretrieval-augmented-generationrobots-txtscrapertelegram-botwebsite-scraper
  Language:Python 71

• MLArtist / WebScraper

  Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

  beautiful-soupbeautifulsoupbeautifulsoup4crawlercrawling-pythoniprotationrobots-txtscraperscrapingscrapperscrapping-pythonuser-agentwebsite-crawlerwebsite-scraper
  Language:Python 70
• goClone

  shurco / goClone

  🌱 goClone - clone websites in seconds

  clonercloningcrawlercrawlinggogoclonegolanghacktoberfestscrapingscraping-websitesscrapperwebsite-clonerwebsite-scraperwp2static
  Language:Go 64

• website-scraper / node-website-scraper-phantom

  Plugin for website-scraper which returns html for dynamic websites using PhantomJS.

  hacktoberfestjavascriptnodejsphantomjsscraperwebsite-scraper
  Language:JavaScript 59

• CRAKZOR / linkedin-post-automator

  Automatically curates and posts content to LinkedIn. It can optionally use web scraping to gather data, which is then fed to ChatGPT to craft engaging LinkedIn posts.

  chatgpt-apilinkedinlinkedin-botlinkedin-postlinkedin-posts-automationlinkedin-scraperopenai-apipost-schedulerpost-schedulingwebsite-scraperchatgpt-bot
  Language:Python 57

• yuis-ice / jseval

  Evaluate JavaScript on a URL through headless Chrome browser.

  command-lineheadless-browserweb-browserbrowser-automationpupeteerheadless-browserscmdlinecommandline-interfacecli-utilitiesevalevaluatorscrapersdatascrapingscrappingdata-scrapingwebscrappingweb-crawlingscrapperwebsite-scraperweb-scrapping
  Language:JavaScript 25

• vlmaier / marvel-snap-scrapr

  Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

  marvelmarvel-charactersmarvel-snapcrawlercrawler-pythongamescraperwebsite-crawlerwebsite-scraper
  Language:Python 21

• faheel / file-extensions

  JSON collection of scraped file extensions, along with their description and type, from FileInfo.com

  file-extensionsscraperpython3scraped-datajsonfileinfowebsite-scraper
  Language:Python 18

• jeanrauwers / followers-scraper-serverless

  Now you can keep track of your followers from YouTube, Instagram and Twitter accounts - Followers scraper API on AWS serverless

  awsaws-lambdaaws-serverlessfollowers-scraperinstagraminstagram-scraperinstagramscraperlambdanodejs-lambdascrapertwittertwitter-scrapertwittersctypescriptwebscraperwebscraper-apiwebscrapingwebsite-scraperyoutube
  Language:TypeScript 18

• Ashwin-op / Email-Extractor

  A spider to crawl webpages

  spidercrawlerscrapywebsite-scraperpython
  Language:Python 16

• cometolearnofficial / WebHawk

  Website Penetration Testing Tool With Dos Attack Feature

  websitehackingpenetration-testingwebsite-scraperwebhawkcome-to-learnddos-attacktermux
  Language:Python 16

• dtflare / GPTparser

  Use GPTparser with your OpenAI API to scrape & parse files into structured JSON files.

  dataset-creationjson-modejson-parseropenai-api-chatbotwebsite-scraper
  Language:Python 12

• orangmuda / SECTOOL

  sᴇᴀʀᴄʜ ᴇɴɢɪɴᴇ sᴄʀᴀᴘᴇʀ ᴛᴏᴏʟ (ʙᴀsʜ)

  crawlercrawlingwebsite-scraperscraper
  Language:Shell 12

• website-scraper / website-scraper-existing-directory

  Plugin for website-scraper which allows to save resources to existing directory

  hacktoberfestjavascriptnodejswebsite-scraper
  Language:JavaScript 11

• dann1 / ndown

  Bandwidth efficient scheduled downloads

  aria2bandwidthschedulerwebsite-scraperwgetyoutube-downloader
  Language:Shell 10

• epegzz / node-scraper

  Scraping websites made easy! A minimalistic yet powerful tool for collecting data from websites.

  axioscheeriojavascriptnodescraperscrapingwebsite-scraper
  Language:JavaScript 9

• nigeld3v / Tumblr_Image_scrape

  Download ALL the images (JPEG/GIF/PNG) from any Tumblr website! This project employs Python3 and BeautifulSoup4 to scrape a Tumblr site (with the url provided by the user) to download, page by page, all the images from the Tumblr site's posts. Ideal for archiving other peoples' Tumblrs <3

  tumblrtumblr-image-scrapebeautifulsoupbeautifulsoup4archiveimageimagescomicswebcomicsfashionartgifgifsscraperwebsite-scrapergraphicsgraphics-librarydesignbloggingblog
  Language:Python 9

• codassassin / website-url-scraper

  This is a website url scraper built using python.

  website-scraperwebsite-scannerurl-finderurl-parser
  Language:Python 8

• methylDragon / news-anaCrawler

  Article Dataset Generator for Internet News Sites. Crawls news sites, analyses them with NLP (sentiment analysis), and pushes to a database.

  scrapingwebsite-scraperdataset-generationpython3jupyter-notebookscript
  Language:Jupyter Notebook 8

• jasniec / WebsiteParser

  Simple library which parses web pages into objects usin attributes

  parsercsharpnetstandard-librariesdotnetwebsite-scraperweb-crawler
  Language:C# 7

• SamuraiPolix / openbible-verse-scraper

  This script scrapes the verses and references from an openbible.info page into a JSON file - if needed, we use bible-api.com to translate to another bible version.

  biblebible-versebible-verse-referencesbible-versesbibledatapythonjsonwebsite-scraperbible-scraperverse-scraperbible-apibible-searchbible-search-enginebible-studybible-translationsbiblequote
  Language:Python 7

• thenurhabib / linkext

  A python Script for automatically collect links from a web page.

  scraperwebsite-scraperlink-extractorpython3automationhacking-tool
  Language:Python 7

• anaustinbeing / website-scraper

  Scrapes any website to retrieve all hyperlinks from it in a matter of seconds. Scraping made easy!

  websitewebscrapingwebscraperwebsite-scraperpython
  Language:Python 6

• Hatim315 / Manhua-Manga-Manhwa_Downloader

  This script downloads manhua, manga or manhwa and save them in a same name directory.

  image-scrapermangamanga-scrapermanhuamanhua-scrapermanhwamanhwa-scraperwebsite-scraperscraperadvanced-scraperpython3pythonpython-3python-scraperasynchronousasyncioasyncasynchronous-programming
  Language:Python 6

• Sachinart / alexa-rank-checker

  Alexa Bulk Website Rank Checker PHP Script 2020 Latest! you can grab 200+ URL's website ranking at once!

  phprankscriptseowebsite-scraperamazon-alexawebsiteranks
  Language:CSS 6

• ajaygithub2 / yellow-pages-scraper

  There is a script for scraping yellowpages.com website for name, contact, address and link

  scrapingscraperwebsite-scraperyellow-pagesyellow-pages-scraper
  Language:Python 5