webscraper

There are 23 repositories under webscraper topic.

• anaskhan96 / soup

  Web Scraper in Go, similar to BeautifulSoup

  beautifulsoupgogolanghtml-nodeweb-scraperwebscraperwebscraping
  Language:Go 2133

• benibela / xidel

  Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

  xqueryxmlhtmljsonxpathclicommand-linehttpwebrestcss-selectorwgetcurlhttpiexmlstarletwebscraperwebscrapingscraperdatascrapingdata-processing
  Language:Pascal 652

• rootVIII / proxy_requests

  a class that uses scraped proxies to make http GET/POST requests (Python requests)

  pythonrequests-modulerequestsproxyproxy-serverproxy-listwebscrapingwebscraperwebscraper-apirecursion-problemrecursionhttp-getterhttp-gethttphttp-proxyhttp-proxy-middlewarepython3python-requestsproxy-requests
  Language:Python 388

• salimk / Rcrawler

  An R web crawler and scraper

  rrpackagecrawlerscraperwebcrawlerwebscrapingwebscraperwebscrappingcrawlers
  Language:R 349

• onepointAI / onepoint

  An AI assistant tool that integrates coding, writing, and reading functions. For better alternatives see https://monica.im/desktop

  aielectronchatgptall-in-onemacostoolkitreactwebscrapercodingreadingxiaoai-ttsgpt-35-turbo
  Language:TypeScript 297

• toby-p / rightmove_webscraper.py

  Python class to scrape data from rightmove.co.uk and return listings in a pandas DataFrame object

  rightmovewebscraperpandaspandas-dataframecsvpythonpython3data-sciencedata-analysisdata-mining
  Language:Python 240
• daath-ai-parser

  kagermanov27 / daath-ai-parser

  Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

  aiartificial-intelligenceclassificationdatasciencegpt-3htmlmachine-learningopenaiparserparser-generatorparser-librarypythonscrapertoolswebappwebscraperwebscraping
  Language:Python 221

• mehmetozkaya / DotnetCrawler

  DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

  dotnetcorecrawlercrawlingscrapingscrapyscrapy-crawlerentity-framework-coreddd-architecturecsharpwebcrawlerwebcrawlingwebscrapingwebscraperwebcrawler-htmlagilitypackhtmlagilitypack
  Language:C# 170

• bitsummation / pickaxe

  SQL Based DSL Web Scraper/Screen Scraper

  webscraper
  Language:C# 156

• dwallach1 / Stocker

  Financial Web Scraper & Sentiment Classifier

  data-sciencesentiment-analysisfinanceinvestingweb-parserwebscraper
  Language:Python 149

• CuriousLearner / GeeksForGeeksScrapper

  Scrapes g4g and creates PDF

  scrapperpdfgeeksforgeekswebscrapingwebscraperhacktoberfest
  Language:Python 142

• JesseVent / crypto

  Cryptocurrency Historical Market Data R Package

  coinmarketcapcryptocurrencieskagglecryptodatasetcryptocurrencywebscraperrpackagefinancialopen-datamarket-datacranhistorical-cryptocurrency-prices
  Language:R 141

• chuanenlin / shutterscrape

  Web scrapper for Shutterstock

  shutterstockgettyimageswebscraperscraperchromedriverseleniumbeautifulsouppython
  Language:Python 139

• hedii / php-crawler

  A php crawler that finds emails on the internets

  php-crawlercrawlerphplaravelvuejsvuewebscrapingwebscraperwebcrawler
  Language:PHP 134

• AliAkhtari78 / SpotifyScraper

  Spotify Scraper to extract all the information from spotify, download mp3 with cover of the song

  spotfiywebscrapingwebscraperspotify-downloaderspotify-scraperspotify-scrapingscrapercrawlerpythonspotify-crawlerpython3freepreview-mp3spotify-web-playeralbum-titleinfromationspotify-songs
  Language:Python 125

• ayushi7rawat / CoWin-Vaccine-Notifier

  Automated Python Script to retrieve vaccine slots availability and get notified when a slot is available.

  pythonpython3covid-19covid19covidcoronavirusvaccinationvaccine-trackervaccine-availabilitynotifierautomationjsonwebscrapingwebscrapercowincowin-alertscowin-notifiercowin-vaccine-tracker
  Language:Python 106

• brandonrobertz / autoscrape-py

  An automated, programming-free web scraper for interactive sites

  webscraperscraperdata-journalismselenium
  Language:HTML 102

• mirkoschubert / gdpr-cli

  A command line tool for checking your website for GDPR compliance.

  clicommand-line-toolwebscrapergdprdsgvometadata
  Language:JavaScript 96

• MakeYourLifeEasier / Wuxiaworld-2-eBook

  This Python script will download chapters from novels availaible on wuxiaworld.com saves then into the .epub format

  ebookebook-downloaderepubwebnovelwuxiaworldpythonpython-3-6webscraper
  Language:Python 95

• A-Wheeto / Dashboard

  A tkinter GUI collating various data

  guidashboardapiswebscraperwebscrapingtkinter
  Language:Python 94

• MichaelYochpaz / iSubRip

  A Python package for scraping and downloading subtitles from AppleTV / iTunes movie pages.

  pythonscriptitunesappletvm3u8pypi-packagesubtitleswebscraper
  Language:Python 93

• ZoranPandovski / BookingScraper

  :earth_americas: :hotel: Scrape Booking.com :hotel: :earth_americas:

  scraperweb-scrapingpython3beautifulsouprequestwebscrapingwebscraperbooking
  Language:Python 87

• giuseppegambino / Scraping-TripAdvisor-with-Python-2020

  Python implementation of web scraping of TripAdvisor with Selenium in a new 2019 website

  pythonwebscrapingwebscraperwebscraper-websitetripadvisortripadvisor-scrapertripadvisorreviewselenium
  Language:Python 86

• JonathanVusich / pcpartpicker

  This is an unofficial API for the website pcpartpicker.com.

  python3pcpartpickerwebscraperpippypi
  Language:Python 81

• RamonWill / price-comparison-project

  A webscraper for the Django Framework that compares the product prices for various UK supermarkets

  price-comparisondjangowebscraper
  Language:Python 79

• 3xploitGuy / webscrape

  A web scraper to scrape email's and phone numbers from Websites.

  bash-scriptwebscraperwebscraping
  Language:Shell 65

• Kungger-git / Jobs_LinkedIn

  Finds Jobs on LinkedIn using web-scraping

  python3webscraperdatalinkedinlinkedin-scraper
  Language:Python 65

• nunux-keeper / keeper-core-api

  Nunux Keeper core API

  content-curationcontent-managementnodejsrestful-apiwebscraperwebscraping
  Language:JavaScript 64

• iulspop / slack-web-scraper

  Puppeteer configured to scrape the posts and threads of any channel on Slack.

  puppeteerslackwebscraper
  Language:HTML 61

• daijro / SearchifyX

  Fast flashcard searcher study tool

  educationscraperwebscraperwebscrapingquizletquizizz
  Language:Python 59

• boringPpl / Linkedin-profiles-scraping

  Automatically scrape the web data of people profiles on Linkedin based on a specific search query

  pythonpython3webscrapingwebscraperwebscrappingwebscrapperwebscraping-dataseleniumselenium-webdriverbeautifulsoup4beautifulsoup
  Language:Jupyter Notebook 57
• PHUB

  EchterAlsFake / PHUB

  A lightweight API for Pornhub

  apipornhubpornhub-apipornhub-downloaderpornhub-scraperpython3webscraperdownloaderporn-downloadpornhubapipornographypythonscrapper-apixxx
  Language:Python 53

• media-centre / makenews

  MakeNews is for journalists and newsrooms. It helps you track news from web and social media in real-time.

  news-aggregatorjournalistsnews-feedtwitter-feedfacebook-feedwebscraper
  Language:JavaScript 53

• xtream1101 / scraperx

  Library for scraping websites or apis at any scale

  frameworklibrarypythonpython-libraryscraperswebscraper
  Language:Python 53

• anouarbensaad / wsvuls

  wsvuls - website vulnerability scanner detect issues [ outdated server software and insecure HTTP headers.]

  detectionissues-trackerscannertrackervulnerabilityvulnerability-scannerswebscraper
  Language:Python 52

• daijro / EssayGen

  Essay generator

  pythonbotwebscraperwebscrapingtoressay-generationessayscraper
  Language:Python 52