scraping-tool

There are 7 repositories under scraping-tool topic.

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 6357
• botasaurus

  omkarcloud / botasaurus

  The All in One Framework to build Awesome Scrapers.

  anti-botanti-detectioncloudflare-bypasscloudflare-scrapeanti-detectanti-detect-browserantidetect-browserundetectedundetected-chromedriverbypass-cloudflarepython-web-scraperpython-web-scrapingscraping-frameworkscraping-toolundetectableweb-scraping-pythonbot-detectionscraping-pythonweb-crawlingpython-scraper
  Language:Python 956

• ispras / web-scraper-chrome-extension

  Web data extraction tool implemented as chrome extension

  javascriptscrapingscraping-toolwebscraping
  Language:JavaScript 196

• pavlovtech / WebReaper

  Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

  crawlerparserwebcrawlerwebscrapingdataminingscraperparsingscrapingscraping-apiscraping-datascraping-toolscraping-webscraping-websites
  Language:C# 94

• OpenByteDev / SourceScraper

  Simple library which helps you to retrieve the source of various video streaming sites.

  nodejsextractorsource-extractionscraperscrapingscraping-toolnpm-packagescrapers
  Language:TypeScript 65

• fernandod1 / Instagram-downloader

  Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

  pythoninstagraminstagram-photosinstagram-scraperinstagram-downloaderinstagram-feedscraperscrapscraping-pythonscrapingscraping-toolscraping-websitescrawlercrawling-python
  Language:Python 63

• lspahija / torchestrator

  Spin up Tor containers and then proxy HTTP requests via these Tor instances

  scrapingscraping-websitesscraping-tooldata-scrapingscraping-datakotlintordockerproxyproxy-serverproxy-listproxy-checkerproxypoolproxiesproxies-generatorproxies-https-trafficfree-proxiesproxy-scraperprivoxyscrapy
  Language:Kotlin 43

• harismuneer / Android-Apps-Downloader

  📱 A tool to download android apps from Google Play Store and Xiaomi App Store (the famous Chinese Store).

  scrapercrawlerandroid-researchpython-scraperxiaomi-appsandroid-application-downloaderscraping-toolcrawling-toolopen-source-projectgoogle-play-store-scraperxiaomi-store-scraperwget-utilitygplaycliresearch-toolandroid-scraperandroid-apps-crawlerandroid-toolgoogle-play-application-downloaderapp-downloaderandroid-market-scraper
  Language:Python 31

• ScrapingAnt / zoominfo_scraper

  Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

  scrapingscraping-apiscraping-websitesscraping-datascraping-toolpythonweb-harvestingweb-crawlerweb-crawlingweb-crawler-pythonscraperzoominfo-clientdataminingleadgenleadgeneration
  Language:Python 28

• fernandod1 / ProductHunt-scraper

  Producthunt.com famous website scraper script. Scrap all offers and save in spreadsheet excel file.

  pythonpython3python-scriptscraperscrapingscraping-websitesscraping-pythonscraping-toolscraping-botdata-miningdataminingproducthuntproducthunt-usersproducthunt-apiscrapescraped-datascraper-enginecrawlercrawling-sitescrawling
  Language:Python 18

• Joabutt / ScrapeDogg

  GPT4 Assisted Web Scraping Library

  gpt4scraperscraping-toolscraping-webscraping-websites
  Language:JavaScript 18

• ScrapingAnt / alibaba_scraper

  Alibaba scraper with using of rotating proxies and headless Chrome from ScrapingAnt

  scrapingscraping-apiscraping-websitesscraping-webscraping-dataprice-scrapingprice-scraperscraping-toolpythonweb-crawlerweb-crawler-pythonweb-crawlingscraperalibaba-scraperdatamining
  Language:Python 17

• mapmeld / aoc_reply_dataset

  Building a dataset of Twitter replies for unsupervised learning / bot-blocking

  nlpnlp-machine-learningabuse-detectiontwitterscraping-tool
  Language:Python 13

• omkarcloud / botasaurus-starter

  🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolheadlessnode-crawlerpython-crawlerscraperscrapingscraping-frameworkscraping-pythonscraping-toolseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:TypeScript 13

• DemonMartin / scrappey-wrapper

  An API wrapper for Scrappey.com written in Node.js (cloudflare bypass & solver)

  cloudflare-bypasscloudflare-solverdata-extractionscraping-frameworkscraping-toolweb-data-extractionweb-scrapingweb-scraping-solutionapi-scrapingcloudflare-anti-botdata-scraping-toolscraping-libraryscraping-servicescraping-solutionturnstile-solverwebsite-data-extractionwebsite-scraping-tool
  Language:JavaScript 10

• pim97 / scrappey-wrapper-python

  An API wrapper for Scrappey.com written in Python (cloudflare, datadome bypass & solver)

  akamaicaptchacaptcha-solvercloudflare-anti-botcloudflare-bypassdata-extractiondatadomeincapsulaqueue-itscraping-frameworkscraping-libraryscraping-servicescraping-toolshapeweb-scrapingweb-scraping-solutionanti-bot-apiperimetexweb-data-extration
  Language:Python 10

• skvrahul / chegg_dl

  Python script to automate the download of textbooks from Chegg

  cheggdownloaderpythonscraping-toolseleniumweb-automation
  Language:Python 9

• omkarcloud / web-scraping-template

  🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolheadlessnode-crawlerpython-crawlerscraperscrapingscraping-frameworkscraping-pythonscraping-toolseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:Python 7

• jaeyk / digital_data_collection_workshop

  Digital Data Collection Workshop

  scraping-websitesscraping-toolscraping-datascraping-botscraping-api
  Language:HTML 6

• rija / ghost-ssg

  A Docker-based pipeline to publish the content of a local Ghost 4 server as static pages.

  dockerdocker-composeghostghost-cmsstatic-site-generatornodejsnginxbashblogginggitlabintegrationjamstackpublishingself-hostedwgetcliworkflowscraping-toolweb-scraper
  Language:Shell 6

• MustakAbsarKhan / DSE_COMPANY_SCRAPER_Python

  The DSE Company Scraper is a Python program that extracts data from the Dhaka Stock Exchange website and saves it to an Excel file for analysis.

  pythonstock-analysisstock-marketautomationdata-analyticsdata-automationdata-extractiondata-miningdata-processingdata-sciencedata-visualizationdhaka-stock-exchangeexcelfinancefinancial-datamachine-learningopen-sourcescraping-toolweb-scraping
  Language:Python 4

• ayushsoni1010 / portfoliogram

  ⚡️Elevate your portfolio analysis with our cutting-edge web scraping tool. Uncover valuable insights about individuals, their skills, and social profiles effortlessly.

  javascriptopenaiopenai-apiportfoliopuppeteertypescriptweb-scrapingwebsiteanalyticsnodejsportfolio-websitereactscrapinghacktoberfesthactoberfest2023scraping-toolmongodb
  Language:JavaScript 3

• mannasoumya / instapy_pubprofiles

  Download Instagram Photos and Videos given Link to Posts

  download-instagram-photosinstagraminstapy-pubprofilesphotospython-3python3scrapingscraping-pythonscraping-toolscraping-websiteswebscraperwebscrapingwebscrapperwebscrapping
  Language:Python 3

• pim97 / scrappey.js

  Scrappey.js: A versatile JavaScript wrapper for Scrappey API for solving Cloudflare, datadome, enabling seamless web scraping of anti-bot protected websites. Simplify data extraction with robust functionality and reliable results. Unlock valuable insights effortlessly. Get started with Scrappey

  api-scrapingcloudflare-anti-botcloudflare-bypasscloudflare-solverdata-extractionscraping-toolturnstile-solverweb-scrapingakamai-solverincapsula-solverperimeter-x-solverweb-scraping-apicloudflare
  Language:JavaScript 3

• patgdut / GoogleMapsScraper

  By scraping leads from Google Maps, you can build a database of potential customers who have shown interest in products or services related to your business. This data can be used for targeted marketing campaigns, email outreach, or sales prospecting.

  crawlercrawlerscrawlingcrawling-toolgoogle-mapsgoogle-maps-scrapinggoogle-serp-apiscraperscrapersscrapingscraping-toolscraping-toolsweb-crawlingweb-scraperweb-scrapersweb-scrapingwebscraping
  2

• rifki / web-scraping-job-postings

  Web Scraping wirh Node.js - Puppeteer https://blog.rifkilabs.net/web-scraping-dengan-node-js.html

  puppeteernodescraping-toolnodejs
  Language:JavaScript 2

• CyberSudo / Udemy-Enroller

  Python script that written by selenium to automate the process of enrolling in Udemy courses.

  udemypaid-coursespythonscraping-toolweb-automationselenium
  Language:Python 1

• Ekans111 / Aliexpress-scraper-without-api-free

  Aliexpress products scraping.

  aliexpresspythonrecaptcha-solverscrapingseleniumno-api-keyaliexpress-scraperno-apiscraping-toolscrapping
  Language:Python 1

• Kareem-Emad / youtube_metadata_scraper

  An expansion over the Youtube-8m Dataset to get more data about the videos such likes/views and channel info through scrapping youtube

  youtubepythonscrapingscraping-toolyoutube8mdataset-generationmetadata-extractiontensorflow2
  Language:Python 1

• kawsarlog / AmerisourceBergen

  🛠️ Python 🐍 script automates the extraction of product pricing details from the AmerisourceBergen 🌐 website https://abcorder.amerisourcebergen.com By inputting your username, password, and National Drug Code (NDC) codes and the 📜 script navigates the website and retrieves the 💰 Average Wholesale Price (AWP) and Acquisition Cost (AC) 📊 data

  automationbrowser-automationdata-automationdata-extractiondata-processingexcel-automationexcel-manipulationopenpyxlpythonscraping-toolseleniumweb-automationweb-scrapingwebdriverabcorder-amerisourcebergenamerisourcebergepharmaceutical-datapharmaceutical-industry
  Language:Python 1

• MaxValue / IsJavascriptWorking

  test if your damn browser has JS enabled

  javascripttesting-toolsbrowser-testingjavascript-toolsweb-testweb-toolscraping-toolscraping-testcrawler-testingscrapy-splashscrapyscrapy-crawlerscrapy-spiderscraping-resources
  Language:HTML 1

• MoutasemZ / Reddit-Scraper

  Reddit-Scraper is a tool that I have developed to scrape the content of specific subreddits, and I have used it in the research of my Ph.D dissertation in Health Informatics at the University of Waterloo, Ontario, Canada.

  redditjavawebscraping-websitesscraping-tool
  Language:Java 1

• scrape-do / python-sample

  Best Rotating Proxy & Scraping API Alternative. Python Example.

  captchacrawlercrawlersdata-miningdata-sciencedata-scrapingfreeproxyfreeproxylistproxyproxy-listrotating-proxyscraperscrapingweb-scrapersweb-scrappingcapcha-solverfreescraping-apiscraping-tool
  Language:Python 1

• Assadzy / Selenium_walmart

  Download and save Walmart product reviews to excel sheet

  seleniumpythonwebscrapingscraping-toolscraping-websitesscraping-pythonselenium-webdriver
  Language:Python 0

• floscodes / sitescraper

  Scraping Websites in Go!

  golanghtmlscraping-datascraping-toolscraping-websiteswebscraperwebscrapingwebscraping-data
  Language:Go 0

• scrape-do / dotnet-example

  Best Rotating Proxy & Scraping API Alternative. C# Example.

  captchacaptcha-solvercrawlercrawlerscrawlingdata-miningdata-sciencedata-scrapingfreefree-proxyfree-proxy-listproxyproxy-listproxylistrotating-proxyscraperscrapingscraping-apiscraping-tool
  Language:C# 0