scraping-framework

There are 5 repositories under scraping-framework topic.

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  captcha-bypasscaptcha-recaptchacrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolspiderweb-scrapingwebscraping
  Language:Makefile 6357
• botasaurus

  omkarcloud / botasaurus

  The All in One Framework to build Awesome Scrapers.

  anti-botanti-detectioncloudflare-bypasscloudflare-scrapeanti-detectanti-detect-browserantidetect-browserundetectedundetected-chromedriverbypass-cloudflarepython-web-scraperpython-web-scrapingscraping-frameworkscraping-toolundetectableweb-scraping-pythonbot-detectionscraping-pythonweb-crawlingpython-scraper
  Language:Python 956

• zhuyingda / webster

  a reliable high-level web crawling & scraping framework for Node.js.

  scraping-frameworkcrawlercrawlingheadless-chromechromiumspiderautomation-uiautomation-testnodejsnodejs-frameworkjavascriptjavascript-frameworkpuppeteer
  Language:JavaScript 505

• alephdata / memorious

  Lightweight web scraping toolkit for documents and structured data.

  crawlingscrapingscraping-framework
  Language:Python 308

• kennethreitz / requests-html

  Pythonic HTML Parsing for Humans™

  requestshtmlscrapingscraping-websitesscraping-framework
  Language:Python 278

• flulemon / sneakpeek

  Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

  crawlercrawler-pythoncrawlerscrawlingcrawling-enginecrawling-frameworkpythonpython3scraperscraper-apiscraper-enginescrapersscrapingscraping-frameworkvuewebsite-crawler
  Language:Python 36

• peterbencze / serritor

  Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

  crawlerjavaseleniumframeworkscraperdynamic-websitedynamic-webpagesautomationdata-miningselenium-crawlerscrapingscraping-frameworkcrawling-frameworkinformation-retrievalinformation-extractionwebspidercrawlingcrawlerscrawlextract-data
  Language:Java 31

• crawlbase / proxycrawl-php

  ProxyCrawl PHP library for scraping and crawling websites

  scraperscrapingcrawlercrawlingproxycrawlscrapersscraping-websitesscraping-frameworkproxycrawl-api
  Language:PHP 19

• omkarcloud / botasaurus-starter

  🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolheadlessnode-crawlerpython-crawlerscraperscrapingscraping-frameworkscraping-pythonscraping-toolseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:TypeScript 13

• omkarcloud / omkar-temp-mail

  🚀 OMKAR TEMP MAIL HELPS YOU USE TEMPORARY EMAILS. 🤖

  10minute10minutemaildisposable-emaildisposable-email-addressesfree-mailmail-apitemp-mailtempmailtemporary-emailbeautifulsoupcrawlingscraperscrapingscraping-frameworkseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:Python 11

• rajanrx / php-scrape

  A simple, easy to use, scalable scraping framework written in PHP

  phpscraping-frameworkscrapescraping
  Language:PHP 11

• DemonMartin / scrappey-wrapper

  An API wrapper for Scrappey.com written in Node.js (cloudflare bypass & solver)

  cloudflare-bypasscloudflare-solverdata-extractionscraping-frameworkscraping-toolweb-data-extractionweb-scrapingweb-scraping-solutionapi-scrapingcloudflare-anti-botdata-scraping-toolscraping-libraryscraping-servicescraping-solutionturnstile-solverwebsite-data-extractionwebsite-scraping-tool
  Language:JavaScript 10

• pim97 / scrappey-wrapper-python

  An API wrapper for Scrappey.com written in Python (cloudflare, datadome bypass & solver)

  akamaicaptchacaptcha-solvercloudflare-anti-botcloudflare-bypassdata-extractiondatadomeincapsulaqueue-itscraping-frameworkscraping-libraryscraping-servicescraping-toolshapeweb-scrapingweb-scraping-solutionanti-bot-apiperimetexweb-data-extration
  Language:Python 10

• umihico / minigun-requests

  Web scraping API to outsource tons of GET & xpath to cloud computing

  scrapingweb-scrapingscraping-apiscraping-pythoncrawlercrawlingscraping-framework
  Language:Python 8

• omkarcloud / web-scraping-template

  🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolheadlessnode-crawlerpython-crawlerscraperscrapingscraping-frameworkscraping-pythonscraping-toolseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:Python 7

• omkarcloud / selenium-2captcha-recaptcha-solver-demo

  🚀 FINAL CODE FOR TUTORIAL ON HOW TO SOLVE CAPTCHA IN SELENIUM USING 2CAPTCHA 🤖

  2captchacaptcha-bypasscaptcha-solvercaptcha-solvingcaptchacaptcha-breakcaptcha-breakercaptcha-breakingcaptcha-generatorcaptcha-imagecaptcha-librarycrack-captchascrapingscraping-frameworkseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscraping
  Language:Python 6

• JimmyLaurent / node-crawling-framework

  ✨ NodeJs crawling & scraping framework heavily inspired by Scrapy

  crawlercrawlingcrawling-frameworkelasticsearchheadless-chromemiddlewaremongodbnodejs-frameworkscraperscrapingscraping-frameworkscrapyspider
  Language:JavaScript 4

• lauramatilda / newsScraping

  M.A. Thesis work, news scraping framework/pipeline using python, beautifulsoup, newspaper3k, flask and mongodb with a custom api.

  scraping-websitesmongodbflaskscraperscraping-framework
  Language:Python 3

• omkarcloud / dentalkart-scraper

  🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖

  beautifulsoupcrawlercrawlingcrawling-frameworkcrawling-pythonnode-crawlerscraperscrapingscraping-frameworkscraping-pythonseleniumweb-crawlerweb-crawlingweb-scraperweb-scrapingwebscrapingdentalkartdentalkart-product-scraperdentalkart-scraperdentalkart-scraping
  Language:Python 3

• tor4z / crabs

  Scraping framework for Python

  scraping-frameworkscrapy-crawlerpython3scraper
  Language:Python