spider-framework

There are 5 repositories under spider-framework topic.

• projectdiscovery / katana

  A next-generation crawling and spidering framework.

  clicrawlergocrawlerheadlessspider-frameworkweb-spider
  Language:Go 8673

• Symbo1 / wsltools

  Web Scan Lazy Tools - Python Package

  securitysecurity-toolssecurity-scannersecurity-automationpackagesecurity-auditspiderspider-frameworkscanner-websecurity-toolcrawling-frameworkweb-vulnerability-scannerpython-package
  Language:Python 310

• zhao94254 / pspider

  一个简单的分布式爬虫框架

  pythoncrawlspiderspider-frameworkflask-apicelery
  Language:Python 102

• taofen8 / salyut

  This is an engine of a spider markup-language which called trico. (spider framework)

  spider-frameworkcloud-nativeseleniummarkup-languagejavaserverlessautomationautomated-testingyamlspider
  Language:Java 50

• mangenotwork / gathertool

  gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。

  golanggathertoolspidercrawlspider-frameworkgrpc-goproxy-server
  Language:Go 48

• bkeepers / spiderman

  your friendly neighborhood web crawler

  webcrawlercrawlercrawler-enginespiderweb-scrapingrubyspider-frameworknokogirihttphttprbweb-crawlerwebscraping
  Language:Ruby 17

• miroshnikov / scrapyteer

  Web crawling & scraping framework for Node.js on top of headless Chrome browser

  scrapescraperscrapyscraping-websitesscrapy-crawlercrawerweb-crawlerweb-scrapingscrapingcrawlingcrawling-frameworkcrawling-sitesheadlessspider-frameworkspiderweb-scraping-nodejsweb-crawlingcrawling-tool
  Language:TypeScript 16

• HuberTRoy / Seen

  A lightweight crawling/spider framework for everyone(support JavaScript!).:sparkles:

  easy-to-usejavasciprtlightweight-frameworkpython3spider-frameworksupport-javascriptweb-crawling
  Language:Python 13

• leosocy / proksi

  An intelligent proxy server. Provide durable, real-time, high-quality proxies as a middleman or datasource server.

  proxyfreeproxycrawlercrawling-frameworkmitmmitmproxyproxy-scraperspiderspider-frameworkspidermansocks-proxysocks4socks5
  Language:Go 12

• Xunzhuo / AirSpider

  A Fast and Light Python Spider Framework 🕷️

  asynchronouscrawlercrawler-pythondistributedpython3redisspiderspider-frameworkweb
  Language:Python 8

• Windfarer / biu

  biubiubiu~~ I'm a tiny web crawler framework

  crawlerweb-crawlerspiderspider-frameworkpython
  Language:Python 6

• arloor / EasySpider

  A spider framework powered by groovy

  spider-frameworkspider
  Language:Java 5

• Jiramew / async_bowl

  🥣 An Async Distributed Framework

  asynciocrawlerdistributedpythonspider-framework
  Language:Python 5

• KonghaYao / jspider

  This is a JavaScript toolkit for browser crawler testing.

  js-jspiderwebsitebrowsercrawler-enginespider-framework
  Language:JavaScript 3

• sejal-rajput / Amazon-web-scraping-using-Scrapy-Framework

  A web scraping script using scrapy framework of python to extract data from amazon.com website.

  scrapy-crawlerscrapy-spiderpythonanacondaspider-framework
  Language:Python 1

• showx / phpspiderman

  php爬虫

  phpphpspiderspiderspider-framework
  Language:PHP 1

• wycm / diana

  diana是一个基于Java的垂直爬虫框架，目的是通过封装垂直爬虫常用流程，来简化垂直爬虫开发

  javaspider-frameworkspidercrawlerframework
  Language:Java 1

• AllenTom / YouCrawl

  web crawl framework in golang

  golangspider-framework
  Language:Go 0

• dongxl123 / WinCrawler

  crawlerspider-frameworkspiders
  Language:Java 0

• HorsesElectricStick / SchoolTeacherSpider

  这是一个用于爬取中国大学网站数据的爬虫框架

  chinacollegespider-framework
  Language:Python 0

• cqwang / katana

  A next-generation crawling and spidering framework.

  spider-frameworkspider
  Language:Go