web-spider

There are 12 repositories under web-spider topic.

• projectdiscovery / katana

  A next-generation crawling and spidering framework.

  clicrawlergocrawlerheadlessspider-frameworkweb-spider
  Language:Go 10484

• ssssssss-team / spider-flow

  新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

  crawlerjsoupspiderspider-flowweb-crawlerweb-spiderwebcrawlerwebspiderxpath
  Language:Java 9339

• xianhu / PSpider

  简单易用的Python爬虫框架，QQ交流群：597510560

  crawlerspiderpythonproxiesweb-spidermulti-threadingweb-crawlerpython-spidermultiprocessing
  Language:Python 1819

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  spiderrubyspider-linkscrawlerwebscraperweb-scrapingweb-spiderweb-crawlerweb-scraper
  Language:Ruby 798

• bouxin / company-crawler

  天眼查爬虫&企查查爬虫，指定关键字爬取公司信息

  proxypython3qichachatianyanchauaweb-spider
  Language:Python 620
• SpiderSuite

  3nock / SpiderSuite

  Advance web security spider/crawler

  bugbountycpluspluscrawlerguiinformation-gatheringosint-toolpentestqt5reconsecurity-toolsspiderweb-spiderwebcrawler
  564

• VIDA-NYU / ache

  ACHE is a web crawler for domain-specific search.

  web-crawlerfocused-crawlerdomain-specific-searchweb-scrapingweb-spiderweb-searchhacktoberfest
  Language:Java 444

• rivermont / spidy

  The simple, easy to use command line web crawler.

  crawlercrawlingpythonpython3web-crawlerweb-spider
  Language:Python 331

• infinilabs / crawler

  🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

  crawlercrawlingelasticsearchlightweightscrapingspiderweb-crawlerweb-scrapingweb-spider
  Language:Go 303

• antchfx / antch

  Antch, a fast, powerful and extensible web crawling & scraping framework for Go

  golangweb-crawlercrawlingscrapingframeworkcrawlerweb-spider
  Language:Go 258

• s0rg / crawley

  The unix-way web crawler

  clicrawlergogolanggolang-applicationpentestpentest-toolpentestingunix-wayweb-crawlerweb-scrapingweb-spider
  Language:Go 246

• xiayouran / Musicer

  旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体

  kugou-musicmusic-playerpythonwangyiyunmusicweb-crawlerweb-spidermusic-download-scriptkuwo-musicmusic-downloaderqq-musicmusic-robot
  Language:Python 224

• elliotxx / zhihu-crawler-people

  A simple distributed crawler for zhihu && data analysis

  pythoncrawlerspiderweb-spiderweb-crawlerpython-crawler
  Language:Python 190

• Hecate2 / Ignareo-ISML-auto-voter

  Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

  asynciochthollyconcurrencydistributedgeventhigh-performancehttpignareoismlmicroservicepythonsukamokasukasukatiatweb-crawlerweb-spider
  Language:Python 188

• genkio / spider-less

  Web spider as a service, spider on serverless

  apiaws-lambdajavascriptserverlessweb-spider
  Language:JavaScript 186

• Andromeda1957 / netpwn

  Tool made to automate tasks of pentesting.

  hacking-toolsecurity-toolssecurityhacking-frameworknetwork-securitysocial-engineeringinfosecinformation-securityweb-spiderssl-certificatessl-certificatesssl-certpassword-strengthpassword-safetyhexadecimal-converterbase64-encodingbase64-decodingipv4ipv6mac-address
  Language:Python 166

• howie6879 / talospider

  talospider - A simple,lightweight scraping micro-framework

  spiderweb-spiderpythoncrawlingcrawler
  Language:Python 54

• lucasxlu / JiaYuan

  user profile of jiayuan.com

  data-analysisdata-miningmachine-learningweb-spidercomputer-visionpython3
  Language:Python 40

• wmylxmj / Web-Spider-Login-Bilibili-Python3

  网络爬虫模拟登陆bilibili 滑动验证码的破解 弹幕发送 2018-10-9

  web-spideranalog-loginseleniumpython3bilibili
  Language:Python 40

• rzo1 / crawler4j

  Open Source Web Crawler for Java - A maintained fork of yasserg/crawler4j

  crawlerweb-crawlerjavaspiderweb-spidercrawler4j
  Language:Java 22

• esfelurm / spider-web

  admin-panel-finderaparatblogfa-crackerblogfa-hackblogfa-reportcrack-md5cracker-blogfacracker-hashesfelurmreportreporterreporter-aparatreporter-blogfascannerspider-websqlsubdomainweb-spiderweb-wormxss
  Language:Python 11

• FearlessPeople / lianjia_spider

  链家网小区信息爬取

  lianjiaspiderweb-spiderxpath
  Language:Python 11

• acuciureanu / spidertrap-rs

  A simple trap for web crawlers

  anti-botanti-scrapingbot-detectionbot-trapcrawler-trapcybersecuritydata-privacyethical-hackingintrusion-detectionnetwork-securityrustserver-defensespider-trapweb-crawlerweb-monitoringweb-robotsweb-scrapingweb-securityweb-spiderwebsite-protection
  Language:Rust 10

• holisound / haokan

  2018年底到2019上半年，通过抓包工具发现🎦好看视频的参数签名漏洞，并成功利用刷金币兑换现金，持续半年。

  pythonapktooljadxsmaliandroidweb-spiderfiddlermitm-attacks
  Language:Python 10

• MrThanlon / bili-comment

  哔哩哔哩（https://www.bilibili.com ）读取楼层，发送评论，抢楼，抢沙发

  bilibili-apibilibilipythonpython2web-spiderweb-crawler
  Language:Python 10

• jdevelop / webspider

  Open WEB spider platform

  web-spiderakka-clusterakka-httpakkascala
  Language:Scala 6

• AngusMonroe / Intelligent-interrogation

  Use Word2vec model and LDA model for drug recommendation

  jiebaword2vecpython3pythonlda-modellevenshteinweb-spiderdjango
  Language:Python 5

• kan01234 / ur-web-spider

  web spider to scan UR avialbe room and output as csv

  pythonweb-crawlerweb-spidercsvhtml-parsingjson
  Language:Python 5

• lewisakura / spiderboi

  A web crawling library written in TypeScript.

  web-crawlerwebcrawlerweb-crawlingweb-spiderspidertypescripttypescript3
  Language:TypeScript 5

• lucasxlu / DataHouse

  a data mining and machine learning repo

  machine-learningdata-miningweb-spidernlpscikit-learnpandasmongodbscrapydeep-learning
  Language:Python 5

• rdempsey / data-gathering-and-wrangling

  Code and slides for my class: Data Gathering & Wrangling

  data-sciencedata-gatheringdata-wranglingpythonapiweb-spiderwebscrapingwebscraper
  Language:Python 5

• vladimanaev / web-spider

  web crawler allowing full page render crawl using HtmlUnit

  web-spiderhtmlunitcrawlerweb-crawlerweb-crawlingwebpage-scraperweb-scrapingweb-scraper
  Language:Java 5

• fatihyildizli / FYCrawler

  🕷 Custom Web Spider | Frontend: ⚛️ React.js - Backend: ☕️ Java / NodeJS

  web-spiderweb-scrapingreactjsreactjs-uiweb-crawlerreact-crawlerreactjs-crawlercrawler-uicrawler-javaspring-boot-crawlerspring-boot-web-spideribm-carbonibm-componentsibm-react-componentscustom-crawlerspider-webweb-scraperreact-spiderheroku-react
  Language:JavaScript 3

• fxtack / weibo-spider

  单文件项目，使用 Python3（jupyter notebook）调用微博移动端 api 对特定话题进行数据爬取、存储、分析、展示。

  python3jupyter-notebooksweb-spider
  Language:Jupyter Notebook 3

• davidrogger / trybe-project-tech-news

  Projeto de raspagem de dados do blog da trybe

  crawlerparselpymongopythonrequestsscrapingweb-spider
  Language:Python 2

• judemont / JdmSearchIndex-Bot

  Open source public Search Engine indexation Web Crawler.

  crawlerindexationsearch-engineweb-crawlerweb-spiderwebcrawler
  Language:Python 2