webcrawler

There are 17 repositories under webcrawler topic.

• crawlab

  crawlab-team / crawlab

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  crawlabcrawlercrawling-tasksdockergoplatformscrapyscrapyd-uispiderspiders-managementweb-crawlerwebcrawlerwebspider
  Language:Go 10870

• ssssssss-team / spider-flow

  新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

  spidercrawlerjsoupxpathweb-spiderwebspiderwebcrawlerweb-crawlerspider-flow
  Language:Java 9107

• GeneralNewsExtractor / GeneralNewsExtractor

  新闻网页正文通用抽取器 Beta 版.

  python3webcrawlerwebspider
  Language:Python 3412

• zorlan / skycaiji

  蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

  crawlercrawlingphpspiderwebcrawler
  Language:PHP 1871

• amirgamil / apollo

  A Unix-style personal search engine and web crawler for your digital footprint.

  searchposeidonwebcrawlerpersonal-searchunix-like
  Language:Go 1360

• scrapinghub / scrapyrt

  HTTP API for Scrapy spiders

  pythoncrawlingcrawlerscrapyscrapertwistedwebcrawlingwebcrawlerhacktoberfesthacktoberfest2021
  Language:Python 817

• spider-rs / spider

  The fastest web crawler written in Rust. Maintained by @a11ywatch.

  crawlerindexerrustspiderscrapingheadless-chromeai-scrapingllm-crawlerweb-crawler
  Language:Rust 671
• SpiderSuite

  3nock / SpiderSuite

  Advance web security spider/crawler

  cpluspluscrawlerosint-toolqt5reconspiderweb-spiderbugbountyguisecurity-toolswebcrawlerinformation-gatheringpentest
  542

• jaeksoft / opensearchserver

  Open-source Enterprise Grade Search Engine Software

  searchsearch-enginecrawlerwebcrawlerwebcrawlingcustom-searchindexingluceneopensearchserverjavaenterpriseocrsynonyms
  Language:Java 499
• Uscrapper

  z0m31en7 / Uscrapper

  Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

  osintpythonweb-scrapingwebsite-scraperinformation-extractioninformation-gatheringosint-pythonosint-toolreconnaissancewebscrapingwebsitesseleniumselenium-webscraperwebcrawebcrawlerdarkwebdarkweb-crawlertor
  Language:Python 425

• salimk / Rcrawler

  An R web crawler and scraper

  rrpackagecrawlerscraperwebcrawlerwebscrapingwebscraperwebscrappingcrawlers
  Language:R 349

• kingname / SourceCodeOfBook

  《Python爬虫开发 从入门到实战》配套源代码。

  pythonpython3requestsscrapywebcrawler
  Language:Python 336

• adrianosferreira / afrodite.json

  O maior livro de receitas culinárias em língua portuguesa

  javascriptmongodbnodejswebcrawler
  185

• sushant10 / HQ_Bot

  📲 Bot to help solve HQ trivia

  botquestion-answeringpython3questions-and-answerswebcrawlerwebscrapinghq-triviacashshowhqtriviatrivia-gamehq-trivia-bothq-bottesseracthq-trivia-hack
  Language:Python 173

• mehmetozkaya / DotnetCrawler

  DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

  dotnetcorecrawlercrawlingscrapingscrapyscrapy-crawlerentity-framework-coreddd-architecturecsharpwebcrawlerwebcrawlingwebscrapingwebscraperwebcrawler-htmlagilitypackhtmlagilitypack
  Language:C# 170

• codeudan / crawler-china-mainland-universities

  **大陆大学列表爬虫

  spidernodejsuniversitycrawlerchinadataschoolwebcrawler
  Language:JavaScript 165

• DedSecInside / gotor

  This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

  golanggotortorbotcliwebscrapingosintcommand-linecommand-line-toolosint-toolsinformation-extractionrest-apihttp-serverservicewebcrawlergolang-serverwebcrawlinghacktoberfestdocker
  Language:Go 152

• hedii / php-crawler

  A php crawler that finds emails on the internets

  php-crawlercrawlerphplaravelvuejsvuewebscrapingwebscraperwebcrawler
  Language:PHP 134

• brianmadden / krawler

  A web crawling framework written in Kotlin

  webcrawlerkotlinframeworkcrawler4jlink-checkerweb-crawlerweb-crawling
  Language:Kotlin 130

• voliveirajr / seleniumcrawler

  An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

  seleniumselenium-webdriverscraperscrapingscraping-websitesscrapperasp-netpythonscrapywebcrawlerwebcrawling
  Language:Python 127

• topiccrawler / jkcrawler

  使用 Scrapy 写成的 JK 爬虫，图片源自哔哩哔哩、Tumblr、Instagram，以及微博、Twitter

  jkscrapyspiderwebcrawlercrawling
  Language:Python 110

• 52ai / Crawler4Caida

  Stick to doing something interesting and valuable.

  python3webcrawlermachine-learningcaidaopen3dpclmatplotlibpyqt5nodejschatgpt
  98

• pavlovtech / WebReaper

  Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.

  crawlerparserwebcrawlerwebscrapingdataminingscraperparsingscrapingscraping-apiscraping-datascraping-toolscraping-webscraping-websites
  Language:C# 94

• makuto / Liked-Saved-Image-Downloader

  Save content you enjoy!

  redditprawimgurimageswebcrawlerofflinetumblrdownloadergfycatdata-preservationpixivpixiv-downloader
  Language:Python 89
• Intelligent_Document_Finder

  Sarthakjain1206 / Intelligent_Document_Finder

  Document Search Engine Tool

  search-enginesearch-algorithmranking-algorithmindexerreverse-indexdocument-summarizationdocument-similaritylatent-dirichlet-allocationtext-summarizationspellcheckerwebcrawlerscrapyscrapy-spiderwikipedia-searchwikipedia-crawlerbm25
  Language:Python 68

• shenxiangzhuang / PythonDataAnalysis

  The data and code that used in my book.

  python3data-sciencewebcrawler
  Language:Jupyter Notebook 67

• realdennis / igcloud

  *UNSUPPORTED* Use igcloud to generate Instagram Word Cloud ! 🛫 🛫 ✈ 🔝

  pythoninstagramsocial-network-analysisanalyzersocial-mediatext-miningjiebawordcloudwordcloud-generatorwebcrawler
  Language:Python 66
• konadl

  k4yt3x / konadl

  Multithreaded Konachan / Yandere (moebooru based site) Image Bulk Downloader | 多线程K站Y站下载器

  konachanwebcrawleranimeyanderemoebooru
  Language:Python 64

• hysios / coronavirus

  2019 nCoV realtime track system based Scrapy + influxdb + grafana + NLTK + Stanford CoreNLP

  coronavirusncov-2019scrapywebcrawler
  Language:Python 61

• Aravindha1234u / SocialScraper

  Social Scraper is a python tool meant for Detection of Child Predators/Cyber Harassers on Social Media

  cybersecurityscrapergoogle-apitwitter-apifacebookinstagramwebcrawlerpython3toolcyber-securitysocialmediasocialmediascrapercybertoolpythontoolpythontoolscyberharasserschildpredatorsdetectorfindersocialscraper
  Language:Python 53

• BitTigerInst / Pikachu

  Yummy Recipe Crawler and Search

  crawlerwebcrawlerionichybrid-appsmean-stack
  Language:JavaScript 51

• robsonbittencourt / gafanhoto

  Bot para monitoramento de promoções no fórum do Hardmob http://www.hardmob.com.br/promocoes/

  chatbotgafanhotohardmobpromocoestelegramwebcrawler
  Language:Java 50

• hfreire / browser-as-a-service

  A web browser :earth_americas: hosted as a service, to render your JavaScript web pages as HTML

  browserbrowser-as-a-servicecrawlerdockergithub-actionsjavascriptpuppeteerrest-apiscraperserverwebcrawler
  Language:JavaScript 48

• DeuxHuitHuit / algolia-webcrawler

  Simple node worker that crawls sitemaps in order to keep an algolia index up-to-date

  algoliaalgolia-webcrawlerindexingjavascriptsearch-enginewebcrawler
  Language:JavaScript 46

• Conso1eCowb0y / Deepminer

  Deep web crawler and search engine

  tortor-networkdeepwebdark-webcrawlersearch-enginespiderpython-web-scraperpython3security-toolssecurityonionthe-onion-routerosinthackingdeepminergithubwebcrawlercrawlingdata-mining
  Language:Python 44

• opencharles / charles

  Java web crawling library

  webdriverwebcrawlerseleniumdynamic
  Language:Java 32