crawler-engine

There are 1 repository under crawler-engine topic.

• 6677-ai / tap4-ai-crawler

  The crawler opened source by tap4.ai

  aitoolkitaitoolscrawlercrawler-enginecrawler-python
  Language:Python 137
• WebScrapper

  nuhmanpk / WebScrapper

  Simple and powerfull all in one Telegram Bot to scrap / crawl webpages using Requests, html5lib and Beautifulsoup

  telegram-botwebscrapingrequestsbeautifulsoup4pyrogram-botpyrogramtelegramwebscrappingwebscrapperwebscrapping-pythoncrawlerscraperscrapingweb-scrapinghacktoberfesthacktoberfest-acceptedhacktoberfest2023seleniumcrawler-enginecrawler-python
  Language:Python 116

• namhong1412 / browser-clone-web

  Use browser to re-copy a web page

  chromedrivercrawler-enginepythonselenium-pythonclone-uiclone-website
  Language:Python 22

• bkeepers / spiderman

  your friendly neighborhood web crawler

  webcrawlercrawlercrawler-enginespiderweb-scrapingrubyspider-frameworknokogirihttphttprbweb-crawlerwebscraping
  Language:Ruby 18

• fooock / robots.txt

  :robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

  robots-txtrobots-parserjavakotlinredispostgresqlantlr4gradlecrawlercrawler-enginespidersapidockerdocker-composemakefilespring-bootredis-streamsredis-stream
  Language:Java 15

• web-extractors / arachnid-seo-js

  Web crawler for extracting internal site links info for SEO auditing & optimization purposes

  crawler-engineseo-optimizationcrawlerscraperseoseotools
  Language:TypeScript 15

• Sobak / scrawler

  Declarative, scriptable web robot (crawler) and scrapper

  crawlercrawler-enginerobots-txtscraperscraping-websites
  Language:PHP 10

• wetrycode / tegenaria

  Tegenaria is a crawler framework based on golang

  golanggocrawlercrawler-enginespiderspidersframeworkcrawler-framework
  Language:Go 9

• wefindx / metadrive

  Generic Interfaces to Addressable Objects

  driverframeworkcrawler-enginecontroller-managerformatsprotocolssessionsproxiesgeneratorsiteratorsfilters
  Language:Python 8

• BaseMax / NetPHP

  Useful functions for connecting to the network in the PHP based applications.

  phpnetworkcurlcurl-phpcurl-librarycurl-wrappercurl-commandsphp-networkphp-requestphp-httpphp-httpstatuscodephp-httpsphp-cookiecurl-cookiecurl-requestcurl-headercrawler-enginecrawling-frameworkcrawl-pagescrawling-sites
  Language:PHP 7

• crawlbase / crawlbase-ruby

  Fast Crawlbase API crawling library

  crawlercrawler-enginecrawlingscraperscrapingscraping-ruby
  Language:Ruby 7

• spekulatius / spatie-crawler-cached-queue-example

  Example to demonstrate the usage of cached queues across multiple requests.

  laravelqueuescrawlercrawler-enginespatie-crawlerphp-scraperphp-crawler
  Language:PHP 7

• lichang98 / visualize_spider

  基于Spring Boot、Scrapy 的可视化爬虫配置与管理

  crawler-enginevisualization
  Language:HTML 6

• ShiqinHuo / wuhan_house_price_crawler

  武汉东湖高新片区光谷&软件园二手房房价爬虫。data source: 房天下

  housing-priceshouse-price-predictionguanggoohouse-prices-crawlercrawlerscraping-websitesfangtianxiascraping-pythoncrawler-enginecrawler-house-priceswuhanwuhan-house-prices
  Language:Jupyter Notebook 6

• supernebula / shark

  Shark (Plunder)可配置、插件化的爬虫引擎，二次开发框架。Configurable, pluginable crawler engine, secondary development framework.

  crawler-engineschedulerremove-duplicatedownloaderanalyzerpipelineframework
  Language:C# 5

• Colaplusice / zhihu

  数据挖掘实验，抓取用户信息并且进行聚类等处理

  zhihu-crawlerrequestsmongodbcrawler-engine
  Language:Jupyter Notebook 4

• hseghetti / simple-crawler

  Simple crawler using apache nutch and elasticsearch

  crawlercrawler-enginenutchelasticsearchcerebrodockerdocker-composecrawlspidercrawling
  Language:Shell 4

• MCStreetguy / Crawler

  An advanced web-crawler written in PHP.

  phpcrawlerweb-crawlerguzzlehttp-requestscrawler-enginewebcrawlercomposercomposer-libraryphp-7php-library
  Language:PHP 4

• andrrff / BugSearch

  BugSearch é um motor de pesquisa de páginas indexadas pelo crawler BugSearch.Crawler. O projeto é dividido em duas partes: o lado do Bot (Bot side) e o lado do Cliente (Client side).

  azureazurekubernetesservicecrawlercrawler-enginecsharpdockerkubernetessearchsearch-engine
  Language:C# 3

• its-my-data / android-crawler-engine

  An Android app crawling framework, making automatic crawling mobile apps super easy! (if possible, iOS will be supported after Android version)

  crawlercrawling-frameworkcrawler-engineandroidadbprogrammable
  3

• Keerthivasan13 / Targeted_Advertising_Google_AdSense

  Hybrid E-Marketing using Web Page Mining for Website Monetization

  google-adsgoogle-analyticsgoogle-adsensegoogle-adwordswebsite-monetizationtargeted-advertisingnaive-bayes-classifieradvertisement-management-systemgoogleinformation-retrievaldata-engineeringdata-miningcrawler-enginejsoupranking-algorithm
  Language:TSQL 3

• KonghaYao / jspider

  This is a JavaScript toolkit for browser crawler testing.

  js-jspiderwebsitebrowsercrawler-enginespider-framework
  Language:JavaScript 3

• plugnsearch / plugnsearch

  The only real pluggable crawler / spider / webcrawler to search the web for stuff you need to know.

  crawlercrawler-enginesearch-enginewebpage-scraperscraper
  Language:JavaScript 3

• kingzbauer / scraperlang

  A DSL aimed at making writing web scrapers/crawlers a breeze

  scraperscraper-enginecrawlercrawler-enginegolang
  Language:Go 2

• rihenperry / whirlpool-urlfrontier

  mercator scheme/rate-limiting/scheduling part of whirlpool project; handles crawler priority and politeness

  mercatorcrawler-enginecrawlerschedulingrate-limitingbinary-heappriority-queue
  Language:Java 2

• robincloud / robinbot

  robin micro web crawling engine with nodejs

  crawlingcrawlercrawler-enginestate-machineiotnodejs
  Language:JavaScript 2

• rrmerugu / trawler

  A data gathering/trawling framework to search and get information from web sources like bing

  pythonsearchwebcrawlercrawler-engine
  Language:Python 2

• runjia1987 / crawler-engine

  crawler-engine with HTTP, proxy, JS-Java Interoperability, MQ task consumption, dynamic crawler scripts execution. support deployment in distribution style.

  crawler-enginerabbitmqproxyjs-java-interoperabilitymq-task-consumptionnashornrhino-js
  Language:Java 2

• EYazdpour / DirectoryCrawler

  Simple crawler for a directory (on Windows) which return all possible information about whatever is in that given directory

  read-file-to-stringread-directorycrawler-enginecppcpplusplus
  Language:C++ 1

• johnvanderton / flysh

  HTML type document parser based on jQuery and JSDOM

  crawler-enginehtmljqueryparser-librarytypescript-librarycrawlerweb-crawlerdomdom-manipulationjavascriptweb-parserscrapertypescriptjavascript-libraryjsdom
  Language:TypeScript 1

• MaximeGuinard / Gtool-projects-crawler-seo

  🤖 A Google extension that facilitates project management with various tools

  crawlercrawler-enginecrawler-pythoncrawler-seocrawlersextensibleextensionextension-chromeextension-methodsextensionsseoseo-friendlyseo-optimizationseo-toolsseotoolsurl
  Language:HTML 1

• nicolasmelo1 / price_miner

  Price miner from e-commerces that i made for Price Management class of my Marketing Graduation and want to turn on my possible TCC for price analysis of e-commerces

  minercrawlercrawlingscraperscrapingflaskflask-apicelerye-commerceseleniumselenium-webdrivercrawlerscrawler-enginescraping-websitesscrapperpython3
  Language:HTML 1

• paganini2008 / greenfinger

  A high-performance distributed web crawling framework based on SpringBoot framework. It provides rich APIs to customize business and easily embedded your system.

  javacrawler-enginedistributed-systemshigh-performancemircoservice
  Language:Java 1

• renanbm / WebCrawler

  Open source, multi-threaded website crawler written in C#, persisting in IBM's Cloudant NoSQL DB and configured for a Linux Docker image.

  webcrawlerspidercrawler-enginenosqlcloudantasp-net-coredocker
  Language:C# 1

• setulparmar / Landslide-Detection-and-Prediction

  This project named "Landslide Detection and Prediction" was done during my summer internship under Visiting Associate Prof. Gagan Raj Gupta at IIT - Bhilai.

  machine-learningdeep-learningnlpcrawler-enginepython-3
  Language:Jupyter Notebook 1

• ShubhamThakurela / global-social-media-ms

  Functionality to Extract Social data.

  crawler-enginelogginglogsmailmysql-databasesocial-media
  Language:Python 1