crawlers

There are 5 repositories under crawlers topic.

• g1879 / DrissionPage

  基于python的网页自动化工具。既能控制浏览器，也能收发数据包。可兼顾浏览器自动化的便利性和requests的高效率。功能强大，内置无数人性化设计和便捷功能。语法简洁而优雅，代码量少。

  automation-frameworkcrawlersrequestsselenium-python
  Language:Python 7891
• ai.robots.txt

  ai-robots-txt / ai.robots.txt

  A list of AI agents and robots to block.

  aicrawlerscrawlingprivacy
  Language:Python 1009
• isbot

  omrilotan / isbot

  🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

  crawlersuser-agentuser-agent-analysisuser-agent-parserweb-crawlers
  Language:TypeScript 911

• flathunters / flathunter

  A bot to help people with their rental real-estate search. 🏠🤖

  2captchachromedrivercrawlersidealistaimagetyperzimmobiliareimmobilienscout24immoweltkleinanzeigenmattermost-botpythonreal-estaterealestaterentalseleniumsubitotelegram-botvrmimmowg-gesucht
  Language:HTML 838

• salimk / Rcrawler

  An R web crawler and scraper

  rrpackagecrawlerscraperwebcrawlerwebscrapingwebscraperwebscrappingcrawlers
  Language:R 350

• StJudeWasHere / seonaut

  Open source SEO auditing tool.

  auditcrawlercrawlergocrawlerscrawlingdockerdocker-composegogolangmultiusersearch-engine-optimizationseoseo-auditseotoolsweb
  Language:Go 234

• Norconex / crawlers

  Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

  search-engineweb-crawlerjavacollector-httpflexiblecrawlercrawlersfilesystem-crawlercollector-fs
  Language:Java 183
• Proxy-List-Scrapper

  narkhedesam / Proxy-List-Scrapper

  Proxy List Scrapper

  proxyfreeproxyfreeproxylistproxyscrapeproxiesscrapperfree-proxiesweb-scrapperdata-miningcrawlercrawlersweb-scrappingrotating-proxyproxy-listproxypoolhttphttpssockssocks4socks5
  Language:Python 100

• ArchiveTeam / wget-lua

  Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

  webarchivingwarcwgetluaarchivingcrawlercrawlcrawlingspiderarchiveteamwget-luazstdftpscraperscrapingcrawlersdownloader
  Language:C 94

• jonasjacek / robots.txt

  Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

  googlebotbingbotrobots-txtrobots-exclusion-standardblocking-botsuser-agentweb-robotsseosearch-enginewhitelistcrawlersweb-crawlingcrawlingsearch-engine-optimizationbaiduspidertwitterbot
  87

• behitek / social-scraper

  Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

  instagramyoutubescraping-websitesscraperselenium-pythonrequestscrawlercrawling-frameworkcrawlers
  Language:Python 70

• howie6879 / hproxy

  hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

  asynciocrawlercrawlershproxyproxyproxy-poolproxy-spidersanicschedule
  Language:Python 66

• Potelo / laravel-block-bots

  Block crawlers and high traffic users on your site by IP using Redis

  crawlerslaravelbotsscrapper
  Language:PHP 44

• flulemon / sneakpeek

  Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

  crawlercrawler-pythoncrawlerscrawlingcrawling-enginecrawling-frameworkpythonpython3scraperscraper-apiscraper-enginescrapersscrapingscraping-frameworkvuewebsite-crawler
  Language:Python 37

• BaseMax / GooglePlayWebServiceAPI

  Tiny script to crawl information of a specific application in the Google play/store base on PHP.

  phpgoogle-playgoogle-play-servicesgoogle-play-storegoogle-play-gamesgoogle-playstoregoogle-play-servicegoogle-play-apiapicrawlercrawlerscrawler-phpphp-crawlerhacktoberfesthacktoberfest2020
  Language:PHP 36
• Raven

  Symbolexe / Raven

  Raven is a powerful and customizable web crawler written in Go.

  bugbountycrawlercrawlerscrawlinggolangpentesting
  Language:Go 36

• peterbencze / serritor

  Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

  crawlerjavaseleniumframeworkscraperdynamic-websitedynamic-webpagesautomationdata-miningselenium-crawlerscrapingscraping-frameworkcrawling-frameworkinformation-retrievalinformation-extractionwebspidercrawlingcrawlerscrawlextract-data
  Language:Java 31

• herrbischoff / user-agents

  User agent database in JSON format of bots, crawlers, certain malware, automated software, scripts and uncommon ones.

  user-agentjsonbotscrawlersmalwareautomationdatabasedata
  Language:Shell 27

• zcrawl / zcrawl

  An open source web crawling platform

  web-crawlingwebcrawlinggolangcrawlersscrapingcrawling
  Language:Go 22

• p0dalirius / crawlersuseragents

  Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.

  pentesttooluser-agentrequestwebcrawlerbugbountycrawlers
  Language:Python 20

• anapaulagomes / licitacoes-de-feira

  Licitações de Feira de Santana de fácil acesso aos cidadãos 🏦

  open-datatransparenciadados-abertoscrawlers
  Language:Python 18

• delvelabs / htcap

  htcap is a web application scanner able to crawl single page application (SPA) in a recursive manner by intercepting ajax calls and DOM changes.

  crawlercrawlerscrawlingjavascript-crawlerscannerscanningsecurity-auditsingle-page-appsingle-page-applicationsvulnerability-detectionvulnerability-scanners
  Language:Python 18

• robertciotoiu / mobile-de-car-data-collector

  Crawl, scrape and persist Mobile.de car listings data in a smart & responsible way

  crawlercrawlerscrawlingdockerjavajava19jsoupkubernetesmavenparsingresponsible-scrapingscraperscrapersscrapingsitemap
  Language:Java 17

• ElektroStudios / Google-Search-URL-Crawler

  Desktop app that crawls urls from Google's search engine results

  crawlcrawlercrawlerscrawlingdotnetgooglegoogle-crawlergoogle-searchgooglesearchhackingsearchsearch-enginesearchertooltoolsurlurl-crawlervbnetwindowswinforms
  Language:Visual Basic .NET 16

• arquejadalucy / jus_crawler

  API que busca dados de um processo em todos os graus dos Tribunais de Justiça de Alagoas (TJAL) e do Ceará (TJCE).

  cerberuscrawlersdeta-spacefastapijinja2-templatesjuridicopytestpython
  Language:Python 13

• BryanMorgan / isbot

  Rust library to detect bots using a user-agent string

  botscrawlersuser-agentdevice-detectionbot-detectioncrawler-detectionuser-agent-detection
  Language:Rust 13

• Hsins / Daily-GitHub-Trending

  📰 Fetch daily trending repositories information on GitHub Trending Page by script writen in JavaScript and executed with GitHub Actions Service.

  github-actionsgithub-actions-javascriptcrawlers
  Language:JavaScript 13

• solidusio-contrib / solidus_sitemap

  Provide a sitemap of your Solidus store.

  solidussitemapecommercegooglecrawlersproduct
  Language:Ruby 13

• shaoxiongdu / SkyEye

  一个基于SpringBoot的全网热点爬虫项目，原始热搜数据会入库，分词统计会存入Redis。方便之后的数据分析。

  crawlercrawlersmysqlredisspringspring-boot
  Language:Java 11

• tranlv / wiki-link

  Scraping the wiki pages and find the minimum number of links between two wiki pages

  applicationcrawlerspythonpython3machine-learningwebcrawlwebcrawlerwikipedia
  Language:Python 11

• BaseMax / StackoverflowCrawler

  A web crawler which crawls the stackoverflow website.

  stackoverflow-crawlercrawlercrawlerscrawler-testingcrawler-detectorcrawler-pythonpython-crawlerweb-crawlerweb-crawler-pythontest-crawlerstackoverflowstackoverflow-questionsstackoverflow-apistackoverflow-answerstackoverflow-tagsstackoverflow-getstackoverflow-analysecrawlingtext-processingtext-processor
  Language:Python 10

• versioneye / crawl_r

  VersionEye crawlers implemented in Ruby.

  versioneyerubycrawlers
  Language:Roff 10
• born2crawl

  arthur3486 / born2crawl

  A highly performant and versatile crawling engine, designed with scalability and extensibility in mind.

  crawlercrawlerscrawlingjavajvmkotlinlibraryscraperscrapingweb-crawlerweb-crawlingcrawling-enginedata-miningminingweb-scraping
  Language:Kotlin 8

• BaseMax / StockExchangeCrawler

  A crawler program to extract all of the data and the price for symbols in the global stock exchange.

  stockstock-marketstock-pricesstock-datastock-price-predictionstock-tradingstock-analysisstock-predictionstock-exchangestock-exchangesstock-exchange-simulatorstock-exchange-platformstock-exchange-crawlercrawlercrawlingcrawling-sitescrawlerscrawling-frameworkcrawl-pagescrawlspider
  Language:PHP 8

• FEZIRO / wechat-miniprogram-spider-demo

  微信小程序云开发网络爬虫教程

  axioscheeriocloudfunctioncrawlers
  Language:JavaScript 8

• romis2012 / is-bot

  Detect bots/crawlers/spiders via user-agent string

  botscrawlerspythonuser-agentuser-agent-parserweb-crawlersbot-detection
  Language:Python 8