crawler

There are 401 repositories under crawler topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  crawlercrawlingframeworkhacktoberfestpythonscrapingweb-scrapingweb-scraping-python
  Language:Python 52152

• NaiboWang / EasySpider

  A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

  batch-processingbatch-scriptcode-freecrawlerdata-collectionfrontendguihtmlinput-parameterslaymanparametersroboticsrpascraperspidervisualvisualizationvisualprogrammingwebwww
  Language:JavaScript 33059

• iawia002 / lux

  👾 Fast and simple video download library and CLI tool written in Go

  downloadergocrawlerscrapervideobilibiliyoutubeyoukuiqiyitumblrqqgolangdownload
  Language:Go 26616

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  crawlercrawlingframeworkgogolangscraperscrapingspider
  Language:Go 22898

• jhao104 / proxy_pool

  Python ProxyPool for web spider

  crawlerhttpproxyredisspider
  Language:Python 21190

• binux / pyspider

  A Powerful Spider(Web Crawler) System in Python.

  pythoncrawler
  Language:Python 16450
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  web-scrapingweb-crawlingnpmheadless-chromepuppeteerautomationapifyscrapingcrawlingcrawlerheadlessscraperweb-crawlerjavascriptnodejsplaywrighttypescript
  Language:TypeScript 14615

• codelucas / newspaper

  newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

  crawlercrawlingnewsnews-aggregatorpythonscraper
  Language:Python 13994

• shengqiangzhang / examples-of-web-crawlers

  一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

  crawlerspidertaobaotmallexamplepythonseleniumpyquerystockfundmultithreadingagent-poolwechatwechat-reportwereader
  Language:Python 13742

• code4craft / webmagic

  A scalable web crawler framework for Java.

  crawlerjavascrapingframework
  Language:Java 11365
• crawlab

  crawlab-team / crawlab

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  crawlabcrawlercrawling-tasksdockergoplatformscrapyscrapyd-uispiderspiders-managementweb-crawlerwebcrawlerwebspider
  Language:Go 11172

• s0md3v / Photon

  Incredibly fast crawler designed for OSINT.

  crawlerinformation-gatheringosintpythonspider
  Language:Python 10803

• projectdiscovery / katana

  A next-generation crawling and spidering framework.

  clicrawlergocrawlerheadlessspider-frameworkweb-spider
  Language:Go 10739

• injetlee / Python

  Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

  crawlerexcelpythonwechat
  Language:Python 9507

• ssssssss-team / spider-flow

  新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

  crawlerjsoupspiderspider-flowweb-crawlerweb-spiderwebcrawlerwebspiderxpath
  Language:Java 9424

• guyueyingmu / avbook

  AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

  adultadult-videoavmoocrawlerdatabaseguzzlehttpjavbusjavlibrarylaravelmagnetmagnet-linkscraperspider
  Language:PHP 9420

• mendableai / firecrawl

  🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

  aiai-scrapingcrawlerdatahtml-to-markdownllmmarkdownragscraperscrapingweb-crawler
  Language:TypeScript 9193
• Douyin_TikTok_Download_API

  Evil0ctal / Douyin_TikTok_Download_API

  🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

  pythonpywebiotiktokdouyinapiscraperfastapino-watermarkonline-parsingasyncdouyin-tiktok-apidouyin-tiktok-downloadcrawlerspiderweb-scrapingasgiasynciotiktok-scraperdouyin-scraperhttpx
  Language:Python 8423

• bda-research / node-crawler

  Web Crawler/Spider for NodeJS + server-side jQuery ;-)

  cheeriocrawlerextract-datajavascriptjquerynodejsspider
  Language:TypeScript 6672

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 6521

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  web-crawlercrawlerweb-scraperspidernode-crawlerscraperawesome
  6330
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
  Language:Python 6133

• chyroc / WechatSogou

  基于搜狗微信搜索的微信公众号爬虫接口

  crawlerpypipythonscrapysogouwechat
  Language:Python 5869
• ferret

  MontFerret / ferret

  Declarative web scraping

  golangquery-languagedata-miningscrapingscraping-websitesdslcdpcrawlingscrapercrawlergochromeclitoollibraryhacktoberfest
  Language:Go 5688

• rmax / scrapy-redis

  Redis-based components for Scrapy.

  scrapycrawlerdistributedredis
  Language:Python 5503

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  headless-chromepuppeteerjquerycrawlercrawlingscraperscrapingchromechromiumpromise
  Language:JavaScript 5500

• SpiderClub / haipproxy

  :sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

  crawlerdistributedhigh-availabilityipproxyredisschedulerscrapyspider
  Language:Python 5424

• DropsDevopsOrg / ECommerceCrawlers

  实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

  python3crawlerbaidu-tiebataobao-spiderdazhong-spiderdouban-moviedouban-musicalitaskbaotuquanjingxianyusohufofawechatbaiduzhilianzhaopinlagoubossscrapyctrip
  Language:Python 4640

• madawei2699 / myGPTReader

  A community-driven way to read and chat with AI bots - powered by chatGPT.

  chatgptslack-botpromptcrawlerembeddinggpt-35-turboscraperaireaderhot-newsdaily-newsopenai
  Language:Python 4421

• niespodd / browser-fingerprinting

  Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

  automationbotbot-detectionbrowser-fingerprintingchromedriverchromiumchromium-browsercrawlerdetectionfingerprintingpuppeteerrecaptchascraperspiderstealthwebwebscraping
  Language:JavaScript 4031

• dotnetcore / DotnetSpider

  DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

  crawlerdotnetcorecross-platformcsharpdistributed
  Language:C# 3952

• imWildCat / scylla

  Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

  crawlerpython3scyllaproxy-poolpython
  Language:Python 3926

• HiddenStrawberry / Crawler_Illegal_Cases_In_China

  Collection of China illegal cases about web crawler 本项目用来整理所有**大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在**大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

  chinacrawlerlaw
  Language:HTML 3816

• zu1k / proxypool

  Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes

  crawlerproxypool
  Language:Go 3811

• constverum / ProxyBroker

  Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

  anonymityanonymouscrawlerhttp-proxyprivacyproxiesproxyproxy-checkerproxy-listproxy-serverproxypoolsocks
  Language:Python 3804

• Arachni / arachni

  Web Application Security Scanner Framework

  arachnidomrubyauditdetectionsecurity-auditanalysismodularjavascriptscannersweb-applicationvulnerability-detectioncrawlerscannerhackhackingpenetration-testingxsssql-injection
  Language:Ruby 3741