crawl

There are 13 repositories under crawl topic.

• kangvcar / InfoSpider

  INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、**移动、**联通、**电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源**博客、简书。

  automationchromecrawlcsdnhotmailoutlookpython3seleniumspidertkinterwxpython
  Language:Python 8128

• 201206030 / novel-plus

  novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、充值订阅、新闻发布等功能。

  bookcrawlnovelreadspider
  Language:Java 4329

• wkunzhi / Python3-Spider

  Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

  crawlcrawlerdianpinggeekmeituanpyppeteerpythonscrapyscrapy-crawlerseleniumspidersplashtaobao
  Language:Python 3284

• any4ai / AnyCrawl

  AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

  aitoolscrawlscrapeserpwebscraperai-scrapingdatahtml-to-markdownragscraping
  Language:TypeScript 2372

• ReaJason / xhs

  基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

  crawlpythonxhs
  Language:Python 1889

• coder-hxl / x-crawl

  Flexible Node.js AI-assisted crawler library

  aiai-crawlchromiumcrawlcrawlerfingerprintflexiblejavascriptmultifunctionnodejspuppeteerspidertypescript
  Language:TypeScript 1758

• ArchiveTeam / grab-site

  The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  archivingcrawlspidercrawlerwarc
  Language:Python 1537

• zhuweiyou / weixin-game-helper

  微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星途WeGoing、猜画小歌、知乎答题王、腾讯**象棋、跳一跳、题多多黄金版）

  weixingamehelperrobotcrawlnodejsmongodbanyproxycharlesfiddlersocketpython
  Language:JavaScript 1388

• darbra / sperm

  浏览过的精彩逆向文章汇总，值得一看

  fridaunidbgcrawlerspidercrawl
  1349

• LoseNine / Crack-JS-Spider

  JS破解逆向，破解JS反爬虫加密参数，已破解极验滑块w（2022.2.19），QQ音乐sign（2022.2.13），拼多多anti_content，boss直聘zp_token，知乎x-zse-96，酷狗kg_mid/dfid，唯品会mars_cid，**裁判文书网（2020-06-30更新），淘宝密码，天安保险登录，b站登录，房天下登录，WPS登录，微博登录，有道翻译，网易登录，微信公众号登录，空中网登录，今目标登录，学生信息管理系统登录，共赢金融登录，重庆科技资源共享平台登录，网易云音乐下载，一键解析视频链接，财联社登录。

  crawlspiderjscrack
  Language:JavaScript 963

• rugantio / fbcrawl

  A Facebook crawler

  scrapycrawlerfacebookcrawlspiderpythonscraper
  Language:Python 684

• liip / TheA11yMachine

  The A11y Machine is an automated accessibility testing tool which crawls and tests pages of any web application to produce detailed reports.

  testaccessibilitywcagcrawl
  Language:JavaScript 628

• markowanga / stweet

  Advanced python library to scrap Twitter (tweets, users) from unofficial API

  twitterapipythontweetsunofficialsearchcrawlscraperscrapperscrapetwintscrap-tweettwitter-apiuseruserssearchrunnertweetscrap
  Language:Python 613

• philschmid / clipper.js

  HTML to Markdown converter and crawler.

  crawlhtml-to-markdownmarkdownnlpretrieval-augmented-generationsearch
  Language:TypeScript 595
• Pinkerton

  000pp / Pinkerton

  🕵️ Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded password or related.

  crawlcrawlerjavascriptpentestpythonredteampython3secretshacktoberfest
  Language:Python 394

• zkqiang / zhihu-login

  知乎模拟登录，支持提取验证码和保存 Cookies

  zhihupythonexecjslogincrawlerspidercrawlcookie
  Language:Python 361

• yaroslaff / nudecrawler

  Crawl telegra.ph searching for nudes!

  nudenudity-detectioncrawlcrawlerfindnsfwnsfw-recognitionnudesonlyfanspythonpython3searchspidertitstelegra-phscrapescraperscrapingweb-scrapingwebscraping
  Language:Python 341

• darbra / geetest

  geetest，滑动验证码

  spidercrawlgeetest
  Language:Python 314

• justoneapi / data-api

  justoneapi数据接口服务。提供：淘宝、小红书、拼多多、同程旅行、京东外卖、抖音（电商）、美团、抖音（视频）、快手、蒲公英、星图、微信公众号、大众点评、哔哩哔哩、知乎、微博、贝壳、Bigo、Temu、Lazada、Shopee、SHEIN、百度指数、携程、Boss直聘、智联招聘、拉钩、今日头条、Facebook、Youtube、Instgram、Twitter。爬虫、采集、scrapy、接口、API。

  apicrawldatawebcrawlingdouyinjingdongkuaishoutaobaotaobao-apixiaohongshuxiaohongshu-apipugongying
  306

• zhangslob / awesome_crawl

  腾讯新闻、知乎话题、微博粉丝，Tumblr爬虫、斗鱼弹幕、妹子图爬虫、分布式设计等

  pythonawesomecrawlredisscrapyscrapy-rediszhihutencentweibotumblrtumblr-botdouyuwebsocketsgolang
  Language:Python 297

• spatie / laravel-site-search

  Create a full-text search index by crawling your site

  laravelphpsitesearchcrawlhacktoberfest
  Language:PHP 293
• SpideyX

  RevoltSecurities / SpideyX

  SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

  crawlcrawlercrawler-enginecrawlingcrawling-frameworkcrawling-sitescrawling-tool
  Language:Python 178

• adamdehaven / fetchurls

  A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.

  wgetcrawlspiderbash-scriptingshell-scriptwebsiteurls
  Language:Shell 131

• dli98 / geetest

  滑动验证码，希望对你们有所帮助❤️

  spidercrawlgeetestpython3bilibili
  Language:Python 131

• ArchiveTeam / wget-lua

  Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

  webarchivingwarcwgetluaarchivingcrawlercrawlcrawlingspiderarchiveteamwget-luazstdftpscraperscrapingcrawlersdownloader
  Language:C 129

• glouw / andvaranaut

  A dungeon crawler

  dungeoncrawlercrawl
  Language:C 124

• WwwwwyDev / crawlist

  A universal solution for web crawling lists. 抓取网页列表的通用解决方案

  crawlcrawlercrawler-pythonpythonreptilecrawling-pythoncrawlist
  Language:Python 110

• monkey-soft / Scrapy_IPProxyPool

  免费 IP 代理池。Scrapy 爬虫框架插件

  scrapyipproxyproxypoolspidercrawlercrawlschedule
  Language:Python 103

• zhao94254 / pspider

  一个简单的分布式爬虫框架

  pythoncrawlspiderspider-frameworkflask-apicelery
  Language:Python 101

• jgravelle / groqcrawl

  GroqCrawl is a powerful and user-friendly web crawling and scraping application built with Streamlit and powered by PocketGroq. It provides an intuitive interface for extracting LLM friendly AI consumable content from websites, with support for single-page scraping, multi-page crawling, and site mapping.

  crawlgroqscrapewebweb-crawlerweb-scraping
  Language:Python 99

• zkqiang / crawler-chrome-extensions

  爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

  awesomechrome-extensionchrome-extensionscrawlerspiderscraperpythonawesome-listscrapingcrawl
  91

• zongdeiqianxing / WebSecurityArticles

  爬取及整理Freebuf\安全客\先知\知道创宇等站点的”web安全“类优质文章

  crawlcrawlerfreebufxianzhivulhubanquankesecurityarticlesseebugleavesongsweb
  Language:Python 84

• mangenotwork / gathertool

  gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。

  golanggathertoolspidercrawlspider-frameworkgrpc-goproxy-server
  Language:Go 55

• Swader / diffbot-php-client

  [Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

  diffbotphpcrawlingcrawlscrapescrapingscraperscraped-datamachine-learningnlpaiartificial-intelligencebot
  Language:PHP 53

• swimmingkiim / n8n-nodes-crawl-and-scrape

  n8n custom node to crawl and scrape website with crawlee

  crawln8nn8n-community-node-packagen8n-nodesscrape
  Language:TypeScript 51

• tavily-ai / tavily-chat

  Conversational agent that fuses chat data with live web results through Tavily search, extract, and crawl.

  agentic-aicrawlllmscrapesearch
  Language:TypeScript 46