scrapy

There are 118 repositories under scrapy topic.

• crawlab

  crawlab-team / crawlab

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  webcrawlerscrapycrawlabspiders-managementgoscrapyd-uispidercrawlerwebspiderweb-crawlerdockerplatformcrawling-tasks
  Language:Go 12048

• lining0806 / PythonSpiderNotes

  Python入门网络爬虫之精华版

  pythonzhihucaptchawechatscrapyseleniumcookie
  Language:Python 7336

• chyroc / WechatSogou

  基于搜狗微信搜索的微信公众号爬虫接口

  wechatsogoupythoncrawlerpypiscrapy
  Language:Python 6140

• rmax / scrapy-redis

  Redis-based components for Scrapy.

  crawlerdistributedredisscrapy
  Language:Python 5645

• SpiderClub / haipproxy

  :sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

  crawlerdistributedhigh-availabilityipproxyredisschedulerscrapyspider
  Language:Python 5531

• DropsDevopsOrg / ECommerceCrawlers

  实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

  python3crawlerbaidu-tiebataobao-spiderdazhong-spiderdouban-moviedouban-musicalitaskbaotuquanjingxianyusohufofawechatbaiduzhilianzhaopinlagoubossscrapyctrip
  Language:Python 5285

• nghuyong / WeiboSpider

  持续维护的新浪微博采集工具🚀🚀🚀

  pythonscrapyweiboweibospider
  Language:Python 3972

• Gerapy / Gerapy

  Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

  dashboarddistributeddjangodockergerapyscrapyscrapydspidervuevuejswebspider
  Language:Python 3488
• feapder

  Boris-code / feapder

  🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

  crawlerfeapderfeaplatpythonscrapyspider
  Language:Python 3459

• my8100 / scrapydweb

  Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

  scrapyscrapydscrapyd-uiscrapyd-apiscrapyd-adminscrapyd-managelog-parsinglog-analysisscrapyd-monitorscrapyd-keeperscrapyd-controlscrapy-log-analysisscrapyd-log-analysisscrapy-visualizationscrapyd-visualizationdashboardspiderscrapyd-cluster-management
  Language:Python 3348

• wkunzhi / Python3-Spider

  Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

  scrapypythoncrawlcrawlergeekspidertaobaodianpingmeituanseleniumpyppeteersplashscrapy-crawler
  Language:Python 3284

• scrapy-plugins / scrapy-splash

  Scrapy+Splash for JavaScript integration

  scrapyheadless-browsers
  Language:Python 3236

• CodeRayZhang / Movie_Recommend

  基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

  spark-mllibspark-streamingssm-mavenscrapyscalahadoopnginxhivemysql
  Language:Java 2978

• DormyMo / SpiderKeeper

  admin ui for scrapy/open source scrapinghub

  dashboardscrapyscrapy-uiscrapydscrapyd-dashboardscrapyd-uispider
  Language:Python 2769

• QianyanTech / Image-Downloader

  Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

  image-downloadergoogle-imagesbingbaidugooglepyqtscrapyspider
  Language:Python 2320

• librauee / Reptile

  🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish

  python3requestsscrapyspider
  Language:Python 1704

• TheWebScrapingClub / webscraping-from-0-to-hero

  The web scraping open project repository aims to share knowledge and experiences about web scraping with Python

  playwrightpythonscrapyscrapy-spiderscrapysplashwebscraping
  1687

• eliasdabbas / advertools

  advertools - online marketing productivity and analysis tools

  marketingadvertisingadwordspythondigital-marketingonline-marketingkeywordssearch-engine-marketingtwitter-apisearch-engine-optimizationseoserpsocial-mediayoutuberobots-txtscrapyseo-crawlerlog-analysisgoogle-adslogfile-parser
  Language:Python 1296

• scrapy-plugins / scrapy-playwright

  🎭 Playwright integration for Scrapy

  playwrightplaywright-pythonscrapypythonpython3python-asynciojavascript-rendererheadless-browserchrome-headlessfirefox-headlesswebkit-headlesshacktoberfest
  Language:Python 1291

• kkoooqq / fakebrowser

  🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

  stealthheadlessautomationcrawlerscrapybrowser-fingerprintanti-bot-detectionbotpuppeteerpuppeteer-extraanti-fingerprintingpuppeteer-extra-pluginfakespoofcheat
  Language:JavaScript 1287

• ityard / python-fxxk-spider

  收集各种免费的 Python 爬虫项目

  crawlerpythonrequestsscrapyspider
  1234

• istresearch / scrapy-cluster

  This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

  distributedkafkapythonredisscrapingscrapy
  Language:Python 1218

• holgerd77 / django-dynamic-scraper

  Creating Scrapy scrapers via the Django admin interface

  pythondjangoscraperscrapingscrapyspiderwebscraping
  Language:Python 1160

• juancarlospaco / faster-than-requests

  Faster requests on Python 3

  pythonpython3python-libraryhttp-requestspython-requestsweb-scrapingdownload-filespeedcurlcythonurllibfaster-than-requestsopen-dataurllib3requests3scrapyndjsonhigh-performancerequests-toolbeltweb-scraper
  Language:Nim 1126

• xingag / spider_python

  python爬虫

  pythonpython3urllibrequestsscrapyxpathbs4
  Language:Python 1095

• bytebuff / JSpider

  JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

  javascriptnodejspython3scrapyspider
  Language:JavaScript 1093

• moyada / stealer

  抖音、快手、火山、皮皮虾，视频去水印程序

  bilibilibilibili-downloaddouyindouyin-downloadkuaishoupipixiascrapy
  Language:Python 1075

• jonbakerfish / TweetScraper

  TweetScraper is a simple crawler/spider for Twitter Search without using API

  twitter-searchscrapytweetstwitter
  Language:Python 1051

• vifreefly / kimuraframework

  Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

  crawlerheadless-chromekimuraiscraperscrapy
  Language:Ruby 1011

• eracle / linkedin

  Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

  scrapyselenium-webdriverbotscraperscrapinglinkedinchromium-browserdockerdocker-compose
  Language:Python 976

• alanchn31 / Data-Engineering-Projects

  Personal Data Engineering Projects

  data-lakeingest-datadata-engineeringdata-warehousecassandraaws-redshiftmongodbscrapysparkairflowpostgresdata-engineering-nanodegreestar-schemadata-modeling
  Language:Jupyter Notebook 961

• clemfromspace / scrapy-selenium

  Scrapy middleware to handle javascript pages using selenium

  scrapyseleniumcrawling
  Language:Python 954

• mtianyan / FunpySpiderSearchEngine

  Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

  pythonscrapyspiderelasticsearchdjangozhihulagouelasticsearch-analysis-ikredismysqlsearch-engine
  Language:Python 938

• hellock / icrawler

  A multi-thread crawler framework with many builtin image crawlers provided.

  bing-imagecrawlerflickr-apigoogle-imagespythonscrapyspider
  Language:Python 903

• scrapinghub / scrapyrt

  HTTP API for Scrapy spiders

  pythoncrawlingcrawlerscrapyscrapertwistedwebcrawlingwebcrawlerhacktoberfesthacktoberfest2021
  Language:Python 871

• tb0hdan / domains

  World’s single largest Internet domains dataset

  collydatasetinternet-domainsscrapysearch-enginesyacy
  Language:JavaScript 816