evido3s / scrapydweb

Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported

ScrapydWeb: Full-featured web UI for monitoring and controlling Scrapyd servers cluster, with Scrapy log analysis and visualization supported

Features

• Multinode Scrapyd Servers

  • Group, filter and select any numbers of nodes
  • Execute command on multinodes with one click

• Scrapy Log Analysis

  • Stats collection
  • Progress visualization
  • Logs categorization

• All Scrapyd API Supported

  • Deploy project, Run Spider, Stop job
  • List projects/versions/spiders/running_jobs
  • Delete version/project

• Others

  • Basic auth for web UI
  • Accessing Scrapyd servers protected by basic auth

Maintainer

• my8100
• simplety (Front-End)

Installation

To install ScrapydWeb, simply use pip:

$ pip install scrapydweb

Start Up

Run "scrapydweb -h" to get help, and a config file named "scrapydweb_settings.py" would be copied to the working directory, then you can custom config with it.

$ scrapydweb

Visit http://127.0.0.1:5000

Screenshot

• Overview

• Dashboard

• Log Analysis

  • Stats collection

  • Progress visualization

  • Logs categorization

• Deploy Project

• Run Spider

• Manage Projects