Xciyang / spiking

A lightweight web crawler.

Spiking

Chinese | English

A common web crawler tool developed using Node.js.

Advantage

• Graphical Interface (Electron)
• Custom initial link
• Headless browser support (puppeteer)
• Concurrent / multithreaded
• Efficient crawling mechanism
• Automatic similarity
• Progress visualization
• Using a proxy

Getting started

git clone https://github.com/Xciyang/spiking.git

cd spiking

npm install

npm start

Usage

Download the latest [Release] (https://github.com/Xciyang/spiking/releases) version.

Please read usage: English(en_US)

To-do list

• Localization
• Tasks Saving
• Custom filtering method

More

Electron

puppeteer