keyplay / crypto-crawlers

The All-in-one Solution to crawl data from cryptocurrency exchanges

crypto-crawlers

The All-in-one Solution to crawl data from cryptocurrency exchanges.

Download Data

For most people, don't bother to run crawlers, I'm publishing data to the following places:

1. BaiduNetDisk https://pan.baidu.com/s/187YGeS5LHuJruq57zZLvmg, code: 3142.
2. BitTorrent. See the table below.

Download Links:

File Name	Size	Magnet Link	Comment
trade-2020-04.zip	1.2G	magnet:?xt=urn:btih:0687d9b0cd1fea2908131845c6adb83796cdeef7&dn=trade-2020-04.zip&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969	April 2020
trade-2020-05.zip	12G	magnet:?xt=urn:btih:029302fcc40a9a8bf322933da22aacf85744d116&dn=trade-2020-05.zip&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969	May 2020
trade-20200601-20200611.zip	1.7G	magnet:?xt=urn:btih:677fb843cda2c24b5516b85ea7b06ec5f3503e50&dn=trade-20200601-20200611.zip&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969	From 20200601 to 20200611

The more seeders, the faster speed. Please help each other by seeding.

How to run

1. Prerequisites

1. Install NodeJS
2. Run a Redis server at localhost:6379
3. Change the DATA_DIR variable in .envrc and load this file via direnv

2. Install typescript and pm2 globally

npm install typescript pm2 -g

3. Compile

npm install
npm run build

4. Run all crawlers

pm2 start pm2.misc.config.json
pm2 start pm2.trade.config.js

How to contribute

Open this project in VSCode and you're all set.