Issue upon change on the S3 hosted config.yaml
emiliobasualdo opened this issue · comments
Describe the bug
This error seems to be from the Cryptofeed library, but the trace from Cryptostore will probably help tp solve it.
When the configuration hosted on S3(as per env.s3config) is changed, cryptostore & cryptofeed successfully manage to change the configurations.
None the less, a small error seems to be occurring in Cryptofeed.
To Reproduce
(I have only tried this with the config.yaml file hosted on S3, not locally)
Steps to reproduce the behavior:
1- configure env.s3config to load the config.yaml from s3 endpoint
2- run cryptostore
3- change the file on S3(in my case i'm adding a symbol to the l2_books in Binance)
- Provide config (at least as much as is relevant)
# Cryptostore sample config file
cache: redis
redis:
ip: redisdb
port: 6379
socket: null
del_after_read: true
retention_time: null
start_flush: true
symbols: &symbols [ETH-USDT, ETH-EUR]
exchanges:
BINANCE:
retries: -1
l2_book:
symbols: *symbols
max_depth: 500
book_delta: true
ticker: *symbols
storage: [parquet]
storage_retries: 5
storage_retry_wait: 30
storage_interval: 69
parquet:
del_file: true
path: ./temp
file_format: [timestamp]
prefix_date: false
S3:
endpoint: null
key_id: x
secret: x
bucket: x
prefix: null
- redis or kafka?
redis - What behavior did you see?
Error output, thou the program continues to execute and successfully save the requested information. - Provide any tracebacks if applicable
cryptostore_2_1 | 2021-02-24 18:24:57,915 : INFO : ticker-BINANCE-ETH-USDT: Removed 68 entries through id 1614191096812-0
cryptostore_2_1 | 2021-02-24 18:24:57,916 : INFO : Write Complete BINANCE-ticker-ETH-USDT
cryptostore_2_1 | 2021-02-24 18:25:55,063 : INFO : loading config from s3 s3://historical-crypto-bucket-pailor/config/config-docker-2.yaml at 2021-02-24 18:25:55.063401
cryptostore_2_1 | 2021-02-24 18:25:55,069 : INFO : applying config file from S3 s3://historical-crypto-bucket-pailor/config/config-docker-2.yaml at 2021-02-24 18:25:55.069098
cryptostore_2_1 | 2021-02-24 18:25:55,132 : INFO : loading config from s3 s3://historical-crypto-bucket-pailor/config/config-docker-2.yaml at 2021-02-24 18:25:55.132624
cryptostore_2_1 | 2021-02-24 18:25:55,138 : INFO : applying config file from S3 s3://historical-crypto-bucket-pailor/config/config-docker-2.yaml at 2021-02-24 18:25:55.138685
cryptostore_2_1 | 2021-02-24 18:25:55,138 : INFO : start: ['BINANCE'] stop: ['BINANCE']
cryptostore_2_1 | 2021-02-24 18:25:55,140 : INFO : message: {"op": "stop", "exchange": "BINANCE"}
cryptostore_2_1 | 2021-02-24 18:25:55,140 : INFO : Terminating BINANCE
cryptostore_2_1 | 2021-02-24 18:25:55,140 : INFO : message: {"op": "start", "exchange": "BINANCE", "collector": {"retries": -1, "l2_book": {"symbols": ["ETH-USDT", "ETH-EUR"], "max_depth": 500, "book_delta": true}, "ticker": ["ETH-USDT", "ETH-EUR"]}, "config": {"cache": "redis", "redis": {"ip": "redisdb", "port": 6379, "socket": null, "del_after_read": true, "retention_time": null, "start_flush": true}, "symbols": ["ETH-USDT", "ETH-EUR"], "storage": ["parquet"], "storage_retries": 5, "storage_retry_wait": 30, "storage_interval": 69, "parquet": {"del_file": true, "path": "./temp", "file_format": ["timestamp"], "prefix_date": false, "S3": {"endpoint": null, "key_id": "x", "secret": "x", "bucket": "historical-crypto-bucket-pailor", "prefix": null}}}}
cryptostore_2_1 | 2021-02-24 18:25:55,141 : INFO : Starting {'op': 'start', 'exchange': 'BINANCE', 'collector': {'retries': -1, 'l2_book': {'symbols': ['ETH-USDT', 'ETH-EUR'], 'max_depth': 500, 'book_delta': True}, 'ticker': ['ETH-USDT', 'ETH-EUR']}, 'config': {'cache': 'redis', 'redis': {'ip': 'redisdb', 'port': 6379, 'socket': None, 'del_after_read': True, 'retention_time': None, 'start_flush': True}, 'symbols': ['ETH-USDT', 'ETH-EUR'], 'storage': ['parquet'], 'storage_retries': 5, 'storage_retry_wait': 30, 'storage_interval': 69, 'parquet': {'del_file': True, 'path': './temp', 'file_format': ['timestamp'], 'prefix_date': False, 'S3': {'endpoint': None, 'key_id': 'x', 'secret': 'x', 'bucket': 'historical-crypto-bucket-pailor', 'prefix': None}}}}
cryptostore_2_1 | Process Collector-1:1:
cryptostore_2_1 | Traceback (most recent call last):
cryptostore_2_1 | File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
cryptostore_2_1 | self.run()
cryptostore_2_1 | File "/cryptostore/collector.py", line 122, in run
cryptostore_2_1 | fh.run()
cryptostore_2_1 | File "/usr/local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 269, in run
cryptostore_2_1 | self.stop(loop=loop)
cryptostore_2_1 | File "/usr/local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 286, in stop
cryptostore_2_1 | task.set_name(f'shutdown_feed_{feed.id}')
cryptostore_2_1 | AttributeError: '_asyncio.Task' object has no attribute 'set_name'
cryptostore_2_1 | 2021-02-24 18:25:55,146 : INFO : Collector for BINANCE running on PID 48
cryptostore_2_1 | 2021-02-24 18:26:06,891 : INFO : l2_book-BINANCE-ETH-USDT: Removed 647 entries through id 1614191166516-0
cryptostore_2_1 | 2021-02-24 18:26:06,893 : INFO : Write Complete BINANCE-l2_book-ETH-USDT
cryptostore_2_1 | 2021-02-24 18:26:06,894 : INFO : Reading BINANCE-l2_book-ETH-EUR
cryptostore_2_1 | 2021-02-24 18:26:06,901 : INFO : l2_book-BINANCE-ETH-EUR: Read 90 messages from Redis
As you can see ETH-EUR was added and correctly executed.
Operating System:
Linux
Cryptofeed Version
Latest: Installed with Dockerfile -> RUN pip install --no-cache-dir git+https://github.com/bmoscon/cryptofeed.git
what version of python are you using?
python:3.7.3-stretch
As per Dockerfile -> FROM python:3.7.3-stretch...
ok, set_name is python 3.8+ only, ill take a look
Again, thanks for the help. The library is great.
this has been fixed on cryptofeed