Launch errors
dawiepoolman opened this issue · comments
Hi guys
Sorry if this is a noob question but I get these errors when I run it first time.
PS - Please add Luno exchange?
Error:
C:\cryptostore>cryptostore
2021-02-24 20:38:59,492 : INFO : Starting cryptostore
2021-02-24 20:38:59,493 : INFO : Cryptostore running on PID 20444
2021-02-24 20:38:59,505 : INFO : Spawner started
2021-02-24 20:38:59,516 : INFO : Aggregator started
2021-02-24 20:38:59,518 : INFO : Cryptostore started
2021-02-24 20:38:59,568 : INFO : start: ['COINBASE', 'BITMEX'] stop: []
2021-02-24 20:39:01,114 : INFO : Spawner running on PID 15348
2021-02-24 20:39:01,115 : INFO : message: {"op": "start", "exchange": "COINBASE", "collector": {"retries": -1, "l3_book": {"symbols": ["BTC-USD"], "book_delta": true, "book_interval": 100000}, "trades": ["BTC-USD", "ETH-USD", "ETH-BTC"], "ticker": ["BTC-USD"]}, "config": {"cache": "redis", "kafka": {"ip": "127.0.0.1", "port": 9092, "start_flush": true}, "redis": {"ip": "127.0.0.1", "port": 6379, "socket": null, "del_after_read": true, "retention_time": null, "start_flush": true}, "storage": ["arctic", "influx"], "storage_retries": 5, "storage_retry_wait": 30, "pass_through": {"type": "zmq", "host": "127.0.0.1", "port": 5678}, "elastic": {"host": "http://127.0.0.1:9200", "user": null, "token": null, "shards": 10, "replicas": 0, "refresh_interval": "30s"}, "influx": {"host": "http://127.0.0.1:8086", "db": "example", "create": true, "username": "None", "password": "None"}, "parquet": {"del_file": true, "append_counter": 4, "file_format": ["exchange", "pair", "data_type", "timestamp"], "compression": {"codec": "BROTLI", "level": 6}, "path": null, "prefix_date": false, "S3": {"endpoint": null, "key_id": null, "secret": null, "bucket": null, "prefix": null}, "GCS": {"service_account": null, "bucket": null, "prefix": null}, "GD": {"service_account": null, "prefix": null, "folder_name_sep": "."}}, "arctic": "mongodb://127.0.0.1", "storage_interval": 60}}
2021-02-24 20:39:01,117 : INFO : Starting {'op': 'start', 'exchange': 'COINBASE', 'collector': {'retries': -1, 'l3_book': {'symbols': ['BTC-USD'], 'book_delta': True, 'book_interval': 100000}, 'trades': ['BTC-USD', 'ETH-USD', 'ETH-BTC'], 'ticker': ['BTC-USD']}, 'config': {'cache': 'redis', 'kafka': {'ip': '127.0.0.1', 'port': 9092, 'start_flush': True}, 'redis': {'ip': '127.0.0.1', 'port': 6379, 'socket': None, 'del_after_read': True, 'retention_time': None, 'start_flush': True}, 'storage': ['arctic', 'influx'], 'storage_retries': 5, 'storage_retry_wait': 30, 'pass_through': {'type': 'zmq', 'host': '127.0.0.1', 'port': 5678}, 'elastic': {'host': 'http://127.0.0.1:9200', 'user': None, 'token': None, 'shards': 10, 'replicas': 0, 'refresh_interval': '30s'}, 'influx': {'host': 'http://127.0.0.1:8086', 'db': 'example', 'create': True, 'username': 'None', 'password': 'None'}, 'parquet': {'del_file': True, 'append_counter': 4, 'file_format': ['exchange', 'pair', 'data_type', 'timestamp'], 'compression': {'codec': 'BROTLI', 'level': 6}, 'path': None, 'prefix_date': False, 'S3': {'endpoint': None, 'key_id': None, 'secret': None, 'bucket': None, 'prefix': None}, 'GCS': {'service_account': None, 'bucket': None, 'prefix': None}, 'GD': {'service_account': None, 'prefix': None, 'folder_name_sep': '.'}}, 'arctic': 'mongodb://127.0.0.1', 'storage_interval': 60}}
2021-02-24 20:39:01,118 : INFO : Aggregator running on PID 17456
2021-02-24 20:39:01,136 : INFO : message: {"op": "start", "exchange": "BITMEX", "collector": {"channel_timeouts": {"l2_book": 30, "trades": 120, "ticker": 120, "funding": -1}, "retries": -1, "l2_book": {"symbols": ["XBTUSD"], "max_depth": 10, "book_delta": true, "book_interval": 100000}, "trades": ["XBTUSD"], "ticker": ["XBTUSD"], "funding": ["XBTUSD"]}, "config": {"cache": "redis", "kafka": {"ip": "127.0.0.1", "port": 9092, "start_flush": true}, "redis": {"ip": "127.0.0.1", "port": 6379, "socket": null, "del_after_read": true, "retention_time": null, "start_flush": true}, "storage": ["arctic", "influx"], "storage_retries": 5, "storage_retry_wait": 30, "pass_through": {"type": "zmq", "host": "127.0.0.1", "port": 5678}, "elastic": {"host": "http://127.0.0.1:9200", "user": null, "token": null, "shards": 10, "replicas": 0, "refresh_interval": "30s"}, "influx": {"host": "http://127.0.0.1:8086", "db": "example", "create": true, "username": "None", "password": "None"}, "parquet": {"del_file": true, "append_counter": 4, "file_format": ["exchange", "pair", "data_type", "timestamp"], "compression": {"codec": "BROTLI", "level": 6}, "path": null, "prefix_date": false, "S3": {"endpoint": null, "key_id": null, "secret": null, "bucket": null, "prefix": null}, "GCS": {"service_account": null, "bucket": null, "prefix": null}, "GD": {"service_account": null, "prefix": null, "folder_name_sep": "."}}, "arctic": "mongodb://127.0.0.1", "storage_interval": 60}}
2021-02-24 20:39:01,137 : INFO : Starting {'op': 'start', 'exchange': 'BITMEX', 'collector': {'channel_timeouts': {'l2_book': 30, 'trades': 120, 'ticker': 120, 'funding': -1}, 'retries': -1, 'l2_book': {'symbols': ['XBTUSD'], 'max_depth': 10, 'book_delta': True, 'book_interval': 100000}, 'trades': ['XBTUSD'], 'ticker': ['XBTUSD'], 'funding': ['XBTUSD']}, 'config': {'cache': 'redis', 'kafka': {'ip': '127.0.0.1', 'port': 9092, 'start_flush': True}, 'redis': {'ip': '127.0.0.1', 'port': 6379, 'socket': None, 'del_after_read': True, 'retention_time': None, 'start_flush': True}, 'storage': ['arctic', 'influx'], 'storage_retries': 5, 'storage_retry_wait': 30, 'pass_through': {'type': 'zmq', 'host': '127.0.0.1', 'port': 5678}, 'elastic': {'host': 'http://127.0.0.1:9200', 'user': None, 'token': None, 'shards': 10, 'replicas': 0, 'refresh_interval': '30s'}, 'influx': {'host': 'http://127.0.0.1:8086', 'db': 'example', 'create': True, 'username': 'None', 'password': 'None'}, 'parquet': {'del_file': True, 'append_counter': 4, 'file_format': ['exchange', 'pair', 'data_type', 'timestamp'], 'compression': {'codec': 'BROTLI', 'level': 6}, 'path': None, 'prefix_date': False, 'S3': {'endpoint': None, 'key_id': None, 'secret': None, 'bucket': None, 'prefix': None}, 'GCS': {'service_account': None, 'bucket': None, 'prefix': None}, 'GD': {'service_account': None, 'prefix': None, 'folder_name_sep': '.'}}, 'arctic': 'mongodb://127.0.0.1', 'storage_interval': 60}}
2021-02-24 20:39:01,194 : INFO : Flushing cache
2021-02-24 20:39:02,672 : INFO : Collector for BITMEX running on PID 12668
2021-02-24 20:39:02,684 : INFO : Collector for COINBASE running on PID 17228
Process Collector-1:2:
Traceback (most recent call last):
File "c:\users\dawie\appdata\local\programs\python\python38\lib\multiprocessing\process.py", line 313, in _bootstrap
self.run()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\collector.py", line 112, in run
cb[L2_BOOK].append(BookZMQ(host=host, port=port))
Process Collector-1:1:
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptofeed\backends\zmq.py", line 19, in init
self.con = ctx.socket(zmq.PUB)
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\sugar\context.py", line 235, in socket
s = self._socket_class(self, socket_type, **kwargs)
Traceback (most recent call last):
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq_future.py", line 150, in init
self._init_io_state()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\multiprocessing\process.py", line 313, in _bootstrap
self.run()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\asyncio.py", line 51, in _init_io_state
self.io_loop.add_reader(self._fd, lambda: self._handle_events(0, 0))
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\collector.py", line 114, in run
cb[L3_BOOK].append(BookZMQ(host=host, port=port))
File "c:\users\dawie\appdata\local\programs\python\python38\lib\asyncio\events.py", line 501, in add_reader
raise NotImplementedError
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptofeed\backends\zmq.py", line 19, in init
self.con = ctx.socket(zmq.PUB)
NotImplementedError
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\sugar\context.py", line 235, in socket
s = self._socket_class(self, socket_type, **kwargs)
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq_future.py", line 150, in init
self._init_io_state()
Exception ignored in: <function Socket.del at 0x000002005CFF6A60>
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\asyncio.py", line 51, in _init_io_state
self.io_loop.add_reader(self._fd, lambda: self._handle_events(0, 0))
Traceback (most recent call last):
File "c:\users\dawie\appdata\local\programs\python\python38\lib\asyncio\events.py", line 501, in add_reader
raise NotImplementedError
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\sugar\socket.py", line 66, in del
NotImplementedError
self.close()
Exception ignored in: <function Socket.del at 0x0000026FCE476A60>
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq_future.py", line 168, in close
Traceback (most recent call last):
self._clear_io_state()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\sugar\socket.py", line 66, in del
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\asyncio.py", line 58, in _clear_io_state
self.close()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq_future.py", line 168, in close
self.io_loop.remove_reader(self._fd)
self._clear_io_state()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\asyncio\events.py", line 504, in remove_reader
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\zmq\asyncio.py", line 58, in _clear_io_state
raise NotImplementedError
self.io_loop.remove_reader(self._fd)
NotImplementedError:
File "c:\users\dawie\appdata\local\programs\python\python38\lib\asyncio\events.py", line 504, in remove_reader
raise NotImplementedError
NotImplementedError:
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<Aggregator.loop() done, defined at c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\aggregator\aggregator.py:43> exception=ConnectionError('Error 10061 connecting to 127.0.0.1:6379. No connection could be made because the target machine actively refused it.')>
Traceback (most recent call last):
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\connection.py", line 559, in connect
sock = self._connect()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\connection.py", line 615, in _connect
raise err
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\connection.py", line 603, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\aggregator\aggregator.py", line 45, in loop
cache = Redis(ip=self.config.redis['ip'],
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\aggregator\redis.py", line 32, in init
self.conn.flushall()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\client.py", line 1275, in flushall
return self.execute_command('FLUSHALL', *args)
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\client.py", line 898, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\connection.py", line 1192, in get_connection
connection.connect()
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\redis\connection.py", line 563, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 10061 connecting to 127.0.0.1:6379. No connection could be made because the target machine actively refused it.
do you have redis set up and running on that address? Thats at least one of the errors.
The other error looks related to asyncio + ZMQ and Windows. Are you using ZMQ? If not, you should probably remove that section from the config. Any sections you arent using / havent configured you need to remove.
Hi Bryant
Thx for the tips.
I have simplified the config to:
cache: redis
redis:
ip: '127.0.0.1'
port: 6379
socket: null
del_after_read: true
retention_time: null
start_flush: true
exchanges:
BITMEX:
channel_timeouts:
l2_book: 30
trades: 120
ticker: 120
funding: -1
retries: -1
l2_book:
symbols: [XBTUSD]
max_depth: 10
book_delta: true
book_interval: 100000
trades: [XBTUSD]
ticker: [XBTUSD]
funding: [XBTUSD]
COINBASE:
retries: -1
l3_book:
symbols: [BTC-USD]
book_delta: true
book_interval: 100000
trades: [BTC-USD, ETH-USD, ETH-BTC]
ticker: [BTC-USD]
I still get storage related error though i.e.:
cryptostore
2021-03-01 10:42:29,973 : INFO : Starting cryptostore
2021-03-01 10:42:29,973 : INFO : Cryptostore running on PID 12988
2021-03-01 10:42:29,981 : INFO : Spawner started
2021-03-01 10:42:29,989 : INFO : Aggregator started
2021-03-01 10:42:29,991 : INFO : Cryptostore started
2021-03-01 10:42:29,999 : INFO : start: ['BITMEX', 'COINBASE'] stop: []
2021-03-01 10:42:31,257 : INFO : Aggregator running on PID 23408
2021-03-01 10:42:31,261 : INFO : Spawner running on PID 4680
2021-03-01 10:42:31,262 : INFO : message: {"op": "start", "exchange": "BITMEX", "collector": {"channel_timeouts": {"l2_book": 30, "trades": 120, "ticker": 120, "funding": -1}, "retries": -1, "l2_book": {"symbols": ["XBTUSD"], "max_depth": 10, "book_delta": true, "book_interval": 100000}, "trades": ["XBTUSD"], "ticker": ["XBTUSD"], "funding": ["XBTUSD"]}, "config": {"cache": "redis", "redis": {"ip": "127.0.0.1", "port": 6379, "socket": null, "del_after_read": true, "retention_time": null, "start_flush": true}}}
2021-03-01 10:42:31,262 : INFO : Starting {'op': 'start', 'exchange': 'BITMEX', 'collector': {'channel_timeouts': {'l2_book': 30, 'trades': 120, 'ticker': 120, 'funding': -1}, 'retries': -1, 'l2_book': {'symbols': ['XBTUSD'], 'max_depth': 10, 'book_delta': True, 'book_interval': 100000}, 'trades': ['XBTUSD'], 'ticker': ['XBTUSD'], 'funding': ['XBTUSD']}, 'config': {'cache': 'redis', 'redis': {'ip': '127.0.0.1', 'port': 6379, 'socket': None, 'del_after_read': True, 'retention_time': None, 'start_flush': True}}}
2021-03-01 10:42:31,271 : INFO : message: {"op": "start", "exchange": "COINBASE", "collector": {"retries": -1, "l3_book": {"symbols": ["BTC-USD"], "book_delta": true, "book_interval": 100000}, "trades": ["BTC-USD", "ETH-USD", "ETH-BTC"], "ticker": ["BTC-USD"]}, "config": {"cache": "redis", "redis": {"ip": "127.0.0.1", "port": 6379, "socket": null, "del_after_read": true, "retention_time": null, "start_flush": true}}}
2021-03-01 10:42:31,272 : INFO : Starting {'op': 'start', 'exchange': 'COINBASE', 'collector': {'retries': -1, 'l3_book': {'symbols': ['BTC-USD'], 'book_delta': True, 'book_interval': 100000}, 'trades': ['BTC-USD', 'ETH-USD', 'ETH-BTC'], 'ticker': ['BTC-USD']}, 'config': {'cache': 'redis', 'redis': {'ip': '127.0.0.1', 'port': 6379, 'socket': None, 'del_after_read': True, 'retention_time': None, 'start_flush': True}}}
2021-03-01 10:42:31,284 : INFO : Flushing cache
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<Aggregator.loop() done, defined at c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\aggregator\aggregator.py:43> exception=KeyError('storage_interval')>
Traceback (most recent call last):
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\aggregator\aggregator.py", line 56, in loop
interval = self.config.storage_interval
File "c:\users\dawie\appdata\local\programs\python\python38\lib\site-packages\cryptostore\config.py", line 40, in getattr
return self.config[attr]
KeyError: 'storage_interval'
that key, storage_interval, is required