bmoscon / cryptostore

A scalable storage service for cryptocurrency data

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add ability to use password-protected Redis server

arian-alt opened this issue · comments

Is your feature request related to a problem? Please describe.
I'm trying to use a password-protected Redis server (currently using a free server from Redis labs to test things out) but it seems there is no easy way to do such a thing

Describe the solution you'd like
having the ability to just pass the API key in the config file would be awesome because most of the cloud provided Redis servers are password-protected

Describe alternatives you've considered
1-Tried playing around with URLs to somehow pass the API key
For example, tried using "[:PASSWORD@]HOST" format as host IP but wasn't successful
more info about Redis URL formats here

2-I noticed there are 4 files handling Redis:
aggregator.py, collector.py & redis.py from Cryptostore
another redis.py from Cryptofeed backends
I made these changes to them:

aggregator.py:

        if self.config.cache == 'redis':
            cache = Redis(ip=self.config.redis['ip'],
                          port=self.config.redis['port'],
                          socket=self.config.redis.socket,
                          del_after_read=self.config.redis['del_after_read'],
                          flush=self.config.redis['start_flush'],
                          retention=self.config.redis.retention_time if 'retention_time' in self.config.redis else None,
                          password=self.config.redis['pass'])                 # line 51 added password input

redis.py

class Redis(Cache):
    def __init__(self, ip=None, port=None, socket=None, del_after_read=True, flush=False, retention=None, password=None):
        self.del_after_read = del_after_read
        self.retention = retention
        self.last_id = {}
        self.ids = defaultdict(list)
        if ip and port and socket:
            raise ValueError("Cannot specify ip/port and socket for Redis")
        self.conn = StorageEngines.redis.Redis(ip, port, password=password, unix_socket_path=socket, decode_responses=True)
        # line 30 connecting to redis with password

collector.py

if cache == 'redis':
     if 'ip' in self.config['redis'] and self.config['redis']['ip']:
          kwargs = {'host': self.config['redis']['ip'], 'port': self.config['redis']['port'], 'numeric_type': float, 'password': self.config['redis']['pass']}
# line 56 passing password to cryptofeed redis backend

redis.py from cryptofeed

class RedisCallback:
    def __init__(self, host='127.0.0.1', port='6379', socket=None, key=None, numeric_type=float, password=None, **kwargs):
        """
        setting key lets you override the prefix on the
        key used in redis. The defaults are related to the data
        being stored, i.e. trade, funding, etc
        """
        self.redis = None
        self.key = key if key else self.default_key
        self.numeric_type = numeric_type
        self.conn_str = socket if socket else f'redis://{host}:{port}?db=0&password={password}' #line 24 using url formating to connect

Didn't get any Redis related errors but started to get random errors ranging from parquet to unexpected keywords
couldn't get further since I have no clue how different parts work together

Additional context
I tested my installation with a local Redis server (no password required) and everything works fine so I don't think its because of my installation or dependencies

your help will be appreciated

I don't use password protected redis, so its not something I'm familiar with