Bulk import OHLCV data to candles
johngriffin opened this issue · comments
I had a need to import data for many different pairs and wrote the attached code to implement this feature which I would like to contribute back to the project:
# Import candles from a directory of csv files containing OHLCV data into Jesse
# directory contains csv files with filename {symbol}-1m.csv, e.g. BTC-USD-1m.csv
# csv column order is 'timestamp', 'open', 'high', 'low', 'close', 'volume'
# jesse expects each row to be 1m candlestick
import os
import pandas as pd
import numpy
from jesse import research
DIRECTORY = 'data/FTX/'
EXCHANGE = 'FTX'
def flow_from_df(dataframe: pd.DataFrame, chunk_size: int = 10):
for start_row in range(0, dataframe.shape[0], chunk_size):
end_row = min(start_row + chunk_size, dataframe.shape[0])
yield dataframe.iloc[start_row:end_row, :]
for file in os.listdir(DIRECTORY):
filename = os.fsdecode(file)
if filename.endswith(".csv"):
symbol = filename.split('-1m.csv').pop(0)
# We need to convert *-PERP to *-USD otherwise backtests will fail later
symbol = symbol.replace('-PERP', '-USD')
print ("importing " + symbol)
# Read the file into a pandas dataframe
df = pd.read_csv(os.path.join(DIRECTORY, filename))
df.columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume']
# Reorder columns as jesse expects candles to be in format:
# 'timestamp', 'open', 'close', 'high', 'low', volume'
df = df[['timestamp', 'open', 'close', 'high', 'low', 'volume']]
# Import them candles in batches
for chunk in flow_from_df(df, 1000):
candles = chunk.to_numpy()
research.store_candles(candles, EXCHANGE, symbol)
else:
print ("ignoring " + filename)
continue
Hello John. Thank you for your contribution.
This could be a great addition to https://github.com/jesse-ai/awesome-jesse. Could you make it a repository with the CSV files included and submit a PR to the awesome-jesse
repository?