How to import external data to populate entry/exit trend conditions
darioneto opened this issue · comments
Could someone please advise what is the most efficient way to import data from an external database and use that info to populate trend conditions?
What is the best approach to this solution?
Can the data be ingested directly from the strategy file structure from the external csv, sql database format, and imported as dataframe to use for the entry or exit conditions and recalculations?
Will the loop refresh this input as the new data comes in?
The external data will contain simple one row with two columns based on which I want to plug it into the trend conditions ie:
dataframe['external_data'] > 5
enter_long = 1
Well a strategy is just a python file, so almost everything you can code in python, will also work in a strategy.
You'll obviously keep frequency in mind (how often should this be called ...).
For that, best consult basic documentation - which will tell you which method will be called at which times.
To facilitate backtesting, your datasource also needs to contain timestamps - and you'll need to somehow merge this to the original dataframe.
Alternatively, you can also skip backtesting and just trust your external signal - in which case just a "yes/no" will be needed.
Thanks for your reply,
As for the frequency according to the documentation, the loop is executed every few seconds which should be good enough. Not sure how good is the bot error handling/timeouts coming from external calls in the case of those, but this would need to be tested moving forward.
I assume that the reference to the external data is the best to locate inside of "populate entry/exit trend" function?
The "populate_indicators" section in this case would not make much sense.
I'm not concerned with backtesting. This not be neecessary.
well you should do error-handling in your strategy.
freqtrade only places "some" restrictions on the strategy - it shouldn't fail (obviously) - and it must return the dataframe with the new columns.
which of the populate_ methods you place it is really irrelevant, as they're called one after the other - except for hyperopt, where populate_indicators is pulled out of the hyperopt loop and only runs once.
Thanks for your reply,
I managed to install the dependencies and it appears I get a mixture of errors from the logs depending on the data structure. Do we need to blank the index or column name to pass it correctly?
Could you please share an example of how to import example data and populate fields?
The psql call returns a (df) dataframe with one index and one column with a value
then inside of a trend entry function, I'm trying to parse this output
dataframe.loc[
(
(df["pcnt_change_down"] > -1) &
(dataframe['volume'] > 0)
),
'enter_long'] = 1
The dtype: int64.
The value from the df is 0 so I'm expecting to trigger the entry immediately. Not sure how to troubleshoot it, there is not much info on this concept.
The recent error I get with the above setup is "Can only compare identically-labeled Series objects"
you can't do such a comparison - this compares the whole dataframe - so it assumes that they have the same length.
if your df has just one value - assign it to a column in the original dataframe so you can compare it.
the dataframe gotchas apply to this problem even more so - as you're introducing another factor in having a completely separate dataframe.
def populate_entry_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
....
conn = psycopg2.connect(
database="xx", user='xx',
password='xx', host='xx', port='xx'
)
conn.autocommit = True
cursor = conn.cursor()
df_pcnt_change_down = pd.read_sql_query('''
SELECT XXX''', conn)
conn.close()
dataframe.loc[
(
(df_pcnt_change_down["pcnt_change_down"][0] > 3) &
(dataframe['volume'] > 0)
),
'enter_long'] = 1
The above works somehow, however, the execution time is unpredictable. The long gets triggered after minutes sometimes hours or never gets triggered. As for the exit below, it never gets triggered.
....
conn2 = psycopg2.connect(
database="xx", user='xx',
password='xx', host='xx', port='xx'
)
conn2.autocommit = True
cursor = conn2.cursor()
df_pcnt_change_up = pd.read_sql_query('''
SELECT
XXX ''', conn2)
# print(df)
conn2.close()
dataframe.loc[
(
(df_pcnt_change_up["pcnt_change_up"][0] > 2) &
(dataframe['volume'] > 0)
),
'exit_long'] = 1
Any thoughts on what might be wrong with this? Most of the other sections are unchanged and default. I marked out populate_indicators as don't need them.
In logs I can see that below note after the first leg long is opened, not sure if anything related
"No currency pair in active pair whitelist, but checking to sell open trades"
Your enemy will most likely be process_only_new_candles - which means these functions will only be called once per candle.
For such a case, you clearly have to set this to False - so the iteration runs every few seconds.
If you then have latencies on your integration is basically up to you to determine/decide - we can't really provide support on this.
Ok thanks, I can see it's updating every few seconds now, however despite the fact the conditions are met the orders are not being triggered. Does this look correct?
I don't expect any latencies, the db is local to the location and very capable of handling a large number of connections fast.
Ok so to the first part I found the answer, It was related to the auto-lock. I found a way around it by changing to the lower timeframe. However, I have no idea what could be an issue with exiting the long. The data frame is being populated correctly within seconds but trades are not being closed. Any idea why?
I think could be to the fact that both conditions are True. The bot will not exit the trade.
Again - the documentation will contain the answer to your questions.
As this is a python I'd assume Bool AND OR should work for evaluating conditions?
For some reason, I'm not able to generate OR valuations. I couldn't find this in the documentation.
It's a pandas mask expression - | will work as OR.
oh ok, thanks, it works!