Handling missing historical info
IvanBohannon opened this issue · comments
There are some stocks that were lightly traded in previous years, and on some days, they had zero volume, zero trades.
ex: DRIP -- June 2, 2015.
or YINN for many days in 2015.
Yahoo finance historical prices shows these days as entries with all zeros.
It seems that this library just omits these days, and its difficult to determine in code if missing days are a result of yahoo finance having bad data, or if its valid no trading for the day.
What do you think about replicating the zero entries for those dates in this library? I could do this in my own code, but others might have this issue as well, so might make more sense to fix it here.
@IvanBohannon thank you for the feedback. Actually, there's a similar question asked in #3, and the solution at that time is to ignore all the invalid entries.
But I think what you are telling is probably a better approach on handling the nulls, it seems we could make it the default behaviour but an optional parameter for ignoring all invalids, and marked it as a breaking change.
In addition, it might be more appropriate to change the value type from decimals to nullable decimals, and mark it null instead of zero, as others may misuse the zeros for calculation which is a fatal problem when doing analysis. What do you think?
Nullable instead of zeros makes sense, and would let us still have entries for those days - which from a test perspective is useful. Yahoo chose to return zeros for those days with no recorded trades, which is confusing.
We might be able to avoid a breaking change if we add a field to indicate if data was invalid, instead of returning zeros or nullables.
I'm fine either way, but leaning towards less breaking 😄
Update: just read issue #3 -- sounds like yahoo displays zeros for those days (and ignores them for graphing), but returned nulls in the CSV download.
In fact, the invalid data in csv are empty strings 😃 , so there is error when converting it to decimal, hence the skipping logic.
I agree that it should be less breaking but I think adding a field is too much for each candle. Perhaps we could add an optional flag "leaveZeroIfInvalidRow" in GetHistoricalMethods & default it to false.
In this way, there is no breaking for current user who favors the skip logic, and it also favor users like you who wants the leave zero logic.
If it's ok for you, I will start the implementation 😃