[ENH] Throttle API requests to support polygon's free plan for market miners?

[TheSnoozer]

Sorry for the spam of tickets, but I noticed this while running some tests.

Is your feature request related to a problem? Please describe.
Essentially it currently seems that I have setup everything correctly (YAY). I started the example setup with running the miner.toml under the examples section and the swing-momentum/portfolio.py like such:

    [miners.portfolio]
        filename = "swing-momentum/portfolio.py"
        portfolio_size = 2000
        debug = true
        index = 'SP500'
        rank_days = 90
        atr_days = 20
        risk_factor = 0.002
        indicators = ['SMA100']

initially I was getting errors like the following:

[load_data()][9942]2021-01-21 22:43:05.755232:loading 200 days for symbol MMM (1/505)
[main()][9942]2021-01-21 22:43:07.067853:[ERROR] aborted w/ exception object of type 'NoneType' has no len()
Traceback (most recent call last):
  File "/.venv/lib/python3.8/site-packages/liualgotrader-0.0.86-py3.8.egg/EGG-INFO/scripts/market_miner", line 73, in main
    await asyncio.gather(*task_list)
  File "swing-momentum/portfolio.py", line 178, in run
    await self.load_data(symbols)
  File "swing-momentum/portfolio.py", line 67, in load_data
    tlog(f"loaded {len(self.data_bars[symbol])} data-points")
TypeError: object of type 'NoneType' has no len()

which I now believed to have traced down on how polygon behaves under the free plan.
In the dashboard I can see that 5 requests per minute are getting answered correctly, but the rest is refused due to rate limitation of the free plan (see https://polygon.io/pricing).

From my end it seems if one is rate limited the call to daily_bars will return Ǹone (and not throw an Exception) causing all sorts of issues.

Describe the solution you'd like
The 'perfect' solution I could imagine is either some rate limitation build in into LiuAlgoTrader, or using another free endpoint without rate limitation that allows to fetch daily data (e.g. yahoo finance).

Certainly if I know my strategies return the expenses needed to pay for the premium for polygon this problem would not exist...but I'm not there yet ;-)

[@amor71]

I've uploaded a potential fix to the master branch, if that's working for you too, I will release the fix w/ thanks.

[TheSnoozer]

Hi, thanks for the fast response and the fix! I guess that would do the trick in most cases, it might still pose an issue if the market miner finds a lot of matching stocks. E.g. if the miners finds more than 25 it might just kill the process with an exception in the end. Why 25? The time sleep of 30 and the retry of 10 will essentially span out the requests to 5 minutes with 5 requests per minute this leads to 25 stocks that would go through without any exception. Again I guess that would do without getting it too complicated....

[@amor71]

The portfolio example runs on SP500, so I think there's enough buffer. The reason to stop the execution is that at times Polygon has other issues, which are not API throttling oriented. However, what I suggest is that you open a live account w/ ALPACA, and transfer a one-time small amount (I'm not sure what is the minimal transfer these days, I remember I opened it up with as low as $100). This gives you immediate access to Polygon live data w/o the monthly charges.. I think it's generally a good idea to practice both paper-trading and live-trading (obviously small amounts under FINRA restrictions). Makes sense?

…

On Fri, Jan 22, 2021 at 7:36 PM TheSnoozer ***@***.***> wrote: Hi, thanks for the fast response and the fix <d2b3ecb>! I guess that would do the trick in most cases, it might still pose an issue if the market miner finds a lot of matching stocks. E.g. if the miners finds more than 25 it might just kill the process with an exception in the end. Why 25? The time sleep of 30 and the retry of 10 will essentially span out the requests to 5 minutes with 5 requests per minute this leads to 25 stocks that would go through without any exception. Again I guess that would do without getting it too complicated.... — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#154 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALTIHBFUERB3FXPCLS6ZVTLS3GZTDANCNFSM4WNVH73A> .

[@amor71]

@TheSnoozer any additional feedback?

[TheSnoozer]

Thanks for the feedback. Yeah for non us people alpaca is not automatically available, but I can live with the provided solution so I'm closing this. Thanks again :-)