Slow stats.nba endpoints
djchie opened this issue · comments
It seems like the endpoints to stats.nba are slow. I tried making a request directly to the stats.nba API and the same thing happened. It all takes about 8~10 seconds to run.
What's interesting is that if I make a request from Postman rather than localhost, the speed is just fine.
I was wondering if I was running into a DNS lookup issue, so I curled with the following:
curl --trace-time -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\n----------\ntime_total: %{time_total}\n" -v http://stats.nba.com/stats/scoreboardV2?DayOffset=0&LeagueID=00&gameDate=12/25/2016
And got this as the result:
time_namelookup: 0.013
time_connect: 0.038
time_appconnect: 0.000
time_pretransfer: 0.038
time_redirect: 0.000
time_starttransfer: 9.077
----------
time_total: 9.078
Both the time_starttransfer and the ttfb from the Chrome console seems to indicate that it's something on their end?
What I'm wondering is if you're experiencing this as well?
Yeah, running the integration tests it seems like the stats endpoints are all super slow for me as well -- all taking 8+ seconds to complete. I just saw your note on nba_py – seems like the same issue.
When trying your curl I noticed it returned a 403 (some endpoints require some extra headers), I was wondering if they were slowing down requests like that. I tried another which is 200-ing for me as of now, with same long delay:
curl --trace-time -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\n----------\ntime_total: %{time_total}\n" -v http://stats.nba.com/stats/playerprofilev2\?PlayerID\=201939\&LeagueID\=00\&PerMode\=PerGame
FWIW I'm also getting instantaneous (<50ms) responses from Postman. No idea how their API is differentiating these requests though.
@djchie can you try on latest (3.2.3)? After a little detective work, it seems like adding Connection: keep-alive header solves this issue!
Hmm... on Travis I'm still seeing these requests take a long time. Perhaps there was some kind of caching that sped this up. Not quite sure what's going on here...
@djchie Here's your time tracing with the headers postman is sending (using Chrome's "Copy as cURL" tool in DevTools)
curl --trace-time -w "\ntime_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_appconnect: %{time_appconnect}\ntime_pretransfer: %{time_pretransfer}\ntime_redirect: %{time_redirect}\ntime_starttransfer: %{time_starttransfer}\n----------\ntime_total: %{time_total}\n" -v 'http://stats.nba.com/stats/playerprofilev2?PlayerID=201939&LeagueID=00&PerMode=PerGame' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Postman/4.4.2 Chrome/51.0.2704.103 Electron/1.2.5 Safari/537.36' -H 'Accept: */*' -H 'Cache-Control: no-cache' -H 'Connection: keep-alive' --compressed
This command runs very fast, but I've mimicked it as close as possible with a Node request and it seems much slower. Compounding this, it seems like there is a caching mechanism somewhere that confuses the whole thing -- when you running npm t twice in a row, the stats routes are much faster the second time. I wonder if you pull master do you see the same behavior?
I'm going to close this -- the running time of the test suite seems to be similar to what it was a couple months ago, and I no longer see response time clustering around 8-10s.
Sorry for the long wait in reply. Just wanted to confirm that I did try it out with the headers you provided and it's faster now. Thanks!