Fix command to fetch NJTransit data for deploy
rachelekm opened this issue · comments
During this month's deploy we were not able successfully fetch the rail_data.zip
or bus_data.zip
files from the NJ Transit feeds using the CLI. Instead we needed to manually download both zip files for use. We can successfully login to the NJ Transit account via CLI and "get" the bus_data.zip
file, however our logs show the command fails during verification and eventually we receive a timeout error. While it seemingly makes it to verification, the "get" is misleading because the zip file can also not be opened locally (although @ktohalloran did not have this issue in her local env and is able to open her zip files locally, but is still receiving a timeout error) so we're suspecting an issue fetching the file vs verifying. Below are the logs after running gtfs % docker-compose run --rm gtfs-feed-fetcher ./fetch_feeds.py --feeds=NJTransit --get-nj
and passing the login credentials:
INFO:root:Going to fetch feeds from sources: ['NJTransit']
INFO:FeedSource:Getting file /usr/local/src/gtfs/nj_bus.zip...
INFO:FeedSource:Download completed successfully.
INFO:FeedSource:Validating feed in nj_bus.zip...
INFO:FeedSource:Feed nj_bus.zip looks ok: 3046 warnings found.
INFO:FeedSource:GTFS verification succeeded.
Traceback (most recent call last):
File "./fetch_feeds.py", line 124, in <module>
main()
File "./fetch_feeds.py", line 119, in main
fetch_all(args.get_nj, nj_username, nj_password, sources=sources)
File "./fetch_feeds.py", line 48, in fetch_all
inst.fetch()
File "/usr/local/src/feed_sources/NJTransit.py", line 34, in fetch
login = session.post(LOGIN_URL, data=self.nj_payload)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 572, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 524, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 637, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.njtransit.com', port=443): Max retries exceeded with url: /users/login (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8141b60e10>: Failed to establish a new connection: [Errno 110] Connection timed out',))
ERROR: 1
Fetching the file should be much better now, but at the moment all the files are returning 502 ERRORs.