azavea / cac-tripplanner

Clean Air Council Circuit Trip Planner and Travelshed

Home Page:https://gophillygo.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Fix command to fetch NJTransit data for deploy

rachelekm opened this issue · comments

During this month's deploy we were not able successfully fetch the rail_data.zip or bus_data.zip files from the NJ Transit feeds using the CLI. Instead we needed to manually download both zip files for use. We can successfully login to the NJ Transit account via CLI and "get" the bus_data.zip file, however our logs show the command fails during verification and eventually we receive a timeout error. While it seemingly makes it to verification, the "get" is misleading because the zip file can also not be opened locally (although @ktohalloran did not have this issue in her local env and is able to open her zip files locally, but is still receiving a timeout error) so we're suspecting an issue fetching the file vs verifying. Below are the logs after running gtfs % docker-compose run --rm gtfs-feed-fetcher ./fetch_feeds.py --feeds=NJTransit --get-nj and passing the login credentials:

INFO:root:Going to fetch feeds from sources: ['NJTransit']
INFO:FeedSource:Getting file /usr/local/src/gtfs/nj_bus.zip...
INFO:FeedSource:Download completed successfully.
INFO:FeedSource:Validating feed in nj_bus.zip...
INFO:FeedSource:Feed nj_bus.zip looks ok:  3046 warnings found.
INFO:FeedSource:GTFS verification succeeded.
Traceback (most recent call last):
  File "./fetch_feeds.py", line 124, in <module>
    main()
  File "./fetch_feeds.py", line 119, in main
    fetch_all(args.get_nj, nj_username, nj_password, sources=sources)
  File "./fetch_feeds.py", line 48, in fetch_all
    inst.fetch()
  File "/usr/local/src/feed_sources/NJTransit.py", line 34, in fetch
    login = session.post(LOGIN_URL, data=self.nj_payload)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 572, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 524, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 637, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.njtransit.com', port=443): Max retries exceeded with url: /users/login (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8141b60e10>: Failed to establish a new connection: [Errno 110] Connection timed out',))
ERROR: 1

Fetching the file should be much better now, but at the moment all the files are returning 502 ERRORs.