ghstack crashing nondeterministically
hsghori opened this issue · comments
I have a stack of >10 PRs and ghstack seems to be crashing with an ambiguous error:
ERROR: Fatal exception
Traceback (most recent call last):
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/hsghori/.pyenv/versions/3.9.11/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/hsghori/.pyenv/versions/3.9.11/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/hsghori/.pyenv/versions/3.9.11/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/xxxx (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/logs.py", line 105, in manager
yield
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/cli.py", line 41, in cli_context
yield shell, config, github
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/cli.py", line 207, in submit
ghstack.submit.main(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/submit.py", line 220, in main
submitter.push_updates()
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/submit.py", line 1058, in push_updates
self.github.patch(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/github.py", line 61, in patch
return self.rest("patch", path, **kwargs)
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/ghstack/github_real.py", line 125, in rest
resp: requests.Response = getattr(requests, method)(
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/requests/api.py", line 145, in patch
return request("patch", url, data=data, **kwargs)
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/hsghori/.local/pipx/venvs/ghstack/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/xxxx (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))
I think this is the thing where if your stack is too long we do too many queries to GH API and we get throttled. The only workaround is to have less diffs in your stack atm
duping as #121