Giters

pytogo / portforward

Kubernetes Port-Forward Go-Edition For Python

Home Page:https://portforward.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

portforward crashes when run immediately after pod creation. Is there a sample fixture that shows cluster health wait?

drawal1 opened this issue · comments

commented
  • portforward version: latest
  • Python version: 3.10.8
  • Operating System: Ubuntu 20.04

Description

Describe what you were trying to get done:

Executing a makefile script that deployed a kube cluster and then ran pytest.

NOTE: It works fine if I run the pytest manually after waiting a bit. Also, I do have a fixture in my pytest that waits for the pods to be in running or succeeded state before calling the test with the portforward

Tell us what happened, what went wrong, and what you expected to happen: Pytest crashed. Expected it to run the test

What I Did

I ran "make test" from the commandline. The makefile has below section

test:
kubectl apply -f ./ <=== Creates the pods
pytest ./ <=== Runs a test using portforward

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

        The URL to perform the request on.

    :param body:
        Data to send in the request body, either :class:`str`, :class:`bytes`,
        an iterable of :class:`str`/:class:`bytes`, or a file-like object.

    :param headers:
        Dictionary of custom headers to send, such as User-Agent,
        If-None-Match, etc. If None, pool headers are used. If provided,
        these headers completely replace any pool-specific headers.

    :param retries:
        Configure the number of retries to allow before raising a
        :class:`~urllib3.exceptions.MaxRetryError` exception.

        Pass ``None`` to retry until you receive a response. Pass a
        :class:`~urllib3.util.retry.Retry` object for fine-grained control
        over different types of retries.
        Pass an integer number to retry connection errors that many times,
        but no other types of errors. Pass zero to never retry.

        If ``False``, then retries are disabled and any exception is raised
        immediately. Also, instead of raising a MaxRetryError on redirects,
        the redirect response will be returned.

    :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.

    :param redirect:
        If True, automatically handle redirects (status codes 301, 302,
        303, 307, 308). Each redirect counts as a retry. Disabling retries
        will disable redirect, too.

    :param assert_same_host:
        If ``True``, will make sure that the host of the pool requests is
        consistent else will raise HostChangedError. When ``False``, you can
        use the pool on an HTTP proxy and request foreign hosts.

    :param timeout:
        If specified, overrides the default timeout for this one
        request. It may be a float (in seconds) or an instance of
        :class:`urllib3.util.Timeout`.

    :param pool_timeout:
        If set and the pool is set to block=True, then this method will
        block for ``pool_timeout`` seconds and raise EmptyPoolError if no
        connection is available within the time period.

    :param release_conn:
        If False, then the urlopen call will not release the connection
        back into the pool once a response is received (but will release if
        you read the entire contents of the response such as when
        `preload_content=True`). This is useful if you're not preloading
        the response's content immediately. You will need to call
        ``r.release_conn()`` on the response ``r`` to return the connection
        back into the pool. If None, it takes the value of
        ``response_kw.get('preload_content', True)``.

    :param chunked:
        If True, urllib3 will send the body using chunked transfer
        encoding. Otherwise, urllib3 will send the body using the standard
        content-length form. Defaults to False.

    :param int body_pos:
        Position to seek to in file-like body in the event of a retry or
        redirect. Typically this won't need to be set because urllib3 will
        auto-populate the value when needed.

    :param \\**response_kw:
        Additional parameters are passed to
        :meth:`urllib3.response.HTTPResponse.from_httplib`
    """

    parsed_url = parse_url(url)
    destination_scheme = parsed_url.scheme

    if headers is None:
        headers = self.headers

    if not isinstance(retries, Retry):
        retries = Retry.from_int(retries, redirect=redirect, default=self.retries)

    if release_conn is None:
        release_conn = response_kw.get("preload_content", True)

    # Check host
    if assert_same_host and not self.is_same_host(url):
        raise HostChangedError(self, url, retries)

    # Ensure that the URL we're connecting to is properly encoded
    if url.startswith("/"):
        url = six.ensure_str(_encode_target(url))
    else:
        url = six.ensure_str(parsed_url.url)

    conn = None

    # Track whether `conn` needs to be released before
    # returning/raising/recursing. Update this variable if necessary, and
    # leave `release_conn` constant throughout the function. That way, if
    # the function recurses, the original value of `release_conn` will be
    # passed down into the recursive call, and its value will be respected.
    #
    # See issue #651 [1] for details.
    #
    # [1] <https://github.com/urllib3/urllib3/issues/651>
    release_this_conn = release_conn

    http_tunnel_required = connection_requires_http_tunnel(
        self.proxy, self.proxy_config, destination_scheme
    )

    # Merge the proxy headers. Only done when not using HTTP CONNECT. We
    # have to copy the headers dict so we can safely change it without those
    # changes being reflected in anyone else's copy.
    if not http_tunnel_required:
        headers = headers.copy()
        headers.update(self.proxy_headers)

    # Must keep the exception bound to a separate variable or else Python 3
    # complains about UnboundLocalError.
    err = None

    # Keep track of whether we cleanly exited the except block. This
    # ensures we do proper cleanup in finally.
    clean_exit = False

    # Rewind body position, if needed. Record current position
    # for future rewinds in the event of a redirect/retry.
    body_pos = set_file_position(body, body_pos)

    try:
        # Request a connection from the queue.
        timeout_obj = self._get_timeout(timeout)
        conn = self._get_conn(timeout=pool_timeout)

        conn.timeout = timeout_obj.connect_timeout

        is_new_proxy_conn = self.proxy is not None and not getattr(
            conn, "sock", None
        )
        if is_new_proxy_conn and http_tunnel_required:
            self._prepare_proxy(conn)

        # Make the request on the httplib connection object.

      httplib_response = self._make_request(

            conn,
            method,
            url,
            timeout=timeout_obj,
            body=body,
            headers=headers,
            chunked=chunked,
        )

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/connectionpool.py:703:

self = <urllib3.connectionpool.HTTPConnectionPool object at 0x7f3e138ede10>
conn = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>, method = 'GET', url = '/'
timeout = Timeout(connect=60, read=60, total=None), chunked = False
httplib_request_kw = {'body': None, 'headers': {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '/', 'Connection': 'keep-alive'}}
timeout_obj = Timeout(connect=60, read=60, total=None)

def _make_request(
    self, conn, method, url, timeout=_Default, chunked=False, **httplib_request_kw
):
    """
    Perform a request on a given urllib connection object taken from our
    pool.

    :param conn:
        a connection from one of our connection pools

    :param timeout:
        Socket timeout in seconds for the request. This can be a
        float or integer, which will set the same timeout value for
        the socket connect and the socket read, or an instance of
        :class:`urllib3.util.Timeout`, which gives you more fine-grained
        control over your timeouts.
    """
    self.num_requests += 1

    timeout_obj = self._get_timeout(timeout)
    timeout_obj.start_connect()
    conn.timeout = timeout_obj.connect_timeout

    # Trigger any extra validation we need to do.
    try:
        self._validate_conn(conn)
    except (SocketTimeout, BaseSSLError) as e:
        # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.
        self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
        raise

    # conn.request() calls http.client.*.request, not the method in
    # urllib3.request. It also calls makefile (recv) on the socket.
    try:
        if chunked:
            conn.request_chunked(method, url, **httplib_request_kw)
        else:

          conn.request(method, url, **httplib_request_kw)

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/connectionpool.py:398:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>, method = 'GET', url = '/', body = None
headers = {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '/', 'Connection': 'keep-alive'}

def request(self, method, url, body=None, headers=None):
    if headers is None:
        headers = {}
    else:
        # Avoid modifying the headers passed into .request()
        headers = headers.copy()
    if "user-agent" not in (six.ensure_str(k.lower()) for k in headers):
        headers["User-Agent"] = _get_default_user_agent()

  super(HTTPConnection, self).request(method, url, body=body, headers=headers)

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/connection.py:239:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>, method = 'GET', url = '/', body = None
headers = {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '/', 'Connection': 'keep-alive'}

def request(self, method, url, body=None, headers={}, *,
            encode_chunked=False):
    """Send a complete request to the server."""

  self._send_request(method, url, body, headers, encode_chunked)

../../miniconda3/envs/sunshine/lib/python3.10/http/client.py:1282:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>, method = 'GET', url = '/', body = None
headers = {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '/', 'Connection': 'keep-alive'}
encode_chunked = False

def _send_request(self, method, url, body, headers, encode_chunked):
    # Honor explicitly requested Host: and Accept-Encoding: headers.
    header_names = frozenset(k.lower() for k in headers)
    skips = {}
    if 'host' in header_names:
        skips['skip_host'] = 1
    if 'accept-encoding' in header_names:
        skips['skip_accept_encoding'] = 1

    self.putrequest(method, url, **skips)

    # chunked encoding will happen if HTTP/1.1 is used and either
    # the caller passes encode_chunked=True or the following
    # conditions hold:
    # 1. content-length has not been explicitly set
    # 2. the body is a file or iterable, but not a str or bytes-like
    # 3. Transfer-Encoding has NOT been explicitly set by the caller

    if 'content-length' not in header_names:
        # only chunk body if not explicitly set for backwards
        # compatibility, assuming the client code is already handling the
        # chunking
        if 'transfer-encoding' not in header_names:
            # if content-length cannot be automatically determined, fall
            # back to chunked encoding
            encode_chunked = False
            content_length = self._get_content_length(body, method)
            if content_length is None:
                if body is not None:
                    if self.debuglevel > 0:
                        print('Unable to determine size of %r' % body)
                    encode_chunked = True
                    self.putheader('Transfer-Encoding', 'chunked')
            else:
                self.putheader('Content-Length', str(content_length))
    else:
        encode_chunked = False

    for hdr, value in headers.items():
        self.putheader(hdr, value)
    if isinstance(body, str):
        # RFC 2616 Section 3.7.1 says that text default has a
        # default charset of iso-8859-1.
        body = _encode(body, 'body')

  self.endheaders(body, encode_chunked=encode_chunked)

../../miniconda3/envs/sunshine/lib/python3.10/http/client.py:1328:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>, message_body = None

def endheaders(self, message_body=None, *, encode_chunked=False):
    """Indicate that the last header line has been sent to the server.

    This method sends the request to the server.  The optional message_body
    argument can be used to pass a message body associated with the
    request.
    """
    if self.__state == _CS_REQ_STARTED:
        self.__state = _CS_REQ_SENT
    else:
        raise CannotSendHeader()

  self._send_output(message_body, encode_chunked=encode_chunked)

../../miniconda3/envs/sunshine/lib/python3.10/http/client.py:1277:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>, message_body = None
encode_chunked = False

def _send_output(self, message_body=None, encode_chunked=False):
    """Send the currently buffered request and clear the buffer.

    Appends an extra \\r\\n to the buffer.
    A message_body may be specified, to be appended to the request.
    """
    self._buffer.extend((b"", b""))
    msg = b"\r\n".join(self._buffer)
    del self._buffer[:]

  self.send(msg)

../../miniconda3/envs/sunshine/lib/python3.10/http/client.py:1037:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>
data = b'GET / HTTP/1.1\r\nHost: localhost:9200\r\nUser-Agent: python-requests/2.28.1\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: /\r\nConnection: keep-alive\r\n\r\n'

def send(self, data):
    """Send `data' to the server.
    ``data`` can be a string object, a bytes object, an array object, a
    file-like object that supports a .read() method, or an iterable object.
    """

    if self.sock is None:
        if self.auto_open:

          self.connect()

../../miniconda3/envs/sunshine/lib/python3.10/http/client.py:975:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>

def connect(self):

  conn = self._new_conn()

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/connection.py:205:

self = <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>

def _new_conn(self):
    """Establish a socket connection and set nodelay settings on it.

    :return: New socket connection.
    """
    extra_kw = {}
    if self.source_address:
        extra_kw["source_address"] = self.source_address

    if self.socket_options:
        extra_kw["socket_options"] = self.socket_options

    try:
        conn = connection.create_connection(
            (self._dns_host, self.port), self.timeout, **extra_kw
        )

    except SocketTimeout:
        raise ConnectTimeoutError(
            self,
            "Connection to %s timed out. (connect timeout=%s)"
            % (self.host, self.timeout),
        )

    except SocketError as e:

      raise NewConnectionError(

            self, "Failed to establish a new connection: %s" % e
        )

E urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f3e138ed420>: Failed to establish a new connection: [Errno 111] Connection refused

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/connection.py:186: NewConnectionError

During handling of the above exception, another exception occurred:

self = <requests.adapters.HTTPAdapter object at 0x7f3e138ee260>, request = <PreparedRequest [GET]>
stream = False, timeout = Timeout(connect=60, read=60, total=None), verify = True, cert = None
proxies = OrderedDict()

def send(
    self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
):
    """Sends PreparedRequest object. Returns Response object.

    :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
    :param stream: (optional) Whether to stream the request content.
    :param timeout: (optional) How long to wait for the server to send
        data before giving up, as a float, or a :ref:`(connect timeout,
        read timeout) <timeouts>` tuple.
    :type timeout: float or tuple or urllib3 Timeout object
    :param verify: (optional) Either a boolean, in which case it controls whether
        we verify the server's TLS certificate, or a string, in which case it
        must be a path to a CA bundle to use
    :param cert: (optional) Any user-provided SSL certificate to be trusted.
    :param proxies: (optional) The proxies dictionary to apply to the request.
    :rtype: requests.Response
    """

    try:
        conn = self.get_connection(request.url, proxies)
    except LocationValueError as e:
        raise InvalidURL(e, request=request)

    self.cert_verify(conn, request.url, verify, cert)
    url = self.request_url(request, proxies)
    self.add_headers(
        request,
        stream=stream,
        timeout=timeout,
        verify=verify,
        cert=cert,
        proxies=proxies,
    )

    chunked = not (request.body is None or "Content-Length" in request.headers)

    if isinstance(timeout, tuple):
        try:
            connect, read = timeout
            timeout = TimeoutSauce(connect=connect, read=read)
        except ValueError:
            raise ValueError(
                f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                f"or a single float to set both timeouts to the same value."
            )
    elif isinstance(timeout, TimeoutSauce):
        pass
    else:
        timeout = TimeoutSauce(connect=timeout, read=timeout)

    try:
        if not chunked:

          resp = conn.urlopen(

                method=request.method,
                url=url,
                body=request.body,
                headers=request.headers,
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=timeout,
            )

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/requests/adapters.py:489:

self = <urllib3.connectionpool.HTTPConnectionPool object at 0x7f3e138ede10>, method = 'GET', url = '/'
body = None
headers = {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '/', 'Connection': 'keep-alive'}
retries = Retry(total=0, connect=None, read=False, redirect=None, status=None), redirect = False
assert_same_host = False, timeout = Timeout(connect=60, read=60, total=None), pool_timeout = None
release_conn = False, chunked = False, body_pos = None
response_kw = {'decode_content': False, 'preload_content': False}
parsed_url = Url(scheme=None, auth=None, host=None, port=None, path='/', query=None, fragment=None)
destination_scheme = None, conn = None, release_this_conn = True, http_tunnel_required = False, err = None
clean_exit = False

def urlopen(
    self,
    method,
    url,
    body=None,
    headers=None,
    retries=None,
    redirect=True,
    assert_same_host=True,
    timeout=_Default,
    pool_timeout=None,
    release_conn=None,
    chunked=False,
    body_pos=None,
    **response_kw
):
    """
    Get a connection from the pool and perform an HTTP request. This is the
    lowest level call for making a request, so you'll need to specify all
    the raw details.

    .. note::

       More commonly, it's appropriate to use a convenience method provided
       by :class:`.RequestMethods`, such as :meth:`request`.

    .. note::

       `release_conn` will only behave as expected if
       `preload_content=False` because we want to make
       `preload_content=False` the default behaviour someday soon without
       breaking backwards compatibility.

    :param method:
        HTTP request method (such as GET, POST, PUT, etc.)

    :param url:
        The URL to perform the request on.

    :param body:
        Data to send in the request body, either :class:`str`, :class:`bytes`,
        an iterable of :class:`str`/:class:`bytes`, or a file-like object.

    :param headers:
        Dictionary of custom headers to send, such as User-Agent,
        If-None-Match, etc. If None, pool headers are used. If provided,
        these headers completely replace any pool-specific headers.

    :param retries:
        Configure the number of retries to allow before raising a
        :class:`~urllib3.exceptions.MaxRetryError` exception.

        Pass ``None`` to retry until you receive a response. Pass a
        :class:`~urllib3.util.retry.Retry` object for fine-grained control
        over different types of retries.
        Pass an integer number to retry connection errors that many times,
        but no other types of errors. Pass zero to never retry.

        If ``False``, then retries are disabled and any exception is raised
        immediately. Also, instead of raising a MaxRetryError on redirects,
        the redirect response will be returned.

    :type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.

    :param redirect:
        If True, automatically handle redirects (status codes 301, 302,
        303, 307, 308). Each redirect counts as a retry. Disabling retries
        will disable redirect, too.

    :param assert_same_host:
        If ``True``, will make sure that the host of the pool requests is
        consistent else will raise HostChangedError. When ``False``, you can
        use the pool on an HTTP proxy and request foreign hosts.

    :param timeout:
        If specified, overrides the default timeout for this one
        request. It may be a float (in seconds) or an instance of
        :class:`urllib3.util.Timeout`.

    :param pool_timeout:
        If set and the pool is set to block=True, then this method will
        block for ``pool_timeout`` seconds and raise EmptyPoolError if no
        connection is available within the time period.

    :param release_conn:
        If False, then the urlopen call will not release the connection
        back into the pool once a response is received (but will release if
        you read the entire contents of the response such as when
        `preload_content=True`). This is useful if you're not preloading
        the response's content immediately. You will need to call
        ``r.release_conn()`` on the response ``r`` to return the connection
        back into the pool. If None, it takes the value of
        ``response_kw.get('preload_content', True)``.

    :param chunked:
        If True, urllib3 will send the body using chunked transfer
        encoding. Otherwise, urllib3 will send the body using the standard
        content-length form. Defaults to False.

    :param int body_pos:
        Position to seek to in file-like body in the event of a retry or
        redirect. Typically this won't need to be set because urllib3 will
        auto-populate the value when needed.

    :param \\**response_kw:
        Additional parameters are passed to
        :meth:`urllib3.response.HTTPResponse.from_httplib`
    """

    parsed_url = parse_url(url)
    destination_scheme = parsed_url.scheme

    if headers is None:
        headers = self.headers

    if not isinstance(retries, Retry):
        retries = Retry.from_int(retries, redirect=redirect, default=self.retries)

    if release_conn is None:
        release_conn = response_kw.get("preload_content", True)

    # Check host
    if assert_same_host and not self.is_same_host(url):
        raise HostChangedError(self, url, retries)

    # Ensure that the URL we're connecting to is properly encoded
    if url.startswith("/"):
        url = six.ensure_str(_encode_target(url))
    else:
        url = six.ensure_str(parsed_url.url)

    conn = None

    # Track whether `conn` needs to be released before
    # returning/raising/recursing. Update this variable if necessary, and
    # leave `release_conn` constant throughout the function. That way, if
    # the function recurses, the original value of `release_conn` will be
    # passed down into the recursive call, and its value will be respected.
    #
    # See issue #651 [1] for details.
    #
    # [1] <https://github.com/urllib3/urllib3/issues/651>
    release_this_conn = release_conn

    http_tunnel_required = connection_requires_http_tunnel(
        self.proxy, self.proxy_config, destination_scheme
    )

    # Merge the proxy headers. Only done when not using HTTP CONNECT. We
    # have to copy the headers dict so we can safely change it without those
    # changes being reflected in anyone else's copy.
    if not http_tunnel_required:
        headers = headers.copy()
        headers.update(self.proxy_headers)

    # Must keep the exception bound to a separate variable or else Python 3
    # complains about UnboundLocalError.
    err = None

    # Keep track of whether we cleanly exited the except block. This
    # ensures we do proper cleanup in finally.
    clean_exit = False

    # Rewind body position, if needed. Record current position
    # for future rewinds in the event of a redirect/retry.
    body_pos = set_file_position(body, body_pos)

    try:
        # Request a connection from the queue.
        timeout_obj = self._get_timeout(timeout)
        conn = self._get_conn(timeout=pool_timeout)

        conn.timeout = timeout_obj.connect_timeout

        is_new_proxy_conn = self.proxy is not None and not getattr(
            conn, "sock", None
        )
        if is_new_proxy_conn and http_tunnel_required:
            self._prepare_proxy(conn)

        # Make the request on the httplib connection object.
        httplib_response = self._make_request(
            conn,
            method,
            url,
            timeout=timeout_obj,
            body=body,
            headers=headers,
            chunked=chunked,
        )

        # If we're going to release the connection in ``finally:``, then
        # the response doesn't need to know about the connection. Otherwise
        # it will also try to release it and we'll have a double-release
        # mess.
        response_conn = conn if not release_conn else None

        # Pass method to Response for length checking
        response_kw["request_method"] = method

        # Import httplib's response into our own wrapper object
        response = self.ResponseCls.from_httplib(
            httplib_response,
            pool=self,
            connection=response_conn,
            retries=retries,
            **response_kw
        )

        # Everything went great!
        clean_exit = True

    except EmptyPoolError:
        # Didn't get a connection from the pool, no need to clean up
        clean_exit = True
        release_this_conn = False
        raise

    except (
        TimeoutError,
        HTTPException,
        SocketError,
        ProtocolError,
        BaseSSLError,
        SSLError,
        CertificateError,
    ) as e:
        # Discard the connection for these exceptions. It will be
        # replaced during the next _get_conn() call.
        clean_exit = False

        def _is_ssl_error_message_from_http_proxy(ssl_error):
            # We're trying to detect the message 'WRONG_VERSION_NUMBER' but
            # SSLErrors are kinda all over the place when it comes to the message,
            # so we try to cover our bases here!
            message = " ".join(re.split("[^a-z]", str(ssl_error).lower()))
            return (
                "wrong version number" in message or "unknown protocol" in message
            )

        # Try to detect a common user error with proxies which is to
        # set an HTTP proxy to be HTTPS when it should be 'http://'
        # (ie {'http': 'http://proxy', 'https': 'https://proxy'})
        # Instead we add a nice error message and point to a URL.
        if (
            isinstance(e, BaseSSLError)
            and self.proxy
            and _is_ssl_error_message_from_http_proxy(e)
            and conn.proxy
            and conn.proxy.scheme == "https"
        ):
            e = ProxyError(
                "Your proxy appears to only use HTTP and not HTTPS, "
                "try changing your proxy URL to be HTTP. See: "
                "https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html"
                "#https-proxy-error-http-proxy",
                SSLError(e),
            )
        elif isinstance(e, (BaseSSLError, CertificateError)):
            e = SSLError(e)
        elif isinstance(e, (SocketError, NewConnectionError)) and self.proxy:
            e = ProxyError("Cannot connect to proxy.", e)
        elif isinstance(e, (SocketError, HTTPException)):
            e = ProtocolError("Connection aborted.", e)

      retries = retries.increment(

            method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
        )

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/connectionpool.py:787:

self = Retry(total=0, connect=None, read=False, redirect=None, status=None), method = 'GET', url = '/'
response = None
error = NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3e138ed420>: Failed to establish a new connection: [Errno 111] Connection refused')
_pool = <urllib3.connectionpool.HTTPConnectionPool object at 0x7f3e138ede10>
_stacktrace = <traceback object at 0x7f3e138f5740>

def increment(
    self,
    method=None,
    url=None,
    response=None,
    error=None,
    _pool=None,
    _stacktrace=None,
):
    """Return a new Retry object with incremented retry counters.

    :param response: A response object, or None, if the server did not
        return a response.
    :type response: :class:`~urllib3.response.HTTPResponse`
    :param Exception error: An error encountered during the request, or
        None if the response was received successfully.

    :return: A new ``Retry`` object.
    """
    if self.total is False and error:
        # Disabled, indicate to re-raise the error.
        raise six.reraise(type(error), error, _stacktrace)

    total = self.total
    if total is not None:
        total -= 1

    connect = self.connect
    read = self.read
    redirect = self.redirect
    status_count = self.status
    other = self.other
    cause = "unknown"
    status = None
    redirect_location = None

    if error and self._is_connection_error(error):
        # Connect retry?
        if connect is False:
            raise six.reraise(type(error), error, _stacktrace)
        elif connect is not None:
            connect -= 1

    elif error and self._is_read_error(error):
        # Read retry?
        if read is False or not self._is_method_retryable(method):
            raise six.reraise(type(error), error, _stacktrace)
        elif read is not None:
            read -= 1

    elif error:
        # Other retry?
        if other is not None:
            other -= 1

    elif response and response.get_redirect_location():
        # Redirect retry?
        if redirect is not None:
            redirect -= 1
        cause = "too many redirects"
        redirect_location = response.get_redirect_location()
        status = response.status

    else:
        # Incrementing because of a server error like a 500 in
        # status_forcelist and the given method is in the allowed_methods
        cause = ResponseError.GENERIC_ERROR
        if response and response.status:
            if status_count is not None:
                status_count -= 1
            cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status)
            status = response.status

    history = self.history + (
        RequestHistory(method, url, error, status, redirect_location),
    )

    new_retry = self.new(
        total=total,
        connect=connect,
        read=read,
        redirect=redirect,
        status=status_count,
        other=other,
        history=history,
    )

    if new_retry.is_exhausted():

      raise MaxRetryError(_pool, url, error or ResponseError(cause))

E urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3e138ed420>: Failed to establish a new connection: [Errno 111] Connection refused'))

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/urllib3/util/retry.py:592: MaxRetryError

During handling of the above exception, another exception occurred:

_kube_client = <kubernetes.client.api.core_v1_api.CoreV1Api object at 0x7f3e138ed120>

def test_logging_cluster(_kube_client):
    """ tests whether all the pods in the cluster are healthy """

    # first the elasticsearch service
    # No path to kube config provided - will use default from $HOME/.kube/config
    with portforward.forward(NAMESPACE, ELASTICSEARCH_HOST, ELASTICSEARCH_PORT, ELASTICSEARCH_PORT):

      response = requests.get(f"http://{KUBE_ENDPOINT}:{ELASTICSEARCH_PORT}", timeout=TIMEOUT_SEC)

logging/tests/kube-local/test_logging_cluster.py:57:

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/requests/api.py:73: in get
return request("get", url, params=params, **kwargs)
../../miniconda3/envs/sunshine/lib/python3.10/site-packages/requests/api.py:59: in request
return session.request(method=method, url=url, **kwargs)
../../miniconda3/envs/sunshine/lib/python3.10/site-packages/requests/sessions.py:587: in request
resp = self.send(prep, **send_kwargs)
../../miniconda3/envs/sunshine/lib/python3.10/site-packages/requests/sessions.py:701: in send
r = adapter.send(request, **kwargs)

self = <requests.adapters.HTTPAdapter object at 0x7f3e138ee260>, request = <PreparedRequest [GET]>
stream = False, timeout = Timeout(connect=60, read=60, total=None), verify = True, cert = None
proxies = OrderedDict()

def send(
    self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None
):
    """Sends PreparedRequest object. Returns Response object.

    :param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
    :param stream: (optional) Whether to stream the request content.
    :param timeout: (optional) How long to wait for the server to send
        data before giving up, as a float, or a :ref:`(connect timeout,
        read timeout) <timeouts>` tuple.
    :type timeout: float or tuple or urllib3 Timeout object
    :param verify: (optional) Either a boolean, in which case it controls whether
        we verify the server's TLS certificate, or a string, in which case it
        must be a path to a CA bundle to use
    :param cert: (optional) Any user-provided SSL certificate to be trusted.
    :param proxies: (optional) The proxies dictionary to apply to the request.
    :rtype: requests.Response
    """

    try:
        conn = self.get_connection(request.url, proxies)
    except LocationValueError as e:
        raise InvalidURL(e, request=request)

    self.cert_verify(conn, request.url, verify, cert)
    url = self.request_url(request, proxies)
    self.add_headers(
        request,
        stream=stream,
        timeout=timeout,
        verify=verify,
        cert=cert,
        proxies=proxies,
    )

    chunked = not (request.body is None or "Content-Length" in request.headers)

    if isinstance(timeout, tuple):
        try:
            connect, read = timeout
            timeout = TimeoutSauce(connect=connect, read=read)
        except ValueError:
            raise ValueError(
                f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
                f"or a single float to set both timeouts to the same value."
            )
    elif isinstance(timeout, TimeoutSauce):
        pass
    else:
        timeout = TimeoutSauce(connect=timeout, read=timeout)

    try:
        if not chunked:
            resp = conn.urlopen(
                method=request.method,
                url=url,
                body=request.body,
                headers=request.headers,
                redirect=False,
                assert_same_host=False,
                preload_content=False,
                decode_content=False,
                retries=self.max_retries,
                timeout=timeout,
            )

        # Send the request.
        else:
            if hasattr(conn, "proxy_pool"):
                conn = conn.proxy_pool

            low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)

            try:
                skip_host = "Host" in request.headers
                low_conn.putrequest(
                    request.method,
                    url,
                    skip_accept_encoding=True,
                    skip_host=skip_host,
                )

                for header, value in request.headers.items():
                    low_conn.putheader(header, value)

                low_conn.endheaders()

                for i in request.body:
                    low_conn.send(hex(len(i))[2:].encode("utf-8"))
                    low_conn.send(b"\r\n")
                    low_conn.send(i)
                    low_conn.send(b"\r\n")
                low_conn.send(b"0\r\n\r\n")

                # Receive the response from the server
                r = low_conn.getresponse()

                resp = HTTPResponse.from_httplib(
                    r,
                    pool=conn,
                    connection=low_conn,
                    preload_content=False,
                    decode_content=False,
                )
            except Exception:
                # If we hit any problems here, clean up the connection.
                # Then, raise so that we can handle the actual exception.
                low_conn.close()
                raise

    except (ProtocolError, OSError) as err:
        raise ConnectionError(err, request=request)

    except MaxRetryError as e:
        if isinstance(e.reason, ConnectTimeoutError):
            # TODO: Remove this in 3.0.0: see #2811
            if not isinstance(e.reason, NewConnectionError):
                raise ConnectTimeout(e, request=request)

        if isinstance(e.reason, ResponseError):
            raise RetryError(e, request=request)

        if isinstance(e.reason, _ProxyError):
            raise ProxyError(e, request=request)

        if isinstance(e.reason, _SSLError):
            # This branch is for urllib3 v1.22 and later.
            raise SSLError(e, request=request)

      raise ConnectionError(e, request=request)

E requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3e138ed420>: Failed to establish a new connection: [Errno 111] Connection refused'))

../../miniconda3/envs/sunshine/lib/python3.10/site-packages/requests/adapters.py:565: ConnectionError
------------------------------------------ Captured stdout setup -------------------------------------------
scheduler, Healthy
controller-manager, Healthy
========================================= short test summary info ==========================================
FAILED logging/tests/kube-local/test_logging_cluster.py::test_logging_cluster - requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries excee...
===================================== 1 failed, 1 deselected in 1.39s ======================================
make: *** [makefile:16: testops] Error 1

commented

Hey @drawal1

thanks for the detailed report. The current solution is the kwarg wait time. But I admit it would be safer to check the pod status.

I will add it to the road map.

Replace waiting by a real check of the pod status

commented

Hi @drawal1,

the lib is now able to check the status of the pod and its container and wait till it becomes ready (but only max. 60 seconds).

Would you like to try it out? It is currently only available on the main branch.

commented

@drawal1 this should also be included. Could you test if it works as you expected?

commented
commented

The mentioned line is the sleep method.

Could you please provide a minimal example to reproduce this issue?

Portforward should detect the state of a target pod and wait.

commented
commented
commented

🙏 Thx for your feedback

I will close this issue for now. Feel free to re-open it can be steady reproduced.