libssl.so.3 observability is broken

[Daniel Kleinstein]

Describe the bug
OpenSSL visibility seems to be broken, at least with libssl.so.3.

• On one of my cluster nodes, I am running a Python 3.10 program that performs outgoing HTTPS traffic against an API endpoint

• The relevant PEM's log shows the following line:

uprobe_manager.cc:624] Attaching OpenSSL uprobes on dynamic library failed for PID 226977: Internal : libssl not found [path = /proc/226977/root/proc/226977/root/usr/lib/x86_64-linux-gnu/libssl.so.3]
i.e. it looks like it's trying to access /proc/226977/root/proc/226977/root/usr/lib/x86_64-linux-gnu/libssl.so.3 and failing

• On the same node that the Python/PEM are running in, I ran a privileged K8s pod that allowed me to run some commands in the host's namespaces - this allowed me to pinpoint the problem: /proc/226977/root/proc/226977/ does not exist. 226977 is indeed the PID of the Python program relative to the host, but it shouldn't appear twice - the last PID should be the PID relative to the container.

To Reproduce

• Run kubectl run ubuntu --image=ubuntu --rm -it -- bash
• Inside that container, create the following Python script as http-traffic.py and run ./http-traffic.py (installing dependencies as necessary):

#!/usr/bin/env python3

import argparse
import requests
import sys
import time

DEFAULT_ENDPOINT = "https://api.sampleapis.com/countries/countries"

def fetch_endpoint(endpoint, interval=1, count=None, quiet=False):
    global total_requests
    total_requests = 0

    while True:
        response = requests.get(endpoint)
        total_requests += 1
        if not quiet:
            print(response.text)

        if count is not None and total_requests >= count:
            break

        time.sleep(interval)
        if not quiet:
            sys.stderr.write(f"Total requests made: {total_requests}\n")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Access an endpoint at regular intervals.")
    parser.add_argument("-e", "--endpoint", default=DEFAULT_ENDPOINT, help=f"URL of the endpoint to access (default is {DEFAULT_ENDPOINT}).")
    parser.add_argument("-i", "--interval", type=float, default=1, help="Interval in seconds between requests (default is 1 second).")
    parser.add_argument("-c", "--count", type=int, help="Limit the number of requests to this count. If not provided, the script will run indefinitely.")
    parser.add_argument("-q", "--quiet", action="store_true", help="Suppress output. If provided, the response text and total requests made won't be printed.")
    args = parser.parse_args()

    start_time = time.time()

    try:
        fetch_endpoint(args.endpoint, args.interval, args.count, args.quiet)
    except Exception as e:
        print(f"An exception occurred: {e}")
    finally:
        end_time = time.time()
        elapsed_time = end_time - start_time
        print(f"\n\nTotal number of requests: {total_requests}")
        print(f"Elapsed time: {elapsed_time:.2f} seconds")

Expected behavior
Since the Python program makes continuous requests against an HTTPS endpoint
https://api.sampleapis.com/countries/countries - I expect the requests to show up under the px/http_data script (filtered to "ubuntu" in the source_filter) - but nothing appears.

Screenshots

Logs
Please attach the logs by running the following command:

./px collect-logs

I ran this command immediately after running px deploy and then http-traffic.py - logs attached.
pixie_logs_20231014150647.zip

App information (please complete the following information):

• Pixie version
• K8s cluster version
• Node Kernel version
• Browser version

^ Not filling out because I'm not sure and I believe it's irrelevant for the issue.

Additional context
I have created a POC PR here that seemingly solves the issue - #1735. When deploying Pixie with these changes, I can see the HTTPS traffic.

if (spec.probe_fn == "probe_ret_SSL_new_syscall_fd_access") {
  spec.probe_fn = "probe_ret_SSL_new";
}

was a bit of a hack - but BCC failed without it with the error:

Attaching OpenSSL uprobes on dynamic library failed for PID 953441: Internal : Can't find start of function probe_ret_SSL_new_syscall_fd_access

[Dom Delnano]

@danielkleinstein thanks for the detailed bug report and as a result I was able to reproduce the problem. This logic has existed for quite some time, so I was confident that this situation was handled properly. I need to investigate what changes have happened, since my suspicion is that a regression was introduced at some point. I believe this would impact all of OpenSSL tracing (not just OpenSSL v3) and as a result want to make sure we understand the situation fully.

Regarding the probe_ret_SSL_new probe, that is NodeJS specific. Our code is not structured well at the moment to properly reflect that, however, that probe should not be attached to non NodeJS processes.

[Dom Delnano]

I think ccb32d5 might be the change that introduce the regression. The FilePathResolver class was mount namespace aware, but it was replaced wholesale with the ProcPidRootPath.

[Daniel Kleinstein]

Wow, that was some quick debugging!
I'm wondering if this means that all usage of ProcPidRootPath is broken? e.g. I see it's used for Java profiling.

Two questions:

1. If ProcPidRootPath replaces all usages of a mount namespace-aware resolver, then it seems that the fix should be embedded directly inside that function... but it also seems like ProcPidRootPath makes an effort to stay away from lower-level details like actual /proc/pid/status parsing.

   Based on my very limited experience with Pixie's codebase my intuition would be to pass a pointer to a ProcParser into ProcPidRootPath and have the function use the ProcParser to populate the mount namespace PID - it seems like there's an instantiated ProcParser in every place that ProcPidRootPath is used.

   What should be the fix in your view? I'd be very happy to help out with the fix if you're open to it.

2. It's slightly concerning that the commit that introduced the regression was committed in December 2022 - I'm wondering what testing infrastructure you guys have in place to deal with regressions like these? If necessary I'd be very happy to help out there as well.

[Dom Delnano]

As for 1, I'm not sure. I believe ProcPidRootPath is what we want to do the majority of the time, and as far as I know the other usages of it are correct. My initial instinct would be to reinstate the FilePathResolver code, but I don't have the context currently for why it was phased out.

Regarding 2, It definitely is concerning that this was missed. Our test suite contains a substantial amount of end to end BPF tests ("trace bpf tests" tests). My guess is that the way we are emulating stirling does not exercise stirling itself and the target containers in isolated PID namespaces.

We absolutely need to ensure that these tests (or another mechanism) verify that the UprobeManager code is exercised under these conditions since that is what we expect in a k8s cluster deployment.

We always welcome contributions, so if you have interest in these areas it would be greatly appreciated.

[Daniel Kleinstein]

My initial instinct would be to reinstate the FilePathResolver code, but I don't have the context currently for why it was phased out.

If the rest of the usages are correct, it's possible that this is overkill - it'd be easy to overload ProcPidRootPath to accept an additional ProcParser parameter, and call this overloaded (and namespace PID-resolving) function specifically in UProbeManager, leaving other usages untouched (and if any of them are discovered to be similarly broken their usage can be fixed too).

This is admittedly a bit of a hacky solution but it seems much smaller in scope than reintroducing the removed class.

[Pete Stevenson]

Hi @danielkleinstein and @ddelnano -

Could you please take a look at this PR: #1740.

I think the issue was that we applied the ProcPidPath method twice. This problem was likely masked in our test env. and perhaps other cases also. It is not masked when the target process is in a container. The following form would work:

/proc/<pid>/root/proc/<ns-pid>/root/<path>

But using the namespace pid is just redundant and not really what we intended. If #1740 helps, it is probably the fix we want.

[Daniel Kleinstein]

Hi Pete, your PR fixes the issue 🚀.

I see you also modified AttachGrpcCUProbesOnDynamicPythonLib, I haven't directly checked this fix but it seems consistent.

[Dom Delnano]

@etep that looks like the correct fix and that confirms that the regression was introduced in ccb32d5. So I believe the impact of this is that dynamically linked OpenSSL tracing for containers within PID namespaces would have been broken since late 2022 or early 2023 (vizier releases weren't published to GitHub around that timeframe, so not sure how to confirm which release it was included in easily). Containers running within the host PID namespace or processes outside of k8s would have be unaffected.

Unfortunately our end to end tests heavily rely on running containers with the host PID namespace (source). Converting all the tests to support that might be challenging, so maybe it would be best to have a single use case or
a UprobeManager specific test to catch this situation.

[Dom Delnano]

@danielkleinstein apologies that it has taken some time for the fix to make it in a release, but v0.14.8 addresses this issue! We really appreciate your help in surfacing this issue and debugging it with us!

[Daniel Kleinstein]

@ddelnano Sure! Thanks in turn for responding so promptly to the issue.

And thanks more broadly for your guys' work on Pixie, I've learned a ton from your project.