Microsoft.Extensions.Diagnostics.ResourceMonitoring could not find a part of the path '/sys/fs/cgroup/memory' in docker container

[Alan.Liu]

Description

The Microsoft.Extensions.Diagnostics.HealthChecks.ResourceUtilization throw unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path '/sys/fs/cgroup/memory/memory.limit_in_bytes'.

When I enter the docker container, I cannot find the /sys/fs/cgroup/memory directory:

Reproduction Steps

Create a simple console app:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <RootNamespace>resource_monitoring</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.Diagnostics.ResourceMonitoring" Version="8.0.0" />
    <PackageReference Include="Spectre.Console" Version="0.48.0" />
  </ItemGroup>

</Project>

Update the Program.cs with the following:

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.ResourceMonitoring;
using Microsoft.Extensions.Logging;
using Spectre.Console;

var services = new ServiceCollection()
    .AddLogging(static builder => builder.AddConsole())
    .AddResourceMonitoring();

var provider = services.BuildServiceProvider();

var monitor = provider.GetRequiredService<IResourceMonitor>();
// </setup>

using var cancellationTokenSource = new CancellationTokenSource();
var token = cancellationTokenSource.Token;

// <monitor>
await StartMonitoringAsync(monitor, token);

async Task StartMonitoringAsync(IResourceMonitor monitor, CancellationToken cancellationToken)
{
    var table = new Table()
        .Centered()
        .Title("Resource Monitoring", new Style(foreground: Color.Purple, decoration: Decoration.Bold))
        .Caption("Updates every three seconds. *GTD: Guaranteed ", new Style(decoration: Decoration.Dim))
        .RoundedBorder()
        .BorderColor(Color.Cyan1)
        .AddColumns(
        [
            new TableColumn("Time").Centered(),
            new TableColumn("CPU %").Centered(),
            new TableColumn("Memory %").Centered(),
            new TableColumn("Memory (bytes)").Centered(),
            new TableColumn("GTD / Max Memory (bytes)").Centered(),
            new TableColumn("GTD / Max CPU (units)").Centered(),
        ]);

    await AnsiConsole.Live(table)
        .StartAsync(async ctx =>
        {
            var window = TimeSpan.FromSeconds(3);
            while (cancellationToken.IsCancellationRequested is false)
            {
                var utilization = monitor.GetUtilization(window);
                var resources = utilization.SystemResources;

                Console.WriteLine($"{utilization.CpuUsedPercentage},{utilization.MemoryUsedPercentage}");

                table.AddRow(
                    [
                        $"{DateTime.Now:T}",
                        $"{utilization.CpuUsedPercentage:p}",
                        $"{utilization.MemoryUsedPercentage:p}",
                        $"{utilization.MemoryUsedInBytes:#,#}",
                        $"{resources.GuaranteedMemoryInBytes:#,#} / {resources.MaximumMemoryInBytes:#,#}",
                        $"{resources.GuaranteedCpuUnits} / {resources.MaximumCpuUnits}",
                    ]);

                ctx.Refresh();
                await Task.Delay(window);
            }
        });

    Console.CancelKeyPress += (_, e) =>
    {
        e.Cancel = true;
        cancellationTokenSource.Cancel();
    };
}

This results in the following:

Expected behavior

Both CPU and memory resource utilization

Actual behavior

throw unhandled exception.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[Martin Taillefer]

Hi there,

Can you supply some details on the execution environment? Which Linux distro and what version?

Thanks.

[Alan.Liu]

Hi，

Sure, here are the details of the execution environment:

root@PveHost612:/# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

root@PveHost612:/# uname -r
6.5.11-7-pve

Let me know if you need any further information.

Thanks,
@geeknoid

[Martin Obrátil]

Hi @lxyruanjian,

I've investigated the issue you're experiencing with Microsoft.Extensions.Diagnostics.ResourceMonitoring. The exception System.IO.DirectoryNotFoundException: Could not find a part of the path '/sys/fs/cgroup/memory/memory.limit_in_bytes' indicates that the library is looking for cgroups v1 specific paths.

Given that you are using Debian GNU/Linux 12 (Bookworm), which likely defaults to cgroups v2, this seems to be a compatibility issue with cgroups v2 in the library. Cgroups v2 has a different filesystem layout compared to cgroups v1 and does not use the /sys/fs/cgroup/memory structure that the library is attempting to access.

This lack of compatibility might be causing the error you're encountering.

As of now, there doesn't appear to be explicit documentation or mention of support for cgroups v2 in Microsoft.Extensions.Diagnostics.ResourceMonitoring. I'll create a PR to correct this.

A potential workaround could be to run your Docker container with cgroups v1, if that's feasible for your setup.

[Martin Obrátil]

I created a new issue specifically for cgroups v2 support #4885

[Alan.Liu]

Hi @mobratil ,

Thank you for the detailed investigation. It's clear that the issue is related to cgroups v2 incompatibility with Microsoft.Extensions.Diagnostics.ResourceMonitoring on Debian GNU/Linux 12 (Bookworm). Your explanation makes sense.

I appreciate your efforts in creating a PR to address the compatibility issue. I'll keep an eye on it.

Regarding the potential workaround, I'll explore running the Docker container with cgroups v1 to see if it fits into my setup.

If there's any additional information or testing you need from my side, please let me know.

Thanks again.