Memory usage keeps increasing during unittest run

[Arjan-Floorganise]

Is there an existing issue for this?

• I have searched the existing issues

Description

I've encountered an issue with the ABP unittest setup where the memory is not correctly de-allocated after each test run.
From what I can find, it seems like a DI container is not properly disposed, resulting in an additional (left over) DI container per test.

This is what I found:

As you can see here, after each test run there is a small increase of the used memory

On the screenshot below you can see the memory allocation delta between run 3 and 4:

And between 4 and 5:

I noticed here that the reference to the Autofac.Core.SelfComponentRegistrationInfo increases every run by 1.
Going through the AutoFac code, it seems like this object references a ConcurrentDictionary containing the resolved instances.

In the example/screenshots provided, the impact is small, however in a larger project with about 2000 DI registrations & over 3000 tests, the impact is quite significant for our build server.

While this problem might be a combination of ABP & AutoFac, I've created the report here as:

1. This is the default ABP project containing this issue
2. As far as I understand, if ABP removes all references to the AutoFac container, it should be removed from memory anyways.

If this also requires a bug report at AutoFac please let me know.

Reproduction Steps

Base setup: The default ABP project from the 'get started' section (https://abp.io/get-started)

• Project type: Multi-layer application
• UI Framework: MVC
• UI Theme: LeptonX Lite Theme
• Database Provider: Entity Framework Core
• Database Management System: Sql Server
• Moblie: None
• Tiered: unchecked
• Preview: unchecked
• Create Solution Folder: checked

In the Acme.BookStore.EntityFrameworkCore.Tests.SampleRepositoryTests duplicate the existing test Should_Query_AppUser a few times to make sure that there are multiple tests in the debug session (for me, all tests are identical)

Place a breakpoint on BookStoreEntityFrameworkCoreTestModule.ConfigureServices in order to take a memory snapshot at every startup

Expected behavior

The memory usage remains similar while running each test

Actual behavior

The memory usage increases after each test

Regression?

No response

Known Workarounds

I've implemented a few 'hack arounds' which reduce the amount of allocated memory, but does not solve the problem.

1. Instead of using the UseAutoFac() extension method, I've copied the source and applied it manually, while storing the ContainerBuilder in a private variable. I've used the override of the Dispose method in the AbpIntegratedTest to call _containerBuilder?.ComponentRegistryBuilder?.Dispose() as this instance is not disposed after calling the base.Dispose()
   

2. I've manually disposed the RootServiceProvider as this instance does not get disposed by the AbpIntegratedTest (from what I can see, this doesn't really impact the allocated memory)
   

3. The hackiest addition: The _containerBuilder.ComponentRegistryBuilder contains a _registeredServicesTracker (IRegisteredServicesTracker -> DefaultRegisteredServicesTracker) which contains the _serviceInfo (ConcurrentDictionary<Service, ServiceRegistrationInfo>). Via reflection, I call the Clear method on this dictionary, this seems to clear up about 50% of the allocated memory. For some reason, the Dispose of the DefaultRegisteredServicesTracker does not clear this dictionary.

Version

8.1

User Interface

Common (Default)

Database Provider

EF Core (Default)

Tiered or separate authentication server

None (Default)

Operation System

Windows (Default)

Other information

No response

[maliming]

hi

Can you share your test project?

[Arjan-Floorganise]

Acme.BookStore.zip
Hi,

See the attached zip, it's the 'default' project as provided on the get started page installed via the CLI with the Should_Query_AppUser test copied a few times

[maliming]

hi @Arjan-Floorganise

Can you try this? dispose the RootServiceProvider.

[Arjan-Floorganise]

Hi @maliming

As mentioned in the original ticket, the disposing of the RootServiceProvider does not seem to have an impact on the issue.
Please see the screenshots below:

Comparison between 4 and 5

*Edit: I forgot to mention, but this is the result with the override of the dispose that you've posted

[maliming]

ok, Thanks, I will continue to check this.

[maliming]

hi

The AbpIntegratedTest class is very simple. I checked all the code, and there seem to be no problems.

And I saw an issue with Autofac, it is similar to yours.

autofac/Autofac.Extensions.DependencyInjection#113

[Arjan-Floorganise]

Hi,

The issue looks similar indeed, however tillig states there that the issue is likely caused by the improper use of the AutoFac library.
Do you agree with his conclusion - in that case something in the startup of the test is using autofac in an incorrect manner?
Or do you conclude that autofac has a memory leak and that there are no issues on the ABP side?
I can't fully get this from your comment.

The test class you mentioned does indeed clean all it's resources after your PR; however the provided boilerplate also bootstraps the autofac setup (among other things). There are things in there that are not properly disposed like the ComponentRegistryBuilder on the ContainerBuilder. The ContainerBuilder itself is created by the UseAutoFac extension method in the Volo.Abp.AbpAutofacAbpApplicationCreationOptionsExtensions

I've validated the results I found between different test platforms, xUnit - provided by the default template & MSTest, but both yield the same results. So the test platform can, in my opinion, be ignored for now.

[maliming]

hi

Thanks for your info, I will deeply investigate this case over the weekend.

[maliming]

hi @Arjan-Floorganise
Can you try to update your BookStoreDomainTestModule as follow:

using System;
using System.Linq;
using System.Text.Json.Serialization.Metadata;
using Microsoft.Extensions.DependencyInjection;
using Volo.Abp.Json.SystemTextJson;
using Volo.Abp.Json.SystemTextJson.JsonConverters;
using Volo.Abp.Json.SystemTextJson.Modifiers;
using Volo.Abp.Modularity;
using Volo.Abp.Reflection;
using Volo.Abp.Timing;

namespace Acme.BookStore;

[DependsOn(
    typeof(BookStoreDomainModule),
    typeof(BookStoreTestBaseModule)
)]
public class BookStoreDomainTestModule : AbpModule
{
    public override void ConfigureServices(ServiceConfigurationContext context)
    {
        context.Services.AddOptions<AbpSystemTextJsonSerializerModifiersOptions>()
            .Configure<IServiceProvider>((options, rootServiceProvider) =>
            {
                options.Modifiers.RemoveAll(x => x.Target is AbpDateTimeConverterModifier);
                options.Modifiers.Add(new MyAbpDateTimeConverterModifier(
                    rootServiceProvider.GetRequiredService<AbpDateTimeConverter>(),
                    rootServiceProvider.GetRequiredService<AbpNullableDateTimeConverter>()).CreateModifyAction());
            });
    }
}

public class MyAbpDateTimeConverterModifier
{
    private AbpDateTimeConverter _abpDateTimeConverter;
    private AbpNullableDateTimeConverter _abpNullableDateTimeConverter;

    public MyAbpDateTimeConverterModifier(AbpDateTimeConverter abpDateTimeConverter, AbpNullableDateTimeConverter abpNullableDateTimeConverter)
    {
        _abpDateTimeConverter = abpDateTimeConverter;
        _abpNullableDateTimeConverter = abpNullableDateTimeConverter;
    }

    public Action<JsonTypeInfo> CreateModifyAction()
    {
        return Modify;
    }

    private void Modify(JsonTypeInfo jsonTypeInfo)
    {
        if (ReflectionHelper.GetAttributesOfMemberOrDeclaringType<DisableDateTimeNormalizationAttribute>(jsonTypeInfo.Type).Any())
        {
            return;
        }

        foreach (var property in jsonTypeInfo.Properties.Where(x => x.PropertyType == typeof(DateTime) || x.PropertyType == typeof(DateTime?)))
        {
            if (property.AttributeProvider == null ||
                !property.AttributeProvider.GetCustomAttributes(typeof(DisableDateTimeNormalizationAttribute), false).Any())
            {
                property.CustomConverter = property.PropertyType == typeof(DateTime)
                    ? _abpDateTimeConverter
                    : _abpNullableDateTimeConverter;
            }
        }
    }
}

Before:

After:

[maliming]

#20091 will fix this issue.

[Arjan-Floorganise]

Hi,

Thanks for the response, that latest patch solves a lot!
I've ran the tests again with both code snippets you've posted (dispose of the rootprovider + the date time convert logic) and got the following result.

On the EntityFrameworkCore tests:

On the Domain tests:

As you can see, by far most of the issue has been solved, however I'm not getting the same results as you.
To re-state the changes I made:

• Dispose override
• BookStoreDomainTestModule.cs contents replaced with the snippet

Are there any other changes you've applied to get the results you've posted?

[maliming]

hi

Just update the BookStoreDomainTestModule file in your Acme.BookStore.zip and run the SampleRepositoryTests.

Here is the dotmemory workspace file.
https://we.tl/t-i2funqBdCR

[maliming]

btw, You can update BookStoreEntityFrameworkCoreTestModule as follows:

using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Microsoft.EntityFrameworkCore.Storage;
using Microsoft.Extensions.DependencyInjection;
using Volo.Abp;
using Volo.Abp.EntityFrameworkCore;
using Volo.Abp.EntityFrameworkCore.Sqlite;
using Volo.Abp.FeatureManagement;
using Volo.Abp.Modularity;
using Volo.Abp.OpenIddict;
using Volo.Abp.OpenIddict.Tokens;
using Volo.Abp.PermissionManagement;
using Volo.Abp.SettingManagement;
using Volo.Abp.Uow;

namespace Acme.BookStore.EntityFrameworkCore;

[DependsOn(
    typeof(BookStoreApplicationTestModule),
    typeof(BookStoreEntityFrameworkCoreModule),
    typeof(AbpEntityFrameworkCoreSqliteModule)
    )]
public class BookStoreEntityFrameworkCoreTestModule : AbpModule
{
    private SqliteConnection? _sqliteConnection;

    public override void ConfigureServices(ServiceConfigurationContext context)
    {
        Configure<FeatureManagementOptions>(options =>
        {
            options.SaveStaticFeaturesToDatabase = false;
            options.IsDynamicFeatureStoreEnabled = false;
        });
        Configure<PermissionManagementOptions>(options =>
        {
            options.SaveStaticPermissionsToDatabase = false;
            options.IsDynamicPermissionStoreEnabled = false;
        });
        Configure<SettingManagementOptions>(options =>
        {
            options.SaveStaticSettingsToDatabase = false;
            options.IsDynamicSettingStoreEnabled = false;
        });
        Configure<TokenCleanupOptions>(options =>
        {
            options.IsCleanupEnabled = false;
        });
        context.Services.AddAlwaysDisableUnitOfWorkTransaction();

        ConfigureInMemorySqlite(context.Services);
    }

    private void ConfigureInMemorySqlite(IServiceCollection services)
    {
        _sqliteConnection = CreateDatabaseAndGetConnection();

        services.Configure<AbpDbContextOptions>(options =>
        {
            options.Configure(context =>
            {
                context.DbContextOptions.UseSqlite(_sqliteConnection);
            });
        });
    }

    public override void OnApplicationShutdown(ApplicationShutdownContext context)
    {
        _sqliteConnection?.Dispose();
    }

    private static SqliteConnection CreateDatabaseAndGetConnection()
    {
        var connection = new SqliteConnection("Data Source=:memory:");
        connection.Open();

        var options = new DbContextOptionsBuilder<BookStoreDbContext>()
            .UseSqlite(connection)
            .Options;

        using (var context = new BookStoreDbContext(options))
        {
            context.GetService<IRelationalDatabaseCreator>().CreateTables();
        }

        return connection;
    }
}

[Arjan-Floorganise]

For anyone coming across this issue:
The results I posted are results from Visual Studio; VS seems to allow the memory to grow in size a lot more before it cleans it.
While validating the fixes posted here on a larger project, the memory gets cleaned as soon as it hits the 1GB in usage.
So the fix here works fine, but VS simply doesn't clean the allocations that aggressively.