Stackoverflow when compiling callsite chain on a background thread
davidfowl opened this issue · comments
See for details aspnet/Mvc#6410.
There isn't an isolated repro yet.
I can confirm we have this issue as well. I don't know if this is the solution, but I can say that refactoring helped for this instance. This exception is very troubling though as it crashes kestrel
Service with dependencies:
Controller
-- Service1
-- Service2
---- Service1
What helped is moving common code to a separate Service
Controller
-- Service1
---- Service3 (common methods from Service1)
-- Service2
---- Service3 (common methods from Service1)
@pakrym can you take a look?
Could anyone provide an isolated repro app?
cc @alohaninja, @xxjthxx
@muratg as I said before, this is a bug we where experiencing on certain data and certain e2e test in our environment. Although the setup was very specific the bug was very real because it was consistently reproduced it with our data. We didn't find simple repro to send it to you unfortunately. As a last resort we switched to Autofac, which helped.
This bug doesn't seem to have a high number of hits so it's likely something specific that the 2 reported applications are doing. We need more information before we can do anything else here.
We are also running into this issue now, we have a few services that inject a fairly large dependency tree of services. Whats strange is that it seems to run well in 32 bit mode but we do get the stack overflow every so often. Once I run it in 64 bit, the first request loads fine and brings back the page with no memory spike, then on the 2nd request, the memory spikes like crazy until it consumes everything on my machine.
I'm going to try autofac or simple injector to see if it helps.
If anyone has any ideas id appreciate it.
@papaytl185 can you please share a memory dump from when the process starts to grow memory to pakrym@microsoft.com. It might help us to diagnose the issue because we weren't able to get any repos until now.
@pakrym the memory dump is going to be huge like 5 gb plus, if I cant get it to dump fast enough before it grows to 20gb, I can't email that.
Do you have another recommendation?
On a side note, I tried lightinject last night, finally got everything running, but it takes awhile to boot up to my login page. After I click login, the site just hangs. So I must have a dependency wrong or something somewhere, maybe the lifetime IDK.
While I was messing around with lightinject, I found a setting called AddControllersAsServices() which is part of IMvcBuilder. When I used this option with default microsoft dependency injection, the memory spiked to 4-5gb and then settled back down to 3ish without crashing my app. So maybe that can help others.
I still want to figure out why lightinject isn't working correctly and if my service dependencies are too big for microsoft dependency injection to see how lightinject handles it. Or if I have some singleton or other lifetime scope incorrect masking an issue somewhere..
When I used this option with default microsoft dependency injection, the memory spiked to 4-5gb and then settled back down to 3ish without crashing my app. So maybe that can help others.
@papaytl185 It might make sense to do a live debug session with you. That just sounds insane.
@davidfowl that would be great, I could really use some help with this. How / when should we set this up? I could do a live assist with you and walk you through the startup etc.
@davidfowl Does the attached DebugDiag memory analysis report help identify anything for you?
memorydump.pdf
@papaytl185 It definitely shows concerning amounts of Expression related data structures that are used to optimize DI operations.
Do you have any recommendations on what I should do?
Are you able to take timeline profile with dotTrace? It would show both memory allocation source and where time is spent during startup.
Ill try to use dotTrace, I keep trying to use visual studio to grab a snapshot but it crashes due to memory.
Some other tidbits.
It gets through the startup fine and I can load my home controller with just static html.
Using Microsoft Depedency Injection
Once I jump to my AccountController it spikes.
Using LightInject:
It loads my AccountController without a spike, however once I try to login to the application it does nothing and just hangs. As if its blocked, the console says User Authenticated and never continues..
It has to be the way I am registering something in DI? I know the below proabably wont help much without the entire context of code. But maybe you see something since I'm pulling at straws.
Images below also.
My account controller injects:
private readonly ABSignInManager _signInManager;
private readonly ABUserManager _userManager;
private readonly ABRoleManager _roleManager;
private readonly IHostLogger _hostLogger;
private readonly IHostingEnvironment _hostingEnvironment;
private readonly IUrlPrefix _urlPrefix;
private readonly IDistributedCache _distributedCache;And my startup registers:
container.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
container.AddScoped<ABHostContext>();
container.AddScoped<HostLogContext>();
container.AddScoped<HostEntityLockContext>();
container.AddScoped<DealerContext>();
container.AddScoped<DealerLogContext>(); //should result in a new context shared for the dealer log
container.AddScoped<DealerEntityLockContext>(); //should result in a new context shared for entity locks
//repository overrides
container.AddScoped<IHostRepository<Log>, HostLogRepository<Log>>();
container.AddScoped<IRepositoryAsync<Domain.Models.Dealer.Log>, DealerLogRepository<Domain.Models.Dealer.Log>>();
container.AddScoped<IRepositoryAsync<aEntityLock>, DealerEntityLockRepository<aEntityLock>>();
//Loggers
container.AddScoped<ISerilogFactory, SerilogFactory>();
container.AddScoped<IHostLogger, HostLogger>();
container.AddScoped<IHostLogService, BusinessLogic.HostServices.LogService>();
container.AddScoped<Serilog.ILogger, DealerLogger>();
container.AddScoped<IDealerLogService, BusinessLogic.DealerServices.LogService>();
container.AddScoped<IUrlPrefix, UrlPrefix>();
container.AddScoped<IDealerDocumentFolder, DealerDocumentFolder>();
container.AddScoped<IDealerEncryptionKey, DealerEncryptionKey>();
container.AddScoped<IDealerInfo, DealerInfo>();
container.AddScoped<IClaimsGenerator, ClaimsGenerator>();
//host services
container.AddScoped((provider) => Postal.Email.CreateEmailService(provider));
//Misc
container.AddScoped<IUtilityWrapper, UtilityWrapper>();
container.AddScoped<IHttpSessionManager, HttpSessionManager>();
container.AddScoped<IHttpHostContextWrapper, HttpHostContextWrapper>();
container.AddScoped<IHostConnectionStringService, HostConnectionStringService>();
container.AddScoped<IDealerConnectionStringService, DealerConnectionStringService>();
//container.AddSingleton(typeof(RedisHubLifetimeManager<>), typeof(RedisHubLifetimeManager<>));
//services.AddSingleton(typeof(HubLifetimeManager<>), typeof(RedisPresenceHublifetimeManager<>));
//services.AddSingleton(typeof(IUserTracker<>), typeof(RedisUserTracker<>));
container.AddScoped<ABSession>();
//add a forwarding registration that will allow us to request ISedonaOneSession and get the same object.
//We can slowly replace references to ABSession with ISedonaOneSession, once all direct references to
//ABSession are replaced we can provide a separate implementation for mobile that doesn't rely on HttpContext
container.AddScoped<ISedonaOneSession>(x => x.GetService<ABSession>());
container.AddScoped<IMessageBus, MessageBus>();
//Register Generic Repositories
container.AddScoped(typeof(IHostRepository<>), typeof(HostRepository<>));
//container.AddTransient(typeof(ITempDataRepository<>), typeof(TempDataRepository<>));
container.AddScoped(typeof(IRepositoryAsync<>), typeof(DealerRepository<>));
//Register Generic Services
container.AddScoped(typeof(IHostService<>), typeof(HostService<>));
container.AddScoped(typeof(IService<>), typeof(DealerService<>));
//Register Generic Manager
container.AddScoped(typeof(IValidator<>), typeof(SimpleValidator<>));
container.AddScoped(typeof(IConverter<,>), typeof(AutoConverter<,>));
container.AddScoped(typeof(IManager<,>), typeof(Manager<,>));
// Refresh token services
container.AddScoped<IRefreshTokenService, OpenIddictService>();`
Garbage collections is in like an infinite loop it looks like:
@davidfowl SimpleInjector works, keeps my memory around 1gb. Its crazy that there can be so many differences between containers. I also need to refactor my classes to use less dependencies and better follow SRP. I think that is the root cause. I have some classes with 10-15 injections.
@pakrym is there any further action for us to take here?
@Eilon there still no isolated repro or memory dump so not a lot of actionable items for now.
Ok we'll give it a few more days and close if we can't repro.
I can give someone a memory dump, just need to know how and where to upload a 5-6 GB file. On a side note, I finally ended up using the autofac extension with 3 lines of code and it works perfect. My app now runs at 600-700 MB, so there is something going on in Microsoft's dependency injection with large dependency graphs. I just don't know how to give you a repro without my project.
@papaytl185 You can use onedrive/dropbox and share a link via email
@pakrym Sounds good, I will start uploading the memory dump to my one drive account tonight. Once it finishes I'll send you an email with the link. Hopefully it helps, I grabbed the dump right when it started skyrocketing and snagged it at about 4.1 GB.
@papaytl185 any luck with memory dump?
@pakrym I can get you a memory dump. I am not a normal use case however. (Im in the slow process of converting a non dotnet core proj to dotnet core. Step one is staying on MVC 5 but swapping out Unity for DependancyInjection which is where I'm at) I'm having the similar symptoms but it only happens every other request.
Just like @alohaninja mentioned in aspnet/Mvc#6410 if i comment out the magical Interlocked.Increment block (https://github.com/aspnet/DependencyInjection/blob/release/2.0.0/src/Microsoft.Extensions.DependencyInjection/ServiceProvider.cs#L92) it works fine.
Also I've tried recreating simpler project that reproduces it to no avail.
@Zoxive thank you for the memory dump. From the quick look it indeed shows very large System.Linq.Expressions.Expression<System.Func<Microsoft.Extensions.DependencyInjection.ServiceProvider, System.Object>> with a lot of nesting. I'll investigate further.
@pakrym let me know if you need anything else.
This issue was moved to dotnet/aspnetcore#2737