[API Proposal]: rate limited logging
verdie-g opened this issue · comments
Background and motivation
When developing a web service, logs are usually added for unhappy path, for example, a database query failed because the remote server was unreachable. The code would look like this:
try
{
QueryDatabase();
}
catch (Exception e)
{
Logger.LogError(e, "Error querying the database");
}
One subtle issue with that code that could have very nasty consequences is that if the database is down, all queries will induce an error log. That would create a huge spikes of logs that could overload the logs collector and make it drop logs that were crucial to understand the root cause of the issue.
Because this issue is easy to miss, I think would be great to integrate a token bucket rate limiter in the LoggerMessageAttribute.
dotnet/runtime#82465 seems to be related.
API Proposal
namespace Microsoft.Extensions.Logging;
public class LoggerMessageAttribute
{
/// <summary>
/// Maximum number of logs to restore each replenishment.
/// </summary>
public int LogsPerPeriod { get; set; }
/// <summary>
/// Period between replenishments in milliseconds.
/// </summary>
public int ReplenishmentPeriod { get; set; }
}
If LogsPerPeriod
or ReplenishmentPeriod
are equal or less than 0 they should be both ignored.
API Usage
try
{
QueryDatabase();
}
catch (DatabaseException e)
{
// A maximum of 5 logs will be sent by second.
LogDatabaseError(e, e.HostName);
}
[LoggerMessage(
EventId = 0,
Level = LogLevel.Error,
Message = "Error querying the database {hostName}"),
LogsPerPeriod = 5,
ReplenishmentPeriod = 1000]
public static partial void LogDatabaseError(ILogger logger, Exception e, string hostName);
Alternative Designs
No response
Risks
No response
Tagging subscribers to this area: @dotnet/area-extensions-logging
See info in area-owners.md if you want to be subscribed.
Issue Details
Background and motivation
When developing a web service, logs are usually added for unhappy path, for example, a database query failed because the remote server was unreachable. The code would look like this:
try
{
QueryDatabase();
}
catch (Exception e)
{
Logger.LogError(e, "Error querying the database");
}
One subtle issue with that code that could have very nasty consequences is that if the database is down, all queries will induce an error log. That would create a huge spikes of logs that could overload the logs collector and make it drop logs that were crucial to understand the root cause of the issue.
Because this issue is easy to miss, I think would be great to integrate a token bucket rate limiter in the LoggerMessageAttribute.
dotnet/runtime#82465 seems to be related.
API Proposal
namespace Microsoft.Extensions.Logging;
public class LoggerMessageAttribute
{
/// <summary>
/// Maximum number of logs to restore each replenishment.
/// </summary>
public int LogsPerPeriod { get; set; }
/// <summary>
/// Period between replenishments in milliseconds.
/// </summary>
public int ReplenishmentPeriod { get; set; }
}
If LogsPerPeriod
or ReplenishmentPeriod
are equal or less than 0 they should be both ignored.
API Usage
try
{
QueryDatabase();
}
catch (DatabaseException e)
{
// A maximum of 5 logs will be sent by second.
LogDatabaseError(e, e.HostName);
}
[LoggerMessage(
EventId = 0,
Level = LogLevel.Error,
Message = "Error querying the database {hostName}"),
LogsPerPeriod = 5,
ReplenishmentPeriod = 1000]
public static partial void LogDatabaseError(ILogger logger, Exception e, string hostName);
Alternative Designs
No response
Risks
No response
Author: | verdie-g |
---|---|
Assignees: | - |
Labels: |
|
Milestone: | - |
We are planning to augment the logging system with something that achieves the same goal, but in more flexible framework. In particular, we'll have config-time control over the sampling rate, there will be global vs. per-request sampling, and a few more bells and whistles. I'm hoping to have a proposal published on this front in the coming weeks.
Thanks @geeknoid. I'll transfer this issue to the extensions repo for tracking.