Q: ImapClient lifetime, am i using mailkit correctly?

Question

Q: ImapClient lifetime, am i using mailkit correctly?

sommmen opened this issue a year ago · comments

Hiya, i've built an app to download attachments from a mailbox for processing with mailkit and i was wondering if i am doing things correctly.

I have the following concrete questions:

I'm using 2 ImapClients, one that idles and one to fetch message. 1 will be used for the app's lifetime (basically days at a time), the other hourly or when new messages are found. I did this because it seems like a client Idling cannot also fetch messages. Is this design correct?
Could i reuse the same ImapClient?
Is it okay to have a very long running instance of ImapClient, or will this take up more and more memory when the hours go by?
IdleAsync has a stoppingToken and a cancellationToken, why are there 2 cancellation tokens? should i use both tokens?

Other than that general feedback or comments around the library's usage are appreciated.

Using .net core 7 (with hangfire for job management).

I have a job that executes every hour.
It looks for emails in the inbox and extracts attachements.
Processed mails are marked as 'seen';

Then i have a .net HostedService that monitors the inbox count email count and starts the mail import job when new mails arrive (Mails are rarely deleted, so i did not bother checking if the count goes up or down).

MailImportJob:

...

var query = _hostEnvironment.IsDevelopment() || _hostEnvironment.IsDevelopmentDatabaseUpdateJob()
	? SearchQuery.DeliveredAfter(DateTime.Now.AddDays(-7).StartOfDay())
	: SearchQuery.NotSeen;

var uuids = await emailClient.Inbox.SearchAsync(query, cancellationToken) ?? new List<UniqueId>();

if (uuids.Any())
{
	Logger.LogInformation("Fetching {UuidCount} messages", uuids.Count);

	// Oldest first
	uuids = uuids
		.Reverse()
		.ToList();

	messages = await emailClient
		.Inbox
		.FetchAsync(uuids,
			MessageSummaryItems.Full
			| MessageSummaryItems.BodyStructure
			| MessageSummaryItems.UniqueId,
			cancellationToken: cancellationToken);
}
else
{
	Logger.LogInformation("No unread messages found.");
	return;
}

...

foreach (var message in messages.WithProgress(messagesProgressBar))
{
	// Check what to do with it, not yet processed

	if (message.Attachments.Any())
	{
		var attachments = message
			.Attachments
			.ToList();

		for (var i = 0; i < attachments.Count; i++)
		{
			var attachment = attachments[i];

			// If the attachment is an embedded mail, download its attachments
			// NOTE this goes only one level deep.
			if (attachment is BodyPartMessage { Body: BodyPartMultipart attachmentMessageBody })
			{
				var bodyPartBasics = attachmentMessageBody
					.BodyParts
					.OfType<BodyPartBasic>()
					.ToList();

				attachments.AddRange(bodyPartBasics);

				continue;
			}

                        ...

                        var part = (MimePart)await emailClient.Inbox.GetBodyPartAsync(message.UniqueId, attachment,
                                                cancellationToken);
                        
                        using var memoryStream = new MemoryStream();
                        await part.Content.DecodeToAsync(memoryStream, cancellationToken);
                        await memoryStream.FlushAsync(cancellationToken);

			...
		}
	}
	else
	{
		Logger.LogWarning("Mail {MailUuid} had no attachments, subject: {Subject}", message.UniqueId.ToString(),
			message.NormalizedSubject);
	}

	// Always mark the mail as seen, even if we've found no relevant attachments.
	if (!_hostEnvironment.IsDevelopment() && !_hostEnvironment.IsDevelopmentDatabaseUpdateJob())
	{
		if(!message.Folder.IsOpen || message.Folder.Access != FolderAccess.ReadWrite)
			await message.Folder.OpenAsync(FolderAccess.ReadWrite, cancellationToken);
		await message.Folder.AddFlagsAsync(message.UniqueId, MessageFlags.Seen, false,
			cancellationToken: cancellationToken);
	}
}

...

Hosted service

/// <summary>
/// A background service that monitors for incoming mails via the Imap protocol
/// Based on https://github.com/jstedfast/MailKit/blob/6c813d02617edc3e3de5481a413b1e2fb43bfe8c/samples/ImapIdle/ImapIdle/Program.cs
/// </summary>
public class MailBackgroundService : BackgroundService
{
    private readonly IImapClientFactory _clientFactory;
    private readonly ILogger<MailBackgroundService> _logger;
    private readonly Action _debounced;

    public MailBackgroundService(IImapClientFactory clientFactory, ILogger<MailBackgroundService> logger, IBackgroundJobClient backgroundJobClient)
    {
        _clientFactory = clientFactory;
        _logger = logger;

        _debounced = Debounce(() =>
        {
            _logger.LogDebug(nameof(MailImportJob) + " scheduled");
            backgroundJobClient.Enqueue<MailImportJob>(x => x.Run(default, default));
        }, 5000);
    }

    #region Overrides of BackgroundService

    /// <inheritdoc />
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            var sleepDurations = new[]
            {
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(5),
                TimeSpan.FromSeconds(30)
            };

            await Policy
                .Handle<Exception>(e => e is not OperationCanceledException)
                .WaitAndRetryAsync(sleepDurations)
                .ExecuteAsync(async () =>
                {
                    using var imapClient = await _clientFactory.GetImapClient(stoppingToken);

                    imapClient.Inbox.CountChanged += InboxOnCountChanged;

                    try
                    {
                        await imapClient.IdleAsync(stoppingToken);
                    }
                    finally
                    {
                        imapClient.Inbox.CountChanged -= InboxOnCountChanged;
                    }
                });
        }
        catch (Exception e) when (e is not OperationCanceledException
                                  && _logger.LogExceptionAndCatch(e,
                                      nameof(MailBackgroundService) + " exiting due to error!"))
        {
            // Swallowed
        }
    }

    private void InboxOnCountChanged(object? sender, EventArgs e)
    {
        _logger.LogDebug(nameof(InboxOnCountChanged) + " fired");
        _debounced();
    }

    /// <summary>
    /// Debounce. Adapted from: https://stackoverflow.com/a/29491927/4122889
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="func"></param>
    /// <param name="milliseconds"></param>
    /// <returns></returns>
    private static Action Debounce(Action func, int milliseconds = 1000)
    {
        CancellationTokenSource? cancelTokenSource = null;

        return () =>
        {
            cancelTokenSource?.Cancel();
            cancelTokenSource = new CancellationTokenSource();

            Task.Delay(milliseconds, cancelTokenSource.Token)
                .ContinueWith(t =>
                {
                    if (t.IsCompletedSuccessfully)
                    {
                        func();
                    }
                }, TaskScheduler.Default);
        };
    }

    #endregion
}

To make this work with DI i implemented a simple factory;

public class ImapClientFactory : IImapClientFactory
{
    private readonly ImapSettings _imapSettings;

    public ImapClientFactory(ImapSettings imapSettings)
    {
        _imapSettings = imapSettings;
    }

    public async Task<IImapClient> GetImapClient(CancellationToken cancellationToken = default)
    {
        var imapClient = new ImapClient();
        imapClient.ServerCertificateValidationCallback = (_, _, _, _) => true;
        imapClient.AuthenticationMechanisms.Remove("XOAUTH2");

        await imapClient.MaybeConnect(_imapSettings.Host, _imapSettings.Port, _imapSettings.UseSsl, _imapSettings.UserName, _imapSettings.Password, cancellationToken);

        return imapClient;
    }
}

...

public static class ImapClientExtensions
{
    public static async Task MaybeConnect(this IImapClient imapClient, string host, int port, bool useSsl,
        string userName, string password, CancellationToken cancellationToken = default)
    {
        if (!imapClient.IsConnected)
            await imapClient.ConnectAsync(host, port, useSsl, cancellationToken);

        if (!imapClient.IsAuthenticated)
            await imapClient.AuthenticateAsync(userName, password, cancellationToken);

        if (!imapClient.Inbox.IsOpen)
            await imapClient.Inbox.OpenAsync(FolderAccess.ReadOnly, cancellationToken);
    }
}

...

public static partial class ServiceCollectionExtensions
{
    public static IServiceCollection AddEmail(this IServiceCollection services)
    {
        services.AddSingleton<IImapClientFactory, ImapClientFactory>();
        
        using var tempProvider = services.BuildServiceProvider();
        var hostEnvironment = tempProvider.GetService<IHostEnvironment>();

        if(hostEnvironment == null || hostEnvironment.IsProduction())
            services.AddHostedService<MailBackgroundService>();

        return services;
    }
}

Jeffrey Stedfast · Answer 1 · Mon Jun 26 2023 21:59:55 GMT+0800 (China Standard Time)

I'm using 2 ImapClients, one that idles and one to fetch message. 1 will be used for the app's lifetime (basically days at a time), the other hourly or when new messages are found. I did this because it seems like a client Idling cannot also fetch messages. Is this design correct?

It is true that a client cannot be used to fetch messages while it is idling, but you can stop idling and then fetch messages and then go back to idling. There's no reason you can't do that. Whether you decide to use 2 clients (1 for idle and 1 for fetching) or whether you switch modes in a single client is up to you. Either is fine.

Is it okay to have a very long running instance of ImapClient, or will this take up more and more memory when the hours go by?

It should not take up more memory the longer it runs. If it does, I would consider that a bug.

IdleAsync has a stoppingToken and a cancellationToken, why are there 2 cancellation tokens? should i use both tokens?

As you've probably noticed with MailKit's API, every method call that makes a network request has a CancellationToken cancellationToken argument. This cancellationToken is what can be used to abort the command at any time, usually by severing the connection (assuming the command is already sent or being sent).

The doneToken "cancels" the IDLE command by sending the DONE command. That's how the IDLE feature in the IMAP protocol works.

In general, you'll want to cancel the doneToken first in all cases.

Jeffrey Stedfast · Answer 2 · Mon Jun 26 2023 22:22:27 GMT+0800 (China Standard Time)

As far as usage goes...

In your factory, you are calling imapClient.AuthenticationMechanisms.Remove("XOAUTH2"); before you connect. That won't have any effect because the Connect/ConnectAsync methods are what populate that property.

It's also not really required anyway because XOAUTH2 isn't attempted when using the client.Authenticate(username, password) API. You have to explicitly request XOAUTH2 authentication by using the Authenticate(SaslMechanism mechanism) API. The reason for this is because XOAUTH2 does not use a password per-se.

imapClient.ServerCertificateValidationCallback = (_, _, _, _) => true;

That's fine for development, but ultimately you'll want something that you can more confidently be sure that a MITM attack won't compromise you (or whoever is running this app).

You could pop up a dialog asking the user to confirm the SSL certificate fingerprint along with the host name, and other X509Certificate details (issuer / serial / common name). Or you could hard-code allowed fingerprints perhaps (or get them from a database/config file).

await imapClient.ConnectAsync(host, port, useSsl, cancellationToken);

I would recommend looking into using the SecoreSocketOptions enum instead of a true/false useSssl value because it gives you more control over what level of security you want.

If your useSsl value is true, then it will be the equivalent of SecureSocketOptions.SslOnConnect. If the value is false, it will be the equivalent of SecureSocketOptions.StartTlsWhenAvailable.

sommmen · Answer 3 · Mon Jun 26 2023 22:38:10 GMT+0800 (China Standard Time)

but you can stop idling and then fetch messages and then go back to idling.

I'd thought of that but that could miss incoming messages maybe (w/o tracking the count). Also writing some synchronisation mechanism to share the client instance and start and stop idling seemed like a lot of work.

It should not take up more memory the longer it runs. If it does, I would consider that a bug.

I'm not tracking memory in production, but a better question would perhaps be if the client was designed to be long lived (singleton lifetime) or short-lived (scoped lifetime). It seems both are okay, a single client would be preferable given it would manage a single socket and not multiple.

The doneToken "cancels" the IDLE command by sending the DONE command. That's how the IDLE feature in the IMAP protocol works.

Right, so if i want to make sure my application shuts down as nice as possible, within a reasonable timeframe i'd do something like:

var cancelCts = new CancellationTokenSource();
stoppingToken.Register(() => cancelCts.CancelAfter(1000)); // Allow for up to a second to cancel idling normally, then cancel the whole ordeal.

await imapClient.IdleAsync(stoppingToken, cancelCts.Token);

As far as usage goes...

This was there before i took a look at things but i'll definitely take a look.

Anyways thanks a bunch for giving your time to do a review for me.