Retrieving a list of all subfolders returns only subscribed subfolders
edgarbro opened this issue · comments
Description:
When discovering a mailbox recursively, only subscribed subfolders are returned regardless of the subscribedOnly parameter of IMailFolder.GetSubfolders([subscribedOnly = false]).
Platform:
- OS: Windows 11
- .NET Runtime: .NET 7
- MailKit Version: 4.2.0
- IMAP-Server: Cyrus (version unknown)
To Reproduce:
Open a folder with at least one unsubscribed subfolder and try to list it:
using ImapClient Client = new(new ProtocolLogger("imap.log"));
Client.Connect("imap.example.com", 993, SecureSocketOptions.SslOnConnect);
Client.Authenticate("username", "password");
var root = Client.GetFolder("root");
root.Open(FolderAccess.ReadOnly);
foreach(var folder in root.GetSubfolders(subscribedOnly: false)) {
_logger.LogDebug("> Subfolder: {name}; Subscribed: {subscribed}", folder.FullName, folder.IsSubscribed);
}
root.Close();Note: The real folder name is changed to "root" to keep the example simple (and for privacy reasons).
Expected behavior:
If using subscribedOnly: true, only subscribed subfolders should be returned.
If using subscribedOnly: false, all subfolders should be returned - both subscribed and unsubscribed.
Protocol Logs:
Logs, when subscribedOnly: false:
C: B00000004 EXAMINE root (CONDSTORE)
S: * ENABLED CONDSTORE
S: * 67 EXISTS
S: * 0 RECENT
S: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
S: * OK [PERMANENTFLAGS ()] Ok
S: * OK [UIDVALIDITY 1670581923] Ok
S: * OK [UIDNEXT 71] Ok
S: * OK [HIGHESTMODSEQ 77] Ok
S: * OK [URLMECH INTERNAL] Ok
S: B00000004 OK [READ-ONLY] Completed
C: B00000005 LIST "" "root/%" RETURN (SUBSCRIBED CHILDREN)
S: * LIST (\Subscribed \HasNoChildren) "/" "root/SubscribedSubfolder"
S: B00000005 OK Completed (0.000 secs)
C: B00000006 UNSELECT
S: B00000006 OK Completed
Logs, when subscribedOnly: true:
C: B00000004 EXAMINE root (CONDSTORE)
S: * ENABLED CONDSTORE
S: * 67 EXISTS
S: * 0 RECENT
S: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
S: * OK [PERMANENTFLAGS ()] Ok
S: * OK [UIDVALIDITY 1670581923] Ok
S: * OK [UIDNEXT 71] Ok
S: * OK [HIGHESTMODSEQ 77] Ok
S: * OK [URLMECH INTERNAL] Ok
S: B00000004 OK [READ-ONLY] Completed
C: B00000005 LIST (SUBSCRIBED) "" "root/%" RETURN (CHILDREN)
S: * LIST (\Subscribed \HasNoChildren) "/" "root/SubscribedSubfolder"
S: B00000005 OK Completed (0.000 secs)
C: B00000006 UNSELECT
S: B00000006 OK Completed
IMAP protocol, when manually connecting via OpenSSL:
# basically the request from above
tag LIST "" "root/%" RETURN (SUBSCRIBED CHILDREN)
* LIST (\Subscribed \HasNoChildren) "/" "root/SubscribedSubfolder"
tag OK Completed (0.000 secs)
# same request, however without "SUBSCRIBED" parameter
tag LIST "" "root/%" RETURN (CHILDREN)
* LIST (\HasNoChildren) "/" root/SubscribedSubfolder
* LIST (\HasNoChildren) "/" root/UnsubscribedSubfolder
tag OK Completed (0.000 secs 2 calls)
Possible fix:
In the implementation of IMailFolder in ImapFolder, in lines 1541 and 1554, the exclamation marks should be removed, since they are causing the unintended SUBSCRIBED keyword getting included in the RETURN statement.
Original code (a bit shortened for simplicity):
async Task<IList<IMailFolder>> GetSubfoldersAsync(StatusItems items, bool subscribedOnly, bool doAsync, CancellationToken cancellationToken) {
// ...
var status = items != StatusItems.None;
var command = new StringBuilder();
var returnsSubscribed = false;
var lsub = subscribedOnly;
if(subscribedOnly) {
if((Engine.Capabilities & ImapCapabilities.ListExtended) != 0) {
command.Append("LIST (SUBSCRIBED)");
returnsSubscribed = true;
lsub = false;
} else {
command.Append("LSUB");
}
} else {
command.Append("LIST");
}
command.Append(" \"\" %S");
if(!lsub) {
if(items != StatusItems.None && (Engine.Capabilities & ImapCapabilities.ListStatus) != 0) {
command.Append(" RETURN (");
if((Engine.Capabilities & ImapCapabilities.ListExtended) != 0) {
if(!subscribedOnly) { // <- Line 1541
command.Append("SUBSCRIBED ");
returnsSubscribed = true;
}
command.Append("CHILDREN ");
}
command.Append("STATUS (");
command.Append(Engine.GetStatusQuery(items));
command.Append("))");
status = false;
} else if((Engine.Capabilities & ImapCapabilities.ListExtended) != 0) {
command.Append(" RETURN (");
if(!subscribedOnly) { // <- Line 1554
command.Append("SUBSCRIBED ");
returnsSubscribed = true;
}
command.Append("CHILDREN");
command.Append(')');
}
}
// ...
}(This is the second time opening an issue in general. I hope that the request makes sense and that I do not violate any best practices in opening issues. Thanks.)
I believe that MailKit is doing the correct thing here...
subscribedOnly: false
B00000005 LIST "" "root/%" RETURN (SUBSCRIBED CHILDREN)subscribedOnly: true
B00000005 LIST (SUBSCRIBED) "" "root/%" RETURN (CHILDREN)The RFC that defines the behavior of the LIST-EXTENDED commands is rfc5258.
This specification introduces 2 additional optional arguments for the LIST command.
The first are considered selection options (section 3.1) which is what the first set of parenthesized options are. For example, in the subscribed: true case, it is requesting LIST (SUBSCRIBED) which would be the equivalent of LSUB.
The second is the RETURN options (section 3.2) which specify which additional flags we want to know the state of:
3.2. Initial List of Return Options
The return options defined in this specification are as follows:
SUBSCRIBED - causes the LIST command to return subscription state
for all matching mailbox names. The "\Subscribed" attribute MUST
be supported and MUST be accurately computed when the SUBSCRIBED
return option is specified. Further, all mailbox flags MUST be
accurately computed (this differs from the behavior of the LSUB
command).
CHILDREN - requests mailbox child information as originally proposed
in [CMbox]. See Section 4, below, for details. This option MUST
be supported by all servers.
MailKit asks for the subscription state because, ideally, ImapFolder.Attributes should try to match the server's state as much as possible. Obviously, when a server does not support LIST-EXTENDED, then these flags will not be fully representative, but ... yea.
The other thing that MailKit does is to use that returnsSubscribed state to decide whether or not to merge the \Subscribed flag from a previously cached LSUB and/or LIST-EXTENDED response (the subscribed state can change between LIST/LSUB requests due to another client).
Based on my understanding of the spec, it looks like Cyrus IMAP is at fault here and is not properly interpreting our command. It SHOULD be returning the full list of subfolders in the subscribedOnly: false case rather than only the subscribed folders.
Would be interesting to see what the Cyrus IMAP devs say and/or whether this issue has already been fixed in newer versions.
Could you submit a bug report to Cyrus and post a link to the bug report here?
In the meantime, maybe also get me the IMAP greeting response (first line in the log) so that maybe I can figure out a way to add a QuirksMode hack.
Cyrus IMAP bugs can be filed here: https://github.com/cyrusimap/cyrus-imapd/issues/
Thank you for the quick and detailed response.
The issue was indeed caused by Cyrus. The used version was 2.4.17, however on version 3.6.1 this behavior doesn't occur:
C: B00000011 LIST "" "INBOX.%" RETURN (SUBSCRIBED CHILDREN)
S: * LIST (\Subscribed \HasNoChildren) "." INBOX.sub
S: * LIST (\HasNoChildren) "." INBOX.unsub
S: B00000011 OK Completed (0.001 secs 2 calls)
Since I have to support the older Cyrus version, I will create a local branch and implement a "fix" there.
From my perspective, this issue can be closed. Again, thanks for your quick response and thanks for maintaining this project. :)
(And sorry for the false alert.)
Kind regards,
Edgar
Is the version string somewhere in Cyrus' greeting when you connect?
For Cyrus 2.4.17, the version number is not in the greeting message included. I asked the server administrator for it.
However, it is included in the newer version:
# Old version (via OpenSSL):
* OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE MUPDATE=mupdate://mail-portal/ AUTH=PLAIN AUTH=LOGIN SASL-IR] mail-srv02 server ready
# New version:
S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] mail.xyz.com Cyrus IMAP 3.6.1-Debian-3.6.1-4 server ready
(Real server names changed.)
ok, thanks - it doesn't look like I'd be able to auto-detect that the old server was Cyrus IMAP v2.4.17.
If I could, I'd hack up a quick patch to add a ImapQuirksMode value for it and make it change behavior if it detected that, but that doesn't look possible.
I guess I'll close this, then.