Unable to parse status response
soywod opened this issue · comments
Some of himalaya users reported me this error while trying to display their messages. The error occurs there:
let fetches = self
.sess()?
.fetch(&range, "(ENVELOPE FLAGS INTERNALDATE)")
.context(format!(r#"cannot fetch messages within range "{}""#, range))?;It looks like the response from the IMAP server cannot be parsed properly. The problem is that:
- I cannot reproduce their error (it looks like it fails in a specific context)
- I cannot see which response fails because it is not included in the final error message:
Line 229 in 6808dfe
Any idea?
Hmm, interesting. We should probably include the invalid content in the error. In the meantime I would offer your users a way to enable imap's debug mode like here:
Line 53 in d8d69a3
Basically, just set .debug = true on your Session. That should print the underlying traffic to stdout, which can be used to debug further!
Thanks for the tip, I will activate them and let you know if I have more information!
Someone was able to reproduce the issue with debug enabled, it looks like the mailbox cannot be found:
C: a2 SELECT "INBOX"
S: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Forwarded $MailFlagBit0 $NotJunk $NotPhishing $Phishing $has_cal LoadRemoteImages Mail Networking NotJunk OIB-Seen- OIB-Seen-INBOX OIB-Seen-OIB/Business OIB-Seen-OIB/Entertainment OIB-Seen-OIB/Finance OIB-Seen-OIB/Groups OIB-Seen-OIB/Home OIB-Seen-OIB/Jobs OIB-Seen-OIB/Shopping OIB-Seen-OIB/Social OIB-Seen-OIB/Social Networking OIB-Seen-[Gmail]/All Mail OIB-Seen-[Gmail]/Important OIB-Seen-[Gmail]/Spam OIB-Seen-[Gmail]/Trash Old receipt-handled)
Error: cannot select mailbox INBOX
But when listing mailboxes we can see the INBOX one:
C: a2 LIST "" *
S: * LIST (\HasNoChildren) "/" "Archive"
S: * LIST (\HasNoChildren) "/" "INBOX"
S: * LIST (\HasChildren \Noselect) "/" "[Gmail]"
S: * LIST (\Drafts \HasNoChildren) "/" "[Gmail]/Drafts"
S: * LIST (\HasNoChildren \Important) "/" "[Gmail]/Important"
S: * LIST (\HasNoChildren \Sent) "/" "[Gmail]/Sent Mail"
S: * LIST (\HasNoChildren \Junk) "/" "[Gmail]/Spam"
S: * LIST (\Flagged \HasNoChildren) "/" "[Gmail]/Starred"
S: * LIST (\HasNoChildren \Trash) "/" "[Gmail]/Trash"
S: a2 OK Success
Any idea?
Hmm, that's interesting — do they not get an a2 OK in that first response? That makes it seem as though the server just sends the flags but doesn't confirm the operation. If there is an OK hiding there, then I wonder if we fail to parse the FLAGS line somehow. Could you try adding that whole line into a unit test and see that it parses without error (maybe even just directly parse it with imap_proto)?
Sorry for the delay! I added this test:
#[test]
fn parse_bad_mailbox() {
let lines = b"* FLAGS (\\Answered \\Flagged \\Draft \\Deleted \\Seen $Forwarded $MailFlagBit0 $NotJunk $NotPhishing $Phishing $has_cal LoadRemoteImages Mail Networking NotJunk OIB-Seen- OIB-Seen-INBOX OIB-Seen-OIB/Business OIB-Seen-OIB/Entertainment OIB-Seen-OIB/Finance OIB-Seen-OIB/Groups OIB-Seen-OIB/Home OIB-Seen-OIB/Jobs OIB-Seen-OIB/Shopping OIB-Seen-OIB/Social OIB-Seen-OIB/Social Networking OIB-Seen-[Gmail]/All Mail OIB-Seen-[Gmail]/Important OIB-Seen-[Gmail]/Spam OIB-Seen-[Gmail]/Trash Old receipt-handled)\r\n";
let (mut send, recv) = mpsc::channel();
parse_mailbox(lines, &mut send).unwrap();
assert!(recv.try_recv().is_err());
}And I got this error:
// thread 'parse::tests::parse_bad_mailbox' panicked at 'called `Result::unwrap()` on an `Err` value: Parse(Invalid([42, 32, 70, …]))', src/parse.rs:571:41When I log the sub error I have this:
// err: Error(Error { input: [42, 32, 70, …], code: TakeWhile1 })After narrowing the string, it looks like the issue comes from the brackets OIB-Seen-[Gmail] 🤔
According to the RFC:
flag = "\Answered" / "\Flagged" / "\Deleted" /
"\Seen" / "\Draft" / flag-keyword / flag-extension
; Does not include "\Recent"
A flag-keyword is:
flag-keyword = "$MDNSent" / "$Forwarded" / "$Junk" /
"$NotJunk" / "$Phishing" / atom
And an atom is:
atom = 1*ATOM-CHAR
ATOM-CHAR = <any CHAR except atom-specials>
atom-specials = "(" / ")" / "{" / SP / CTL / list-wildcards /
quoted-specials / resp-specials
resp-specials = "]"
So the error is totally normal, a closing bracket is not a valid char for a custom Flag…
Knowing that, which strategy would you advise to deal with this kind of situation? This error seems to come often (see https://github.com/soywod/himalaya/issues/269). Would it make sense to add an option to be "less strict" on the parsing?
I'm curious what @djc (the maintainer of imap-proto) has to say here.
Seems to be a duplicate of djc/tokio-imap#36, see also things like https://bugs.python.org/issue21815 and jstedfast/MailKit#193. At this point probably make sense to just allow [ and ] in flags, I'd open to taking a patch for this in imap-proto. Would be really nice to figure out where this OIB-Seen crap comes from...
I have no idea how to write a clean solution 🙁 I will propose a PR so we can discuss on!
The PR has been merged, let me know when a new release is available 🙂
imap-proto 0.16.2 is on crates.io now.
@jonhoo when could you possibly release a new version? No pressure of course, it is just to know when I will be able to release mine 🙂
@soywod I'm not sure what you mean? Given that all that's needed here is the newer version of imap-proto, and they did a point release, you should be able to get the fix just by doing cargo update. No new version of imap should be necessary.
@jonhoo I need a new version because I am using rust-imap, not imap-proto directly.
I mean, if I do not mistake, rust-imap needs to update imap-proto version so the fix can be accessible, right?
Nope, that shouldn't be necessary since imap specifies imap-proto = "0.16.1", which is compatible with 0.16.2 (which contains the fix).