v2: FETCH can produce empty/imcomplete message (imapclient.FetchMessageBuffer)

[Vladimir]

Hello!

Giving the wollowing line from responce to FETCH:

* 85648 FETCH (UID 369004 BODY ((("text" "plain" ("charset" "koi8-r") NIL NIL "quoted-printable" 5716 0)("text" "html" ("charset" "koi8-r") NIL NIL "quoted-printable" 6181 0) "alternative")("message" "delivery-status" () NIL NIL "7bit" 340)("message" "rfc822" () NIL NIL "7bit" 23925) "report") ENVELOPE ("Thu, 02 Nov 2023 13:28:10 +0000" "=?utf-8?B?...?=" (("" NIL "foo" "example.org")) NIL NIL (("" NIL "bar" "example.net")) NIL NIL "<[e-mail]>" "<[e-mail]>") BODY[] {40909}

the imapclient.FetchCommand refuses to parse this line in the middle, silently producing imcomplete imapclient.FetchMessageBuffer with a data, fulfilled before the error occured.

And the error is returned by cmd.Close() is:

FETCH command Close error: in response-data: in body-type-mpart: in body-type-1part: imapwire: expected SP, got ")" / IMAP imap.mail.ru

Not sure exactly, but I suppose the error caused by empty list () in:

...("message" "delivery-status" () NIL NIL "7bit" 340)...

This case happens with different mail vendors.

v2.0.0-alpha.7

cmd := client.UIDFetch(wantedUids, &imap.FetchOptions{
	UID:           true,
	Envelope:      true,
	BodyStructure: &imap.FetchItemBodyStructure{},
	BodySection: []*imap.FetchItemBodySection{{
		Peek: true,
	}},
})
defer func() {
	if err := cmd.Close(); err != nil {
		log.Printf("FETCH command Close error: %v", err)
	}
}()
for msg := cmd.Next(); msg != nil; msg = cmd.Next() {
	buf, err := msg.Collect()
	if err != nil {
		return errors.Wrap(err, "msg.Collect()")
	}
	//...
}

[Simon Ser]

BODY re-formatted for readability:

(
  (
    ("text" "plain" ("charset" "koi8-r") NIL NIL "quoted-printable" 5716 0)
    ("text" "html" ("charset" "koi8-r") NIL NIL "quoted-printable" 6181 0)
    "alternative"
  )
  ("message" "delivery-status" () NIL NIL "7bit" 340)
  ("message" "rfc822" () NIL NIL "7bit" 23925)
  "report"
)

I think the issue is that the message/rfc822 part does not specify the required extra parameters, stops at part size. However the RFC requires the extra parameters (see rule body-type-msg).

[Simon Ser]

Can you try #559?

[Vladimir]

It works!

{Children: [{Children: [{Type:          "text",
                         Subtype:       "plain",
                         Params:        {charset: "koi8-r"},
                         ID:            "",
                         Description:   "",
                         Encoding:      "quoted-printable",
                         Size:          5716,
                         MessageRFC822: nil,
                         Text:          {NumLines: 0},
                         Extended:      nil},
                        {Type:          "text",
                         Subtype:       "html",
                         Params:        {charset: "koi8-r"},
                         ID:            "",
                         Description:   "",
                         Encoding:      "quoted-printable",
                         Size:          6181,
                         MessageRFC822: nil,
                         Text:          {NumLines: 0},
                         Extended:      nil}],
             Subtype:  "alternative",
             Extended: nil},
            {Type:          "message",
             Subtype:       "delivery-status",
             Params:        {},
             ID:            "",
             Description:   "",
             Encoding:      "7bit",
             Size:          340,
             MessageRFC822: nil,
             Text:          nil,
             Extended:      nil},
            {Type:          "message",
             Subtype:       "rfc822",
             Params:        {},
             ID:            "",
             Description:   "",
             Encoding:      "7bit",
             Size:          23925,
             MessageRFC822: nil,
             Text:          nil,
             Extended:      nil}],
 Subtype:  "report",
 Extended: nil}

[Damian Poddebniak]

Doing some interop testing... :-)

I believe @Vovan-VE had the right idea.

When I paste ...

* 1 FETCH (BODY ((("text" "plain" ("charset" "koi8-r") NIL NIL "quoted-printable" 5716 0)("text" "html" ("charset" "koi8-r") NIL NIL "quoted-printable" 6181 0) "alternative")("message" "delivery-status" () NIL NIL "7bit" 340)("message" "rfc822" () NIL NIL "7bit" 23925) "report"))

... into imap-codec, it errors out ...

But when I replace () with NIL ...

* 1 FETCH (BODY ((("text" "plain" ("charset" "koi8-r") NIL NIL "quoted-printable" 5716 0)("text" "html" ("charset" "koi8-r") NIL NIL "quoted-printable" 6181 0) "alternative")("message" "delivery-status" NIL NIL NIL "7bit" 340)("message" "rfc822" NIL NIL NIL "7bit" 23925) "report"))

... it works.

I believe this is a defect in mail.ru.

According to IMAP4rev1, we have ...

body-fld-param = "("
                   string SP string
                   *(SP string SP string)
                 ")" / nil

... and () is not valid.

It's interesting to note, that these "empty list" vs "no list (nil)" cases come up frequently. See modern-email/defects#12 (comment).

Let's collect all these things! :D