Caching mime attachments
DavidKelleySCSC opened this issue · comments
This is "how to best do it" question.
We will be running in a separate thread (async) and sending periodic reports to 200~500 users that contains PNG files as mime attachments. The same image (file) may get sent to multiple recipients along with recipient-unique content. But at any given time, one or more of these might have issues with mail delivery to the remote SMTP machine. So we have to make some provisions to re-try failed messages and to keep the shared PNG file around until this is accomplished or declared a failure (several hours?). We envision a Qlist of pending messages that is processed by the thread. [Keeping the file as an inline byte array copy for each message would consume a ton of memory, a concern]
I presume we cannot 'share' a single mime attachment instance, if that right? If so we end up with a unique message structure for each recipient, each with one of more mime attachments, including the one pointing to the shared PNG file. And once these are sent, the program can delete all but the PNG itself. We will have to come up with some sort of reference counting system, or perhaps loop over the pending messages and examining the QList< MimePart* > for a match, to determine when the finial message is away and the PNG file can safely be deleted. If we could share the mime attachment instance, I presume we could then search it it being used. And we will have to remove the file delete call in the mimeFile distructor.
[Aside, a few documentation words on how the memory clean up process works after the message has been sent would be helpful.]
I presume we will have to use mimeFile.cpp and not mimeinline.cpp due to memory issues, but is that also a correct presumption for this use? Or is the in-line variant just intended for use where a pointer to the data is at hand? But I remain confused as it look like mimeFile.cpp is just reading the file content into a QByteArray when used (hence the memory concern).
Any architectural suggestions on how to approach this?
The inline variant is for inlining, ie show an image on the body of the email.
Now with MimeFile you can passa a QByteArray, this will create a QBuffer and set the data there, now I'd need to see what QBuffer does to the data, if it simply asigns then as QByteArray is already ref counted this would be the best option. In any case since you are sending the same data to many emails the best imo is to cache it as QByteArray anyway.
Now I understand inline, thanks. That's a very good point that QByteArray is already ref counted. While it mean memory bloat for a time, it cleans up very nicely. I see from further code review the moment the prepare call is made, the memory is used anyway, so keeping the source file around is not going to work as I first thought.
As far as I can tell the mime part structure does not seem to be changed by what other part are within the same message. So to repeat a prior question: I presume we cannot 'share' a single mime attachment instance, if that right? If we could, and that is a QObject so it also has a ref count as well, the reuse could be cleaner.
QObjects aren't ref counted, and the mime attachments get deleted once the mail is sent iirc
We ended up with a bunch of message objects held in memory and then purge then once sent or too many tries on a slow loop, the entire mail process runs on a low priority thread of its own. Each of these messages, once ready to send, has its own copy of the contents (files are brought into memory by that time). So my original thought/hope to manga a shared file was misguided.