overlapping recipients when sending by scheduler task

[mangopulp]

I recognized, that some recipients get a newsletter multiple times when its send by scheduler task. This belongs a bit to my previews issue "long sending time when big recipientlist". When the task runs and took the first group from the recipientlist, it takes a while until each newsletter for each recipient is prepared for sending. Espacially when the options "injectLinksSpy" and "injectOpenSpy" are used. So we got the situation that the next task starts and not all mails from the first newsletter are send. So the second task again selected mails which already processed from the first task. This leeds to duplicated and more sending of a newsletter to the same recipient.

As I could find out, in the class Ecodev\Newsletter\Tools in the method "runAllSpool" the SQL-query checks for end_time > ' . (time() - 15). But the end_time isn't necessarily be set at this point, because the end_time is set in the method runSpool() of the same Class after $mailers[$language]->send($email); is called.

So I think it would be better to check for the begin_time or maybe for both. With checking the begin_time the script could recognize that the mail is already in process. Or when checking both it's possible to define a time how much sending-time is allowed as reference to decide that sending failed and should be stopped and restarted again. In this case it makes sense to add a table-counter-field. When sending failed a third time the Mails get's marked as problematic and the user could check the related recipient settings.

I hope I could give some helpful feedback for further improve this very nice extension! If you need more information or help please let me know!

[Adrien Crivelli]

Thank you for reporting. Thanks to your information I was able to patch it, like you suggested. I will be available in next release.

On a side note, if you used paragraphs in your texts, to split different thoughts, they would be much easier to read.

[mangopulp]

The solution doesn't work. Checking for the "begin_time" is not enough and leads to the same behavior as checking for the "end_time". Setting the "begin_time" has to be done before going into the send-loop.

foreach ($allUids as $uids) {

	...

	// Mark it as started sending
	$email->setBeginTime(new DateTime()); // <- It's to late set begin_time here
	$emailRepository->update($email);

	// Send the email
	$mailers[$language]->send($email);

	// Mark it as sent already
	$email->setEndTime(new DateTime());
	$emailRepository->update($email);

	...

}

Solution 1
The "begin_time" has to be set directly after selecting the pairs of newsletter-email UIDs. In the "NewsletterRepositiory" in "findAllNewsletterAndEmailUidToSend" the newsletter-email-pairs get selected. After the selection you loop over the result and copy them to the array "$result". In this loop it could be a good place to set the "begin_time".

// Find the uid of emails and newsletters that need to be sent
$rs = $db->sql_query('SELECT tx_newsletter_domain_model_newsletter.uid AS newsletter, tx_newsletter_domain_model_email.uid AS email
				FROM tx_newsletter_domain_model_email
				INNER JOIN tx_newsletter_domain_model_newsletter ON (tx_newsletter_domain_model_email.newsletter = tx_newsletter_domain_model_newsletter.uid)
				WHERE tx_newsletter_domain_model_email.begin_time = 0
				' . $newsletterUid . '
				ORDER BY tx_newsletter_domain_model_email.newsletter ' . $limit);

$result = [];
while ($record = $db->sql_fetch_assoc($rs)) {
	// I think setting "begin_time" at this point should do the trick
	$result[] = $record;
}

Solution 2
Or back in method "runSpool" doing a previous loop over "$allUids" before going into the send-loop

$allUids = $newsletterRepository->findAllNewsletterAndEmailUidToSend($limitNewsletter);

// do a prev loop to just mark it as started sending
foreach ($allUids as $uids) {
	$emailUid = $uids['email'];
	$email = $emailRepository->findByUid($emailUid);
	// Mark it as started sending
	$email->setBeginTime(new DateTime());
	$emailRepository->update($email);	
}

$oldNewsletterUid = null;
foreach ($allUids as $uids) {

	...

	// Mark it as started sending
	// $email->setBeginTime(new DateTime()); // <- remove at this point in send-loop
	// $emailRepository->update($email); // <- remove at this point in send-loop

	// Send the email
	$mailers[$language]->send($email);

	// Mark it as sent already
	$email->setEndTime(new DateTime());
	$emailRepository->update($email);

	...

}

I think Solution 2 is the better one in case of code-structure. What do you think?

[mangopulp]

I have send a newsletter with Solution 2 and it worked perfect.