[Catalyst] Sending 3000 emails.

[Andrew]

Your talk of an email alias has intrigued me, =).
I'll try and read up on them.

I managed to solve the problem of the Catalyst app crashing, using some of the advice given.
- To queue the emails in the background,
- To try not to waste memory.

I stopped trying to be all modern perl, and gave up using Email::Stuffer.
Instead, I went back to oldskool perl, and used:

my $mailprog= 'usr/lib/sendmail -i -t -odq -f myreturn at emailaddress.com';
open(MAIL,"|$mailprog");
print MAIL $email;
close MAIL;

Using the "-odq" flag, adds it to the exim4 mail queue without waiting for it to send.

When I was using Email::Stuffer in a loop, I gather it was creating an object...so it could have been creating 3000 objects - maybe that caused memory issues, which could have been responsible for the crash.

So this time, I had my email source code as a text file, that was just read in once at the start of the script,
and the only thing left to do in the foreach loop (now iterating over a list of userids to email), was to add in the TO line - with the name and email address.
Rather than do a find and replace (via pattern matching) of the email source 3000 times to substitute in a TO address, I merely concatenated the TO line onto the rest of the email source, during each iteration... in the hope it would reduce processing.

my $toline='To: "'.$id_fullname{$userid}.'" <'.$id_email{userid}.'>';
my $email=$toline."\n".$emailsource;

So what was the result?

The result was - all emails were successfully queued.
Because my last attempt had sent the first 1100 or so emails, I was only sending around 1900 emails this time.
The Catalyst app did not crash - the plackup process stayed up.
However - the Catalyst app was unable to give any response to a web browser until all messages had been queued - which took about 8 minutes to be fair (going by the time given in the dates in the mail queue - where the first email has a time of 3:31pm and the last 3:39pm).
Even other subroutines, mapped to different urls, would not give a response to a web browser, until the exim mail queue had been fully populated.

It would seem then, although I succeeded in not crashing the server or Catalyst App,
the Catalyst App doesn't seem to want to give responses to web browsers while it's doing this stuff.
I was hoping merely queuing emails instead of sending them, meant there was nothing much to wait on.
It seems I will indeed need a separate script in the background to send these things, leaving the main Catalyst App free to handle all the web browser requests.





----- Original Message ----- 
From: "Luca Ferrari" <fluca1978 at infinito.it>
To: "The elegant MVC web framework" <catalyst at lists.scsys.co.uk>
Sent: Friday, December 11, 2015 11:12 AM
Subject: Re: [Catalyst] Sending 3000 emails.


On Thu, Nov 26, 2015 at 4:32 PM, Andrew <catalystgroup at unitedgames.co.uk> wrote:
> ....I just want to know why the whole FastCGI Plack loaded Catalyst app came
> down half way through,
> and how I can have a Perl script process sending 3000 emails without that
> happening every time, =S.
>

As pointed out already the main problem here is that you are trying to
do stuff (sending email) within the app thread, that is you are
breaking the app design and should use a batch/background thread to do
intensive work (even reading all the emails from database could be a
problem, depending on how you do the query and how optimized is the
database).
I would pick up at least an email alias, that can be easibly
auto-generated and updated each time a new user joins. This way you
are at least hiding the 3000+ receivers behind a single email, letting
the email software to do the stuff it has been built for.
The next step would be to use a mailing list, internal or external.
The only reason I could see for looping thru every email address could
be to store in the database the result of the sending process, which
is in my opinion worthless.

Luca

_______________________________________________
List: Catalyst at lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scsys.co.uk/pipermail/catalyst/attachments/20151211/dd503bea/attachment.htm>