[Catalyst] Out of Memory - File delivery issue

[Craig Chant]

Thanks for the replies.

I'm not sure how big the CSV actually is, but yes I am building a CSV string in memory and trying to print (stream) it to the browser.

It is the devel PSGI server which seems to be baulking with 'Out of memory' and then restarting.

The current production app I am refactoring from legacy perl script to Catalyst doesn't have this problem.

I don't use static::simple on the Catalyst system for the production server as that is IIS7 and handles static files outside of Catalyst.

However, not all apps use windows authentication , some use cookies and built in 'are they logged in' mechanism, static files are not an option in this situation as it would mean writing the file to disk to a publicly accessible area of the webserver, and these files should only be available to logged in members.

A legacy app that did provide private files via a public URL used to keep track of all files generated in a DB table for each member and used this list to delete all adhoc files on login/out mechanism, but I find this clunky and so many apps were re-factored to stream the file from a private area direct to the browser, which I thought was a cleaner way of doing it.

It's also worth noting, many of these outputs are adhoc, on the fly generated, there is no file on disk to open, nor do we want to create a file, I want to generate output on-request and return it to the browser as a streaming file,  negating the need to create unwanted files, that will then need some clean up mechanism.

Gleaning from the Neil's reply I get the feeling if I take Catalyst out of the equation for delivering the response (file), this will solve the problem.

However, how do I have a controller that uses the default 'render_view' for some paths and not for others?

I assume it is a breach of MVC to have the model deliver the resulting CSV does that include the controller?

Is it possible to have the controller print direct to browser...

So instead of what I currently have...

----------------------------------
# run export
my ($result,$xls) = $c->model('NBCSXLSExport')->csv_export;

# check for ok status
    if($result->ok)
    {
        $c->response->header(
            Content_Type =>  'application/vnd.ms-excel',
            Content_Disposition => 'attachment;filename=NBCS_Export.csv'
            );
        $c->response->body($xls);
    }
    else
    {
        $c->response->body( $result->message );
    }

    $c->response->status(200);

----------------------------------------------
I replaced it with...
----------------------------------

    # run export
    my ($result,$xls) = $c->model('NBCSXLSExport')->csv_export;

    # check for ok status
    if($result->ok)
    {
        print "Content-Disposition: attachment;filename=NBCS_Export.csv\n";
        print "Content-Type: application/vnd.ms-excel\n";
        print "\n";
        print $xls;
    }
    else
    {
        $c->response->body( $result->message );
        $c->response->status(200);
    }
----------------------------------------------

However, the browser doesn't get delivered the file, the XLS content is displayed in the devel console, but nothing is outputted to the browser?

Plus as I have overridden the default 'RenderView' , how do I call it for those methods in the controller that  want to use the 'RenderView' templating mechanism?

All help in understanding how I get this CSV string outputted to the browser as a streaming XLS compatible file is appreciated.

Regards,

Craig.


-----Original Message-----
From: Lukas Thiemeier [mailto:spamcatcher at thiemeier.net]
Sent: 02 May 2013 10:33
To: catalyst at lists.scsys.co.uk
Subject: Re: [Catalyst] Out of Memory - File delivery issue

On 05/02/2013 09:54 AM, Craig Chant wrote:
> Hi,
>
>
>
> I understand that Catalyst has a known issue with delivering
> authenticated files via the response mechanism.
>
>
>
> What does the Catalyst community do to work around this problem?
>
>
>
> If I want to read a large file from disk or collate a large CSV / XML
> file and deliver it direct to the browser, how do I do this?
>

Hi Craig,

I never had problems delivering large files, but I don't know what you consider "large".

About delivering large static files: I would recommend to use the web server to serve static files, no matter if they are large or not. You can set up authentication in your web server and use Catalyst::Authentication::Credential::Remote to use the webservers authentication in your Catalyst app.

Another possibility is to use Static::Simple to deliver static files. I successfully delivered files > 4gb using Static::Simple. Is this large enough for you?

AFAIK, Static::Simple has no build in support for authentication, but you could add a "around" modifier in YourApp.pm and do the authentication stuff before calling $c->$orig. You will have to take a look at the code in Static::Simple to find out what method to modify. I am aware that this is NOT a optimal solution. In fact, I think that this approach is very hacky and dirty. But without trying it: I am (almost) sure that it will work.

The topic "creating large XML Files" has come up on this mailing list before. This thread might be helpful:

http://www.gossamer-threads.com/lists/catalyst/users/30931

If you are trying to create a XML File in Memory which is larger than your memory, you do have a problem. But that is not Catalyst specific at all. Try writing the XML Data to your hard disk, and deliver the resulting file.

I guess someone with more knowledge about the Cat internals or more experience with processing large files can give you a better advice. But I still hope this info is useful.

Cheers, Lukas

_______________________________________________
List: Catalyst at lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
This Email and any attachments contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority.