[Catalyst] Fetching URL Content

[J. Shirley]

On Wed, Sep 2, 2009 at 6:25 PM, Trevor Phillips
<trevor.phillips at gmail.com>wrote:

> I have a requirement to fetch URL content - this is currently mostly
> to do with the View phase, to include chunks of HTML for core branding
> in the templating. ie; Top header, footer, global nav, that sort of
> thing.
>
> Ideally, I'd like to do it directly from the Template Toolkit
> templates, but TT either can't do this, or it's really hard to Google
> for. ^_^
> It would be nice to cache locally the content, even if for a short
> period. Or ideally, to cache permanently unless there's a
> "Cache-Control: no-cache" request header to force an update.
>
> ie; something like:
>   [% INCLUDE "http://central.server.com/branding/header.html" cache=3D"5
> min" %]
>
> Can TT do anything like this? It doesn't need to be parsed by TT
> (although having the option for TT to parse it would be useful).
>
> The next idea would be to have a Controller fetch the content for the
> remote URLs, and either shove them in the stash, or cache them to
> disk, so that TT can then reference them. Would this just be a case of
> doing a usual LWP UserAgent Request, or does Catalyst have built-in
> handling or plugins to aid in this?
>
> Hmmm. It could be implemented as a Model as well, I guess...
>
> Any ideas/tips/best practices?
>
> Thanks.
>
>
Putting it into the template definitely falls under the bad idea category.
Having something like this as a plugin is also a very bad idea.

While you could, very easily, create this behavior by extending View::TT it
is better to put this in a very simple model class.  The caching would be
very easy as well.

You could then just wrap one of the LWP::UserAgent and be done.  I'm
currently doing something similar with Web::Scraper, and it works quite
well.

This is a non-tested/typed up example of what I have (I don't remember
exactly which app has it :/).  This is just an example of a thin model that
can be used to accomplish mostly what you want, not really for you to just
copy and paste in.  You can add in caching as necessary.

package MyApp::Model::Scraper;

use Moose;
use Web::Scraper;

BEGIN { extends "Catalyst::Model"; }

has 'scraper' =3D> {
    is =3D> 'rw',
    isa =3D> 'Web::Scraper',
    default =3D> sub {
        scraper { ... }
    }
};

sub fetch {
    my ( $self, $url ) =3D @_;

    return $self->scraper->scrape( URI->new( $url ) );
}

no Moose;
__PACKAGE__->meta->make_immutable;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20090902/b8e4b=
128/attachment.htm