[Catalyst-commits] r12396 - in
trunk/examples/CatalystAdvent/root/2009: . pen
t0m at dev.catalyst.perl.org
t0m at dev.catalyst.perl.org
Wed Dec 16 00:50:20 GMT 2009
Author: t0m
Date: 2009-12-16 00:50:20 +0000 (Wed, 16 Dec 2009)
New Revision: 12396
Added:
trunk/examples/CatalystAdvent/root/2009/16.pod
Removed:
trunk/examples/CatalystAdvent/root/2009/pen/media_delivery.pod
Log:
Edits
Copied: trunk/examples/CatalystAdvent/root/2009/16.pod (from rev 12395, trunk/examples/CatalystAdvent/root/2009/pen/media_delivery.pod)
===================================================================
--- trunk/examples/CatalystAdvent/root/2009/16.pod (rev 0)
+++ trunk/examples/CatalystAdvent/root/2009/16.pod 2009-12-16 00:50:20 UTC (rev 12396)
@@ -0,0 +1,242 @@
+
+=head1 Delivering static media with Catalyst
+
+=head2 Why?
+
+Why would you want to deliver static media with Catalyst at all?
+
+Generally you use the L<Static::Simple|Catalyst::Plugin::Static::Simple>
+plugin to serve assets (.css and image files) from the C</root/static> directory.
+However for production use you are recommended to configure your webserver
+to directly serve the static content as this is B<much> more efficient.
+
+Using an entire application server process (i.e. a perl process) to serve static media
+is a massive waste of resources, especially if you're not using a front-end proxy
+(as your web server ends up with a heavyweight perl process tied up sending bytes to
+a user rather than doing useful work).
+
+However, there are a number of cases where just unconditionally serving the static content
+to the user is a bad idea - think of protected .pdf documents, time limited URIs,
+single use (or n-use) download URIs, etc.
+
+=head3 What about a cache?
+
+That's a good question - putting a cache (such as varnish) in front of your application
+and letting it handle most of the requests for static content is a great solution.
+
+However, in some cases (specifically one use URIs, when you want to check ACLs and
+when you want to log all accesses to a resource) then it is not possible to perform
+any sort of caching in front of your application as you won't be able to perform the
+checks you need to at that layer.
+
+=head2 How?
+
+As a first implementation, then using the
+L<serve_static_file|Catalyst::Plugin::Static::Simple/serve_static_file>
+method is a good way to start.
+
+If your app is small and for internal company use only, then you quite possibly never need
+to go any further...
+
+However, if you're serving large files and likely to have an appreciable number of users,
+then sending bytes in perl just won't scale.
+
+=head2 Scaling
+
+The key to scaling is to do things using the minimum resources possible. Sending bytes across
+the internet is something that web servers are very good at scaling to do.
+
+So the planned architecture will be something like:
+
+=over
+
+=item *
+
+Request comes into your web server.
+
+=item *
+
+Request is sent to your Catalyst application running as FastCGI.
+
+=item *
+
+Application validates the request, increments any counters necessary
+and issues a response with C<X-Accel-Redirect> or C<X-Sendfile> header set.
+
+=item *
+
+Your web server takes care of serving the content back to the user as
+specified by that header.
+
+=back
+
+So the key is this C<X-Accel-Redirect> or C<X-Sendfile> thing - these are just
+headers that you get your application to send in the response (which one you
+use depends which web server you're running).
+
+After sending this response (headers only, no content), your application server
+is freed up and able to get on with handling the next request.
+
+=head3 A note on web servers
+
+This topic appears to be something that people get very excited about, but for
+the purposes of this discussion - whichever web server you prefer is likely to
+work fine, I'll go through the common choices below.
+
+=head4 Apache
+
+Apache is generally thought of as I<big>, but this is very much to do with what
+modules you have loaded and what MPM you're using. A lightweight Apache is perfectly
+acceptable to be serving content with. C<mod_xsendfile> provides the C<X-Sendfile>
+support.
+
+=head4 Lighttpd
+
+Lighttpd has simpler config to apache if you're writing a 'new' one, but your
+sysadmins are also less likely know their way around administering it.
+Has X-SendFile support as standard.
+
+=head4 nginx
+
+Works differently (and uses a different header) to either of the above.
+L<X-Accel-Redirect> actually doesn't logicially send a file from disk,
+it internally redirects the request to another path known by the web server,
+bypassing ACL checking.
+
+This means that the correct header to send depends on your web server
+configuration, which is a disadvantage, but as nginx can act as a proxy as well
+as a web server, this can also be more powerful, as nginx can proxy for a
+backend content server.
+
+=head4 perlbal
+
+Perl based load balancer which supports an C<X-REPROXY-URL> header to allow
+you to reproxy content to a backend content server.
+
+C<X-REPROXY-URL> is not covered in this article as I don't have any experience
+with it myself, and you still need a web server to serve your main application.
+
+=head2 Example
+
+Here is a worked example is going to show using C<nginx>, to reproxy a C<MogileFS> installation,
+as that's a what I'm using in production to deliver 150Tb of content at high speed :)
+
+In this case I'm using the MogileFS module for nginx (L<see below|/SEE ALSO>) to serve
+the actual content.
+
+It should be noted that C<MogileFS> isn't trivial to install or maintain, but
+if you're prepared to expend the time and expertise then it is an extremely cost
+effective solution to file storage on a large scale.
+
+You can, however, have a B<much> simpler infrastructure serving media from a directory
+on local disk if the amount of content you're delivering isn't huge.
+
+=head3 nginx config
+
+ http {
+ include /etc/nginx/mime.types;
+ access_log /var/log/nginx/access.log;
+
+ sendfile on;
+ keepalive_timeout 65;
+ tcp_nodelay on;
+
+ gzip on;
+
+ server {
+ listen 80;
+ root /opt/FileServer/webroot;
+
+ location / {
+ include /etc/nginx/fastcgi_params;
+ fastcgi_pass unix:/tmp/fileserver.socket;
+ access_log off;
+ }
+
+ location /private/mogile/ {
+ internal;
+ mogilefs_tracker 1.1.1.1:7001;
+ mogilefs_domain mydomain;
+ mogilefs_pass {
+ proxy_pass $mogilefs_path;
+ proxy_hide_header Content-Type;
+ proxy_buffering off;
+ }
+ access_log off;
+ }
+ }
+ }
+
+=head3 Example code snippet for generating timed uris.
+
+This method assumes that it's being called on a file object which stores various pieces
+of metadata about a file.
+
+ method generate_timed_uri (%p) {
+ $p{ttl} ||= 60 * 60 * 24 * 7; # Default to a week
+
+ # Turn integers into strings to save space in the URIs (and make them less obvious)
+ my $id = Math::BaseCalc->new(digits => ['a' .. 'z'])->to_base( $self->id ); # The primary key of the row
+
+ my $secret = $self->secret; # This is a random secret string associated with
+ # (but not related to the contents of) the file object
+
+ # Generate a time rounded off to last expiry point, then 2x forward of the TTL.
+ $time = Math::BaseCalc->new(digits => ['a' .. 'z'])->to_base( (time() % $ttl) + $ttl * 2 );
+
+ # Note this is theoretically cryptographically weak, should really use HMAC.
+ my $hash = Digest::SHA1::sha1_hex($self->secret . '-' . $time);
+
+ return '/file/' . join('/', $hash, $time, $id, $self->filename . '.' . $self->extension);
+ }
+
+=head3 Example code snippet for serving timed uris.
+
+ sub file : Local {
+ # Note we ignore the filename in the URI generated above, it is just there
+ # to make the uri look 'normal'..
+ my ($self, $c, $hash, $time, $id) = @_;
+
+ $id = Math::BaseCalc->new(digits => ['a' .. 'z'])->from_base($id);
+
+ my $file = eval { $schema->resultset('File')->find($id) }
+ or return $self->error_notfound($c);
+
+ return $self->error_notfound($c)
+ unless $hash eq Digest::SHA1::sha1_hex($file->secret . '-' . $time)
+
+ # Time embedded in the URL is the latest time this image is valid.
+ $expire_time = Math::BaseCalc->new(digits => ['a' .. 'z'])->from_base($time);
+ if ($expire_time < time()) {
+ return $self->error_expired($c, $file);
+ }
+ # NOTE: In this case as you know when the URI expires, it is possible
+ # to serve cache headers and have the hot items cached by your
+ # front end proxy. This is meant to be a simple example and so
+ # this is left as an exercise for the reader :)
+ $c->res->header('Content-Type', $file->mimetype);
+ $c->res->header('Accept-Ranges', 'none');
+
+ # Note: this matches the path in the nginx config above.
+ # With X-Sendfile, it would be the filename on disk instead.
+ my $redirect_to = '/private/mogile/' . $file->mogile_id;
+ $c->res->header('X-Accel-Redirect' => $redirect_to);
+ }
+
+=head2 SEE ALSO
+
+L<http://blog.lighttpd.net/articles/2006/07/02/x-sendfile>
+
+L<http://wiki.nginx.org/NginxXSendfile>
+
+L<http://tn123.ath.cx/mod_xsendfile/>
+
+L<http://www.danga.com/perlbal/>
+
+L<http://github.com/vkholodkov/nginx-mogilefs-module>
+
+L<http://www.danga.com/mogilefs/>
+
+=head2 AUTHOR
+
+ Tomas Doran (t0m) <bobtfish at bobtfish.net>
Deleted: trunk/examples/CatalystAdvent/root/2009/pen/media_delivery.pod
===================================================================
--- trunk/examples/CatalystAdvent/root/2009/pen/media_delivery.pod 2009-12-16 00:40:24 UTC (rev 12395)
+++ trunk/examples/CatalystAdvent/root/2009/pen/media_delivery.pod 2009-12-16 00:50:20 UTC (rev 12396)
@@ -1,245 +0,0 @@
-
-=head1 Delivering static media with Catalyst
-
-=head2 Why?
-
-Why would you want to deliver static media with Catalyst at all?
-
-Generally you use the L<Static::Simple|Catalyst::Plugin::Static::Simple>
-plugin to serve assets (.css and image files) from the C</root/static> directory.
-However for production use you are recommended to configure your webserver
-to directly serve the static content as this is B<much> more efficient.
-
-Using an entire application server process (i.e. a perl process) to serve static media
-is a massive waste of resources, especially if you're not using a front-end proxy
-(as your web server ends up with a heavyweight perl process tied up sending bytes to
-a user rather than doing useful work).
-
-However, there are a number of cases where just unconditionally serving the static content
-to the user is a bad idea - think protected .pdf documents, think time limited URIs, think
-single use (or n-use) download URIs.
-
-Also, how about you have too much media to fit on one physical machine and you're using
-something like L<http://danga.com/mogilefs/>.
-
-=head3 Why not cache?
-
-That's a good question - generally you can deliver static media any way you want to
-by putting a cache (such as varnish) in front of your application and letting it
-handle most of the static hits (as static assets will be cached by it) is a great solution.
-
-However, in some cases (specifically one use URIs, when you want to check ACLs and
-when you want to log all accesses to a resource) then it's not possible to perform
-any sort of caching in front of your application as that will defeat the purpose.
-
-=head2 How?
-
-As a first implementation, then using the
-L<serve_static_file|Catalyst::Plugin::Static::Simple/serve_static_file>
-method is a good way to start.
-
-If your app is small and for internal company use only, then you quite possibly never need
-to go any further...
-
-However, if you're serving large files and likely to have an appreciable number of users,
-then sending bytes in perl just won't scale.
-
-=head2 Scaling
-
-The key to scaling is to do things using the minimum resources possible. Sending bytes across
-the internet is something that web servers are very good at scaling to do.
-
-So the planned architecture will be something like:
-
-=over
-
-=item *
-
-Request comes into your web server.
-
-=item *
-
-Request is sent to your Catalyst application running as FastCGI.
-
-=item *
-
-Application validates the request, increments any counters necessary
-and issues a response with C<X-Accel-Redirect> or C<X-Sendfile> header set.
-
-=item *
-
-Your web server takes care of serving the content back to the user as
-specified by that header.
-
-=back
-
-So the key is this C<X-Accel-Redirect> or C<X-Sendfile> thing - these are just
-headers that you get your application to send in the response (which one you
-use depends which web server you're running).
-
-After sending this response (headers only, no content), your application server
-is freed up and able to get on with handling the next request.
-
-=head3 A note on web servers
-
-This topic appears to be something that people get very excited about, but for
-the purposes of this discussion - whichever web server you prefer is likely to
-work fine, I'll go through the common choices below.
-
-=head4 Apache
-
-Apache is generally thought of as I<big>, but this is very much to do with what
-modules you have loaded and what MPM you're using. A lightweight Apache is perfectly
-acceptable to be serving content with. C<mod_xsendfile> provides the C<X-Sendfile>
-support.
-
-=head4 Lighttpd
-
-Lighttpd has simpler config to apache if you're writing a 'new' one, but your
-sysadmins are also less likely know their way around administering it.
-Has X-SendFile support as standard.
-
-=head4 nginx
-
-Works differently (and uses a different header) to either of the above.
-L<X-Accel-Redirect> actually doesn't logicially send a file from disk,
-it internally redirects the request to another path known by the web server,
-bypassing ACL checking.
-
-This means that the correct header to send depends on your web server
-configuration, which is a disadvantage, but as nginx can act as a proxy as well
-as a web server, this can also be more powerful, as nginx can proxy for a
-backend content server.
-
-=head4 perlbal
-
-Perl based load balancer which supports an C<X-REPROXY-URL> header to allow
-you to reproxy content to a backend content server.
-
-C<X-REPROXY-URL> is not covered in this article as I don't have any experience
-with it myself, and you still need a web server to serve your main application.
-
-=head2 Example
-
-Here is a worked example is going to show using C<nginx>, to reproxy a C<MogileFS> installation,
-as that's a what I'm using in production to deliver 150Tb of content at high speed :)
-
-In this case I'm using the MogileFS module for nginx (L<see below|/SEE ALSO>) to serve
-the actual content.
-
-It should be noted that C<MogileFS> isn't trivial to install or maintain, but
-if you're prepared to expend the time and expertise then it is an extremely cost
-effective solution to file storage on a large scale.
-
-You can, however, have a B<much> simpler infrastructure serving media from a directory
-on local disk if the amount of content you're delivering isn't huge.
-
-=head3 nginx config
-
- http {
- include /etc/nginx/mime.types;
- access_log /var/log/nginx/access.log;
-
- sendfile on;
- keepalive_timeout 65;
- tcp_nodelay on;
-
- gzip on;
-
- server {
- listen 80;
- root /opt/FileServer/webroot;
-
- location / {
- include /etc/nginx/fastcgi_params;
- fastcgi_pass unix:/tmp/fileserver.socket;
- access_log off;
- }
-
- location /private/mogile/ {
- internal;
- mogilefs_tracker 1.1.1.1:7001;
- mogilefs_domain mydomain;
- mogilefs_pass {
- proxy_pass $mogilefs_path;
- proxy_hide_header Content-Type;
- proxy_buffering off;
- }
- access_log off;
- }
- }
- }
-
-=head3 Example code snippet for generating timed uris.
-
-This method assumes that it's being called on a file object which stores various pieces
-of metadata about a file.
-
- method generate_timed_uri (%p) {
- $p{ttl} ||= 60 * 60 * 24 * 7; # Default to a week
-
- # Turn integers into strings to save space in the URIs (and make them less obvious)
- my $id = Math::BaseCalc->new(digits => ['a' .. 'z'])->to_base( $self->id ); # The primary key of the row
-
- my $secret = $self->secret; # This is a random secret string associated with
- # (but not related to the contents of) the file object
-
- # Generate a time rounded off to last expiry point, then 2x forward of the TTL.
- $time = Math::BaseCalc->new(digits => ['a' .. 'z'])->to_base( (time() % $ttl) + $ttl * 2 );
-
- # Note this is theoretically cryptographically weak, should really use HMAC.
- my $hash = Digest::SHA1::sha1_hex($self->secret . '-' . $time);
-
- return '/file/' . join('/', $hash, $time, $id, $self->filename . '.' . $self->extension);
- }
-
-=head3 Example code snippet for serving timed uris.
-
- sub file : Local {
- # Note we ignore the filename in the URI generated above, it is just there
- # to make the uri look 'normal'..
- my ($self, $c, $hash, $time, $id) = @_;
-
- $id = Math::BaseCalc->new(digits => ['a' .. 'z'])->from_base($id);
-
- my $file = eval { $schema->resultset('File')->find($id) }
- or return $self->error_notfound($c);
-
- return $self->error_notfound($c)
- unless $hash eq Digest::SHA1::sha1_hex($file->secret . '-' . $time)
-
- # Time embedded in the URL is the latest time this image is valid.
- $expire_time = Math::BaseCalc->new(digits => ['a' .. 'z'])->from_base($time);
- if ($expire_time < time()) {
- return $self->error_expired($c, $file);
- }
- # NOTE: In this case as you know when the URI expires, it is possible
- # to serve cache headers and have the hot items cached by your
- # front end proxy. This is meant to be a simple example and so
- # this is left as an exercise for the reader :)
- $c->res->header('Content-Type', $file->mimetype);
- $c->res->header('Accept-Ranges', 'none');
-
- # Note: this matches the path in the nginx config above.
- # With X-Sendfile, it would be the filename on disk instead.
- my $redirect_to = '/private/mogile/' . $file->mogile_id;
- $c->res->header('X-Accel-Redirect' => $redirect_to);
- }
-
-=head2 SEE ALSO
-
-L<http://blog.lighttpd.net/articles/2006/07/02/x-sendfile>
-
-L<http://wiki.nginx.org/NginxXSendfile>
-
-L<http://tn123.ath.cx/mod_xsendfile/>
-
-L<http://www.danga.com/perlbal/>
-
-L<http://github.com/vkholodkov/nginx-mogilefs-module>
-
-L<http://www.danga.com/mogilefs/>
-
-=head2 AUTHOR
-
- Tomas Doran (t0m) <bobtfish at bobtfish.net>
More information about the Catalyst-commits
mailing list