[Catalyst] $c->write() buffering?

[Daniel J. Luke]

I've done some more experimentation and this doesn't end up working very well for us. For the list archive:

You can accomplish what I was going for using mod_fastcgi and a standalone FCGI-based script. mod_fastcgi needs to be configured with its -flush option /and/ you need to be able to call FCGI::Flush($request) when you want output sent out to the client.

Catalyst::Engine:FastCGI doesn't expose the FCGI request (lexical variable in the run() method), though, so there's not an easy way to do this (and it's icky anyway). Catalyst::Engine::FastCGI should possibly FCGI::Flush in its write() method [note that Plack::Handler::FCGI doesn't expose the FCGI request either and also doesn't call FCGI::Flush anywhere].

So, I guess no one is doing $c->res->write() with mod_fastcgi and http://wiki.catalystframework.org/wiki/longcomputations should probably be edited to explain that this doesn't work (unless you pad each write() you want to go to the client to 8k).

We'll likely implement some AJAX ui (possibly tying up 2 catalyst processes instead of just one and/or fork/exec a worker) for now, until we can get some time to move things over to a job queue that isn't as horrible as the one we currently have :-)

On Nov 1, 2013, at 11:42 AM, John Napiorkowski <jjn1056 at yahoo.com> wrote:
> I was reviewing code last night for release of Catalyst Hamburg dev5 and saw that $c->res->write is just calling the writer under the hood.  So as long as your headers are ready to finalize, that should be fine as well.  Just remember this approach is going to block, so be careful with anything  that is taking a lot of time.  --jnap
> 
> 
> On Thursday, October 31, 2013 4:34 PM, John Napiorkowski <jjn1056 at yahoo.com> wrote:
> Recent releases of Catalyst makes it possible to stream write, although you need to be careful when using a blocking web server (long stream will of course block the available listener).
> 
> Older versions of Catalyst had similar ability with the write method, I never used it, and would be happen to get patches or test cases to make it work as we'd like.
> 
> Here's a link to an example, this one assume AnyEvent as a loop, but you could use a similar technique with any web server, as long as you don't mind blocking:
> 
> https://metacpan.org/pod/release/JJNAPIORK/Catalyst-Runtime-5.90049_005/lib/Catalyst/Response.pm#res-write_fh
> 
> If you follow that, you will see stuff 'stream' as you wish unless there is some buffering going on at some other level of the stack.
> 
> Johnn
> 
> 
> On Thursday, October 31, 2013 4:22 PM, Daniel J. Luke <dluke at geeklair.net> wrote:
> I replicated this today outside of Catalyst (just a small FCGI/FCGI::ProcManager test script). If anyone else has seen this/fixed it, I'd appreciate a pointer. I'll report back to the list if/when I get it resolved so that there's an answer in the list archives for future people to google otherwise.
> 
> On Oct 31, 2013, at 10:12 AM, Daniel J. Luke <dluke at geeklair.net> wrote:
> > We're actually running Catalyst::Runtime 5.80031 (currently), so I believe it's using Catalyst::Engine::FastCGI which just does *STDOUT->syswrite()
> > 
> > I guess I try to do some testing with newer Catalyst (and maybe alternate deployment methods), to see if that changes anything. Looking through the Changelog, I didn't see anything specifically related to this, though (although I imagine the Plack stuff makes it somewhat different in the more recent releases). 
> > 
> > On Oct 30, 2013, at 11:03 PM, Hailin Hu <i at h2l.name> wrote:
> >> It is an engine relevant stuff.
> >> Find which engine you are using ( for example, Plack::Handler::FCGI )
> >> and look around codes around write(r), you may find something.
> >> Good luck :)
> >> 
> >> On Wed, Oct 30, 2013 at 9:51 AM, Daniel J. Luke <dluke at geeklair.net> wrote:
> >>> I've got some legacy CGI code that does a bunch of processing and uses the old hack of $| = 1; print "foo\n"; do_work(); print "foo done\n"; etc. (solution #1 from http://wiki.catalystframework.org/wiki/longcomputations)
> >>> 
> >>> While I'll eventually convert it to a job queue, I'd like to create an output-identical implementation first with Catalyst, however it seems like I'm getting output buffering when I don't want it.
> >>> 
> >>> As a very simple test, I've set up apache 2.2, mod_fastcgi (FastCgiExternalServer with -flush and without -flush) and a method like this:
> >>> 
> >>> sub test: Local {
> >>>  my ($self, $c) = @_;
> >>> 
> >>>  $c->res->body('');
> >>>  $c->response->content_type( 'text/plain; charset=utf-8' );
> >>>  $c->finalize_headers;
> >>> 
> >>>  my $i;
> >>>  for($i=1;$i<8;$i++) {
> >>>      $c->write("$i: foo bar baz\n");
> >>>      sleep(1);
> >>>  }
> >>> }
> >>> 
> >>> I see all the data at once in my browser instead of a line every second, and with tcpdump, can see that all of the data is coming back in one packet and not in 8+ smaller packets like I expect. If I make the string that gets passed to write longer, I get several packets, but all at once (and not with each iteration through the for loop).
> >>> 
> >>> Am I missing something obvious? Is there some way to get the behavior I'm expecting?

--
Daniel J. Luke                                                                   
+========================================================+                        
| *---------------- dluke at geeklair.net ----------------* |                          
| *-------------- http://www.geeklair.net -------------* |                          
+========================================================+                        
|   Opinions expressed are mine and do not necessarily   |                          
|          reflect the opinions of my employer.          |                          
+========================================================+