<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2800.1609" name=GENERATOR></HEAD>
<BODY
style="WORD-WRAP: break-word; webkit-nbsp-mode: space; webkit-line-break: after-white-space">
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008>Perhaps I should take this opportunity to summarise my
investigation of this so far:</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>In my
error log, I see this:</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008> [warn] mod_fcgid: stderr: UTF-16 surrogate
0xda5a at /usr/local/lib/perl/5.8.8/IO/Handle.pm line 461</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>(A
different line number but then again a different release of
perl.)</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>The
corresponding access log extract lookslike this:</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>
"POST /cp/files/picker_start HTTP/1.1" 200 55898</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008>FireBug agrees that the response was 55898 bytes and
shows that it does not contain either of the surrogate bytes cited in the error
message - it is plain HTML and contains nothing unusual. (I have also used
tcpdump to verify this.)</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008>However, I note that 55898 == 0xDA5A, which is the
surrogate cited in the error message.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>This
is a pattern I have seen in several test cases now, and varying the size of the
returned HTML has a predictable effect on the reported
surrogate.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>I
therefore believe that the reported surrogate is actually the size field in the
FastCGI protocol header and that the fault is in the FCGI library, more
specifically in the FCGI.so part of the FCGI module. This appears to be bound as
an I/O handler at this point so it is plausible that it might cause a warning to
be reported "at /usr/local/lib/perl/5.8.8/IO/Handle.pm line 461". I have poked
around a bit in the relevant code but I have some learning to do before I get
any further - I'm not familiar with perl's I/O architecture at this level or
with .so modules, and re-building the experimental pure-perl FCGI.pm causes
different problems which break things catastrophically before I can test this
particular piece of code.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>If I
find a solution to this I will post it here as there is relevant discussion but
I do now believe it is basically off topic (unless I am seeing a different
problem with annoyingly similar symptoms - what are the chances of that?
:). I suppose it's possible that there is something FCGI.pm doesn't
like about the environment Catalyst sets up for it, but my feeling is that
it ought not to be sensitive in this regard.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=707103813-27052008>Merlyn
Kline</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=707103813-27052008></SPAN></FONT> </DIV>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> Ash Berlin
[mailto:ash_cpan@firemirror.com]<BR><B>Sent:</B> 24 May 2008
10:43<BR><B>To:</B> The elegant MVC web framework<BR><B>Subject:</B> Re:
[Catalyst] UTF-16 surrogate message when writing
binarydata(image)<BR><BR></FONT></DIV>Hmmm,
<DIV><BR></DIV>
<DIV>That looks fine by itself.</DIV>
<DIV><BR></DIV>
<DIV>Could you put a test case together in a minimal catalyst app for other
ppl to try?</DIV>
<DIV><BR></DIV>
<DIV>-ash</DIV>
<DIV><BR>
<DIV>
<DIV>On 24 May 2008, at 10:12, Martin Ellison wrote:</DIV><BR
class=Apple-interchange-newline>
<BLOCKQUOTE type="cite">Ash,<BR><BR>
<DIV style="MARGIN-LEFT: 40px"> <SPAN
style="FONT-FAMILY: courier new,monospace"> my $pic_image =
$p->ImageToBlob();</SPAN><BR
style="FONT-FAMILY: courier new,monospace"><SPAN
style="FONT-FAMILY: courier new,monospace">
$c->res->content_type(q{image/jpeg});</SPAN><BR
style="FONT-FAMILY: courier new,monospace"><SPAN
style="FONT-FAMILY: courier new,monospace">
$c->res->header( 'Content-Disposition', q{inline} );</SPAN><BR
style="FONT-FAMILY: courier new,monospace"><SPAN
style="FONT-FAMILY: courier new,monospace">
$c->res->body($pic_image);</SPAN><BR></DIV><BR>$p is an ImageMagick
picture object which is creating a byte sequence representing a JPEG image
(<SPAN style="FONT-FAMILY: courier new,monospace">$pic_image</SPAN>).
I have set the output to Unicode using the Unicode plugin. This is going
into an <img> tag.<BR><BR>
<DIV class=gmail_quote>2008/5/23 Ash Berlin <<A
href="mailto:ash_cpan@firemirror.com">ash_cpan@firemirror.com</A>>:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>You're going to have to paste some code I've been happily serving up
PDFs (which has all sorts of binary data in it) fine.
<DIV><BR></DIV>
<DIV>1) How are you serving the file up?</DIV>
<DIV>2) What headers are you setting?</DIV>
<DIV><BR></DIV>
<DIV>-ash</DIV>
<DIV><BR>
<DIV>
<DIV>
<DIV></DIV>
<DIV class=Wj3C7c>
<DIV>On 23 May 2008, at 16:00, Martin Ellison wrote:</DIV><BR></DIV></DIV>
<BLOCKQUOTE type="cite">
<DIV>
<DIV></DIV>
<DIV class=Wj3C7c>Another case... the surrogate is d862 = 55394 but the
image size is 45296, so it is not looking consistent. So perhaps the
'surrogate' data is just random bytes from the JPEG.<BR><BR>
<DIV class=gmail_quote>2008/5/23 Martin Ellison <<A
href="mailto:m.e@acm.org" target=_blank>m.e@acm.org</A>>:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">I've
found a case<BR>
<DIV style="MARGIN-LEFT: 40px"><SPAN
style="FONT-FAMILY: courier new,monospace">...stderr: UTF-16 surrogate
0xdb71 at /usr/lib/perl/5.8/IO/Handle.pm line 199...</SPAN><BR
style="FONT-FAMILY: courier new,monospace"><SPAN
style="FONT-FAMILY: courier new,monospace">FastCGI: ... [info] picture
size is 55966</SPAN><BR></DIV><BR>55966 = da9e<BR>56177 = db71
<BR> <BR>So almost the same. Perhaps I have 100 bytes of
headers.<BR><BR><BR>
<DIV class=gmail_quote>2008/5/20 Martin Ellison <<A
href="mailto:m.e@acm.org" target=_blank>m.e@acm.org</A>>:
<DIV>
<DIV></DIV>
<DIV><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">The
Content-Type is set to 'UTF-8' for most of my pages, but, in this
case, it is image/jpeg and the content disposition is set to
inline.<BR><BR>
<DIV class=gmail_quote>2008/5/19 Ash Berlin <<A
href="mailto:ash_cpan@firemirror.com"
target=_blank>ash_cpan@firemirror.com</A>>:
<DIV>
<DIV></DIV>
<DIV><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>On 19 May 2008, at 12:34, Martin Ellison wrote:<BR><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">I've
finally had the opportunity to test my code a bit more. The
$pic_image is a string of some kind and not a file
handle.<BR><BR>Also I tried doing binmode STDOUT but this fails
-- the binmode call returns false.<BR><BR>There could be
any binary data in my string, so any UTF-8-bad byte pairs are
definitely possible. I would think the solution is to inform
Catalyst that the data is not UTF-8 but binary. In the normal
case of writing to a file, one can use binmode to tell perl that
the data is binary but that does not work here.<BR><BR>Is this a
gap in the Catalyst API?<BR></BLOCKQUOTE><BR></DIV>Are you by any
chance setting a header to something with UTF8 content?<BR><BR>I
had a problem similar to this that was caused by me setting a
Content-Disposition header to something including
unicode.<BR><FONT color=#888888><BR>-ash</FONT>
<DIV>
<DIV></DIV>
<DIV><BR><BR>_______________________________________________<BR>List:
<A href="mailto:Catalyst@lists.scsys.co.uk"
target=_blank>Catalyst@lists.scsys.co.uk</A><BR>Listinfo: <A
href="http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst"
target=_blank>http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst</A><BR>Searchable
archive: <A
href="http://www.mail-archive.com/catalyst@lists.scsys.co.uk/"
target=_blank>http://www.mail-archive.com/catalyst@lists.scsys.co.uk/</A><BR>Dev
site: <A href="http://dev.catalyst.perl.org/"
target=_blank>http://dev.catalyst.perl.org/</A><BR></DIV></DIV></BLOCKQUOTE></DIV></DIV></DIV><BR><BR
clear=all>
<DIV>
<DIV></DIV>
<DIV><BR>-- <BR>Regards,<BR>Martin<BR>(<A href="mailto:m.e@acm.org"
target=_blank>m.e@acm.org</A>)<BR>IT: <A
href="http://methodsupport.com"
target=_blank>http://methodsupport.com</A> Personal: <A
href="http://thereisnoend.org"
target=_blank>http://thereisnoend.org</A>
</DIV></DIV></BLOCKQUOTE></DIV></DIV></DIV>
<DIV>
<DIV></DIV>
<DIV><BR><BR clear=all><BR>-- <BR>Regards,<BR>Martin<BR>(<A
href="mailto:m.e@acm.org" target=_blank>m.e@acm.org</A>)<BR>IT: <A
href="http://methodsupport.com"
target=_blank>http://methodsupport.com</A> Personal: <A
href="http://thereisnoend.org"
target=_blank>http://thereisnoend.org</A>
</DIV></DIV></BLOCKQUOTE></DIV><BR><BR clear=all><BR>--
<BR>Regards,<BR>Martin<BR>(<A href="mailto:m.e@acm.org"
target=_blank>m.e@acm.org</A>)<BR></DIV></DIV>IT: <A
href="http://methodsupport.com"
target=_blank>http://methodsupport.com</A> Personal: <A
href="http://thereisnoend.org" target=_blank>http://thereisnoend.org</A>
_______________________________________________
<DIV class=Ih2E3d><BR>List: <A href="mailto:Catalyst@lists.scsys.co.uk"
target=_blank>Catalyst@lists.scsys.co.uk</A><BR>Listinfo: <A
href="http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst"
target=_blank>http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst</A><BR>Searchable
archive: <A
href="http://www.mail-archive.com/catalyst@lists.scsys.co.uk/"
target=_blank>http://www.mail-archive.com/catalyst@lists.scsys.co.uk/</A><BR>Dev
site: <A href="http://dev.catalyst.perl.org/"
target=_blank>http://dev.catalyst.perl.org/</A><BR></DIV></BLOCKQUOTE></DIV><BR></DIV></DIV><BR>_______________________________________________<BR>List:
<A
href="mailto:Catalyst@lists.scsys.co.uk">Catalyst@lists.scsys.co.uk</A><BR>Listinfo:
<A href="http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst"
target=_blank>http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst</A><BR>Searchable
archive: <A href="http://www.mail-archive.com/catalyst@lists.scsys.co.uk/"
target=_blank>http://www.mail-archive.com/catalyst@lists.scsys.co.uk/</A><BR>Dev
site: <A href="http://dev.catalyst.perl.org/"
target=_blank>http://dev.catalyst.perl.org/</A><BR><BR></BLOCKQUOTE></DIV><BR><BR
clear=all><BR>-- <BR>Regards,<BR>Martin<BR>(<A
href="mailto:m.e@acm.org">m.e@acm.org</A>)<BR>IT: <A
href="http://methodsupport.com">http://methodsupport.com</A> Personal: <A
href="http://thereisnoend.org">http://thereisnoend.org</A>
_______________________________________________<BR>List: <A
href="mailto:Catalyst@lists.scsys.co.uk">Catalyst@lists.scsys.co.uk</A><BR>Listinfo:
<A
href="http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst">http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst</A><BR>Searchable
archive: <A
href="http://www.mail-archive.com/catalyst@lists.scsys.co.uk/">http://www.mail-archive.com/catalyst@lists.scsys.co.uk/</A><BR>Dev
site: <A
href="http://dev.catalyst.perl.org/">http://dev.catalyst.perl.org/</A><BR></BLOCKQUOTE></DIV><BR></DIV></BLOCKQUOTE></BODY></HTML>