[Catalyst] UTF-16 surrogate message when writing binarydata(image)

Tue May 27 15:03:30 BST 2008

Perhaps I should take this opportunity to summarise my investigation of this
so far:

In my error log, I see this:
   [warn] mod_fcgid: stderr: UTF-16 surrogate 0xda5a at
/usr/local/lib/perl/5.8.8/IO/Handle.pm line 461
(A different line number but then again a different release of perl.)

The corresponding access log extract lookslike this:
  "POST /cp/files/picker_start HTTP/1.1" 200 55898

FireBug agrees that the response was 55898 bytes and shows that it does not
contain either of the surrogate bytes cited in the error message - it is
plain HTML and contains nothing unusual. (I have also used tcpdump to verify
this.)

However, I note that 55898 == 0xDA5A, which is the surrogate cited in the
error message.

This is a pattern I have seen in several test cases now, and varying the
size of the returned HTML has a predictable effect on the reported
surrogate.

I therefore believe that the reported surrogate is actually the size field
in the FastCGI protocol header and that the fault is in the FCGI library,
more specifically in the FCGI.so part of the FCGI module. This appears to be
bound as an I/O handler at this point so it is plausible that it might cause
a warning to be reported "at /usr/local/lib/perl/5.8.8/IO/Handle.pm line
461". I have poked around a bit in the relevant code but I have some
learning to do before I get any further - I'm not familiar with perl's I/O
architecture at this level or with .so modules, and re-building the
experimental pure-perl FCGI.pm causes different problems which break things
catastrophically before I can test this particular piece of code.

If I find a solution to this I will post it here as there is relevant
discussion but I do now believe it is basically off topic (unless I am
seeing a different problem with annoyingly similar symptoms - what are the
chances of that? :). I suppose it's possible that there is something FCGI.pm
doesn't like about the environment Catalyst sets up for it, but my feeling
is that it ought not to be sensitive in this regard.

Merlyn Kline

  -----Original Message-----
  From: Ash Berlin [mailto:ash_cpan at firemirror.com]
  Sent: 24 May 2008 10:43
  To: The elegant MVC web framework
  Subject: Re: [Catalyst] UTF-16 surrogate message when writing
binarydata(image)

  Hmmm,

  That looks fine by itself.

  Could you put a test case together in a minimal catalyst app for other ppl
to try?

  -ash

  On 24 May 2008, at 10:12, Martin Ellison wrote:

    Ash,

        my $pic_image = $p->ImageToBlob();
        $c->res->content_type(q{image/jpeg});
        $c->res->header( 'Content-Disposition', q{inline} );
        $c->res->body($pic_image);

    $p is an ImageMagick picture object which is creating a byte sequence
representing a JPEG image ($pic_image).  I have set the output to Unicode
using the Unicode plugin. This is going into an <img> tag.

    2008/5/23 Ash Berlin <ash_cpan at firemirror.com>:

      You're going to have to paste some code I've been happily serving up
PDFs (which has all sorts of binary data in it) fine.

      1) How are you serving the file up?
      2) What headers are you setting?

      -ash

      On 23 May 2008, at 16:00, Martin Ellison wrote:

        Another case... the surrogate is d862 = 55394 but the image size is
45296, so it is not looking consistent. So perhaps the 'surrogate' data is
just random bytes from the JPEG.

        2008/5/23 Martin Ellison <m.e at acm.org>:

          I've found a case

          ...stderr: UTF-16 surrogate 0xdb71 at
/usr/lib/perl/5.8/IO/Handle.pm line 199...
          FastCGI: ... [info] picture size is 55966

          55966 = da9e
          56177 = db71

          So almost the same. Perhaps I have 100 bytes of headers.

          2008/5/20 Martin Ellison <m.e at acm.org>:

            The Content-Type is set to 'UTF-8' for most of my pages, but, in
this case, it is image/jpeg and the content disposition is set to inline.

            2008/5/19 Ash Berlin <ash_cpan at firemirror.com>:

              On 19 May 2008, at 12:34, Martin Ellison wrote:

                I've finally had the opportunity to test my code a bit more.
The $pic_image is a string of some kind and not a file handle.

                Also I tried doing binmode STDOUT but this fails --  the
binmode call returns false.

                There could be any binary data in my string, so any
UTF-8-bad byte pairs are definitely possible. I would think the solution is
to inform Catalyst that the data is not UTF-8 but binary. In the normal case
of writing to a file, one can use binmode to tell perl that the data is
binary but that does not work here.

                Is this a gap in the Catalyst API?

              Are you by any chance setting a header to something with UTF8
content?

              I had a problem similar to this that was caused by me setting
a Content-Disposition header to something including unicode.

              -ash

              _______________________________________________
              List: Catalyst at lists.scsys.co.uk
              Listinfo:
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
              Searchable archive:
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
              Dev site: http://dev.catalyst.perl.org/

            --
            Regards,
            Martin
            (m.e at acm.org)
            IT: http://methodsupport.com Personal: http://thereisnoend.org

          --
          Regards,
          Martin
          (m.e at acm.org)
          IT: http://methodsupport.com Personal: http://thereisnoend.org

        --
        Regards,
        Martin
        (m.e at acm.org)

        IT: http://methodsupport.com Personal: http://thereisnoend.org
_______________________________________________

        List: Catalyst at lists.scsys.co.uk
        Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
        Searchable archive:
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
        Dev site: http://dev.catalyst.perl.org/

      _______________________________________________
      List: Catalyst at lists.scsys.co.uk
      Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
      Searchable archive:
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
      Dev site: http://dev.catalyst.perl.org/

    --
    Regards,
    Martin
    (m.e at acm.org)
    IT: http://methodsupport.com Personal: http://thereisnoend.org
_______________________________________________
    List: Catalyst at lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive:
http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20080527/f319c7ea/attachment.htm