[Catalyst] Unicode trouble with Catalyst::Engine::FastCGI

[Bernhard Graf]

Good news everyone!
I found the gist of the matter and some places that should be fixed.

> Googling around I found
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg00051.html ,
> but I couldn't verify Jonathan's assumption in my case - even before the
> buffer is written into the FastCGI pipe in
> Catalyst::Engine::FastCGI::write the whole buffer (header and body)
> looked fine right before calling STDOUT->syswrite.

Meanwhile I realized, that the final output buffer (header + body)
actually /has/ the UTF-8 flag set. So it seems, that Jonathan's idea
(above link) also matches my case.
Everything seems fine again, when I insert this line

  utf8::downgrade($buffer) if utf8::is_utf8($buffer);

into Catalyst::Engine::FastCGI::write() before
*STDOUT->syswrite($buffer).

While this fixes the problem, it is still unclear, why the utf8 flag is
set for the whole buffer.

Since the body has already be turned into a clean octet-stream by
C:P:Unicode, the header must have been flagged as utf8 - and this was
actually the case. Digging further I found out, that only the cookie
string had the utf8 flag. The cookie is built with CGI::Simple::Cookie
from a hash, and in this hash only one value had the utf8 flag:
$cookie->{domain}. And this comes from myapp.yml, which is a YAML file,
of course.

So Ladies and Gentleman, may I present you the culprit?
It is YAML::XS! Everything read by YAML::XS

perl -MYAML::XS -E '
my $config = YAML::XS::LoadFile("myapp.yml");
say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
'
is utf8

On the old installation YAML::XS was not installed. Actually I installed
that module right after I re-installed everything from the snapshot.

perl -MYAML::Syck -E '
my $config = YAML::Syck::LoadFile("myapp.yml");
say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
'
is not utf8

So my first step is to kick YAML::XS. KICK, KICK, KICK!!!
Restart Cat-App - now it tells me:

  Use of YAML::Syck or YAML to parse config files is DEPRECATED. Please
  install YAML::XS for proper YAML support at
  /opt/perl/5.10.1/lib/site_perl/Config/Any.pm line 198

But my output is fixed! WoohooOO!

So first of all YAML::XS must be fixed (I'll file a bug report now).

Also Catalyst::Engine(::FastCGI) should assure, that this implicit
ut8::upgrade does not happen when the body has already been encoded into
an octet-stream. But I'm not sure if utf8::downgrade() is the right
approach to fix this, because other Cat engines are not affected by this
utf8-flag bug, and those don't explicitly utf8::downgrade the buffer either.


Bernhard Graf