[Catalyst] Re: Unicode trouble with Catalyst::Engine::FastCGI

[Aristotle Pagaltzis]

* Bernhard Graf <catalyst4 at augensalat.de> [2009-11-23 19:10]:
> Meanwhile I realized, that the final output buffer (header
> + body) actually /has/ the UTF-8 flag set. So it seems, that
> Jonathan's idea (above link) also matches my case. Everything
> seems fine again, when I insert this line
>
>   utf8::downgrade($buffer) if utf8::is_utf8($buffer);
>
> into Catalyst::Engine::FastCGI::write() before
> *STDOUT->syswrite($buffer).

The conditional is superfluous. If you downgrade a string that
doesn’t need downgrading, it’s a no-op.

> While this fixes the problem, it is still unclear, why the utf8
> flag is set for the whole buffer.

It shouldn’t matter.

> Since the body has already be turned into a clean octet-stream
> by C:P:Unicode, the header must have been flagged as utf8 - and
> this was actually the case. Digging further I found out, that
> only the cookie string had the utf8 flag. The cookie is built
> with CGI::Simple::Cookie from a hash, and in this hash only one
> value had the utf8 flag: $cookie->{domain}. And this comes from
> myapp.yml, which is a YAML file, of course.
>
> So Ladies and Gentleman, may I present you the culprit? It is
> YAML::XS! Everything read by YAML::XS
>
> perl -MYAML::XS -E '
> my $config = YAML::XS::LoadFile("myapp.yml");
> say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
> '
> is utf8

No, that’s not the culprit.

The culprit is Catalyst::Engine::FastCGI, which does not pay
attention to the UTF8 flag.

> On the old installation YAML::XS was not installed. Actually
> I installed that module right after I re-installed everything
> from the snapshot.
>
> perl -MYAML::Syck -E '
> my $config = YAML::Syck::LoadFile("myapp.yml");
> say((utf8::is_utf8($config->{name}) ? "is" : "is not"), " utf8");
> '
> is not utf8
>
> So my first step is to kick YAML::XS. KICK, KICK, KICK!!!
> Restart Cat-App - now it tells me:
>
>   Use of YAML::Syck or YAML to parse config files is DEPRECATED. Please
>   install YAML::XS for proper YAML support at
>   /opt/perl/5.10.1/lib/site_perl/Config/Any.pm line 198
>
> But my output is fixed! WoohooOO!
>
> So first of all YAML::XS must be fixed (I'll file a bug report now).

No. YAML::XS is completely correct.

> Also Catalyst::Engine(::FastCGI) should assure, that this
> implicit ut8::upgrade does not happen when the body has already
> been encoded into an octet-stream.

Nope.

> But I'm not sure if utf8::downgrade() is the right approach to
> fix this, because other Cat engines are not affected by this
> utf8-flag bug, and those don't explicitly utf8::downgrade the
> buffer either.

I’m not sure why ::FastCGI is affected specifically. Whether
a string is upgraded or not should make no difference, though. In
both cases the output should be the same. If it’s not, then
something is broken in ::FastCGI.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>