[Catalyst] mod_perl converts latin1 to utf8 !?

[Jonathan Rockway]

On Sun, Dec 21, 2008 at 11:52:27PM +0100, Bjørn-Helge Mevik wrote:

> I am developing a simple Catalyst application using MySQL, FormFu and
> TT.  I have everything encoded in ISO-8859-1: the data in MySQL, the
> Perl files, the FormFu .yml files, and the TT templates.  (I am even
> running in a ISO-8859-1 locale, for what it's worth.)
> 
> Following the advice in an earlier thread ("Charset best practice" - 
> <http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg00284.html>),
> I have overridden the process() method of Catalyst::View::TT with a
> version that does $c->response->content_type('text/html;
> charset=iso-8859-1');
> ...
> 
> What can I do about this?  Can I tell mod_perl to "leave my characters
> alone"?  Have I made an error somewhere?

Sort of.  Working with latin-1, or any character encoding, is just
like working with UTF-8.  When you work with UTF-8, you need to say
something like this:

   my $data = Encode::decode('utf8', $raw_data);
   process($data);
   print Encode::encode('utf8', $data);

We decode the binary data into text, then we work with the text, then
we encode the text back into binary data for the wire (or the user's
xterm, in this case). 

When you are working with iso-8859-1, you need to do exactly the same
thing.  Everything you read needs to be decoded, and everything you
need to write needs to be encoded.  (In this case, it is sort of an
uphill battle since most people use Unicode now, and that is the "code
path" with the most testing.  There are probably places that helpfully
treat your latin-1 as utf-8, which is definitely incorrect of it.)

Decoding is probably a no-op, so focus on the encoding part.  Take a
look at Catalyst::Plugin::Unicode (specifically finalize_body, I
think), and change the Encode::encode('utf-8', ...) to
Encode::encode('iso-8859-1', ...)

I bet this will solve your problem, but if not, let us know what code
you tried and where, and we will try to help you some more.

Regards,
Jonathan Rockway