<p dir="ltr">Hello,</p>
<p dir="ltr">How are you running your application? Ie: mod_perl, fcgi, fcgi + httpd/nginx, plack + ... also what version of perl are you using and what os?</p>
<p dir="ltr">-roman</p>
<div class="gmail_quote">On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <<a href="mailto:bauch@zsi.at">bauch@zsi.at</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hey all,<div><br></div><div>on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.</div><div>the usually appear if a spider </div><div><span style="white-space:pre-wrap"> </span>Mozilla/5.0 (compatible; EasouSpider; +<a href="http://www.easou.com/search/spider.html" target="_blank">http://www.easou.com/search/spider.html</a>)</div>
<div>comes accross.</div><div><br></div><div>the error is:</div><div><span style="white-space:pre-wrap"> </span>Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.</div>
<div><br></div><div>It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...</div><div><br></div><div>—————————————</div><div>[2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/…./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***<br>
[2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/…./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown<br>[2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)<br>
.---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.<br>| Action | Time |<br>
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+<br>'---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'</div>
<div>—————————————</div><div><br></div><div>i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends … the results are these:</div><div>UTF8 trash arrives - and the module seems unable to deal with it…</div>
<div><br></div><div>————————————</div><div>Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.<br> -<br><br>
URL: notice/list<br><br>PARAMS:$VAR1 = {<br> 'X*Ö^K^@^@^@^@¸®ä<br>^@^@^@^@8<83>^H^K^@^@^@^@h¡ä<br>^@^@^@^@Hµä<br>^@^@^@^@X^Z^N^Q^@^@^@^@ø<91>^F^Q^@^@^@^@Ø^F^N^Q^@^@^@^@¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@¸úÝ^P^@^@^@^@^X%q^G^@^@^@^@اñ^O^@^@^@^@ØøB.^@^@^@^@èâÝ^P^@^@^@^@XÛ_^L^@^@^@^@ÈíÝ^P^@^@^@^@¸~P^S^@^@^@^@èåÝ^P^@^@^@^@Øný^O^@^@^@^@<88>úÝ^P^@^@^@^@^Xá( ^@^@^@^@ئÆ<br>
^@^@^@^@Øï*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Т^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@'<br>
};<br><br><br> // value: $VAR1 = '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Т^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@';<br>
<br><br>headers: Connection: close<br>Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1<br>Accept-Encoding: gzip, deflate<br>Accept-Language: zh;q=0.9,en;q=0.8<br>
Host: <a href="http://wbc-inco.net" target="_blank">wbc-inco.net</a><br>User-Agent: Mozilla/5.0 (compatible; EasouSpider; +<a href="http://www.easou.com/search/spider.html" target="_blank">http://www.easou.com/search/spider.html</a>)<br>
Content-Length: 927<br>Content-Type: application/x-www-form-urlencoded<br>REFER: <a href="http://b------.net/“" target="_blank">http://b------.net/“</a></div><div><br></div><div>————————————<br><br></div><div>to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding</div>
<div><br></div><div>————————————————————</div><div>around line 168:</div><div><br></div><div> my $val;<br> eval {<br> $val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );<br>
};<br> if ($@){<br> # UPS !<br> # get request infos<br>use Data::Dumper;<br>my $params = $self->req->parameters;<br>my $headers= $self->req->headers->as_string;<br>die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;<br>
….</div><div>————————————————————</div><div><br></div><div>I guess my Catalyst Apps are not the only ones with these errors ?</div><div><br></div><div><br></div><div>about my App settings / config:</div><div><br></div><div>
app-config has</div><div><div><span style="white-space:pre-wrap"> </span>encoding UTF-8</div></div><div><br></div><div>App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)</div>
<div><br></div><div>i am using postgres with</div><div><span style="white-space:pre-wrap"> </span>pg_enable_utf8 1</div><div>(but the error about is far away from any DB related problem i guess)</div><div><br></div><div>using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)</div>
<div><br></div><div>i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket</div><div><span style="white-space:pre-wrap"> </span><a href="https://rt.cpan.org/Public/Bug/Display.html?id=94957" target="_blank">https://rt.cpan.org/Public/Bug/Display.html?id=94957</a></div>
<div>but i does not look as it was this problem ...</div><div><br></div><div>Any ideas what todo ?</div><div>Add a issue/ticket ?</div><div><br></div><div>thanks for feedback,</div><div>bernhard bauch<span style="white-space:pre-wrap"> </span></div>
<div><br></div><div><br></div><div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>—</div><div>Bernhard Bauch, Webdevelopment<br>ZSI - Zentrum für soziale Innovation<br>
<a href="mailto:bauch@zsi.at" target="_blank">bauch@zsi.at</a><br>Skype: berni-zsi</div></div>
</div>
<br></div></div><br>_______________________________________________<br>
List: <a href="mailto:Catalyst@lists.scsys.co.uk">Catalyst@lists.scsys.co.uk</a><br>
Listinfo: <a href="http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst" target="_blank">http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst</a><br>
Searchable archive: <a href="http://www.mail-archive.com/catalyst@lists.scsys.co.uk/" target="_blank">http://www.mail-archive.com/catalyst@lists.scsys.co.uk/</a><br>
Dev site: <a href="http://dev.catalyst.perl.org/" target="_blank">http://dev.catalyst.perl.org/</a><br>
<br></blockquote></div>