<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div dir="ltr" id="yui_3_16_0_1_1438626911564_16590"><span id="yui_3_16_0_1_1438626911564_16589">I'd be interesting in having some sort of flag on request, that indicated if the incoming query was bad. I can't do a die here for legacy reasons.</span></div><div dir="ltr"><span><br></span></div><div dir="ltr" id="yui_3_16_0_1_1438626911564_16591"><span>jnap</span></div> <br><div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" style="display: block;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div dir="ltr"> <font size="2" face="Arial"> On Sunday, August 2, 2015 9:39 AM, Bill Moseley <moseley@hank.org> wrote:<br> </font> </div> <br><br> <div class="y_msg_container"><div id="yiv7826684112"><div><div dir="ltr">BTW -- I wonder about the Catalyst behavior here.<br clear="none"><div class="yiv7826684112gmail_extra"><br clear="none"><div class="yiv7826684112gmail_quote">On Sat, Aug 1, 2015 at 10:36 PM, Bill Moseley <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:moseley@hank.org" target="_blank" href="mailto:moseley@hank.org">moseley@hank.org</a>></span> wrote:<br clear="none"><blockquote class="yiv7826684112gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div dir="ltr"><br clear="none"><div class="yiv7826684112gmail_extra"><br clear="none"><div class="yiv7826684112gmail_quote"><span class="yiv7826684112">On Sat, Aug 1, 2015 at 6:31 AM, Stefan <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:maillist@s.profanter.me" target="_blank" href="mailto:maillist@s.profanter.me">maillist@s.profanter.me</a>></span> wrote:<br clear="none"></span><blockquote class="yiv7826684112gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="DE"><div><div class="yiv7826684112MsoNormal"><span lang="EN-US">Hi,<u></u><u></u></span></div><div class="yiv7826684112MsoNormal"><span lang="EN-US">if a URL parameter contains a Unicode character (e.g. <a rel="nofollow" shape="rect" target="_blank" href="http://www.example.com/?param=%D6lso%DF">www.example.com/?param=%D6lso%DF</a> which stands for param=Ölsoße), the parameter is not correctly parsed as Unicode.</span></div></div></div></blockquote></div></div></div></blockquote><div><br clear="none"></div><div>One note here -- data over the wire must be encoded into octets. So, all Unicode characters must be encoded and then decoded when received. (You can't send "Unicode characters".) UTF-8 is used now (for obvious reasons). <a rel="nofollow" shape="rect" target="_blank" href="http://tools.ietf.org/html/rfc3986">http://tools.ietf.org/html/rfc3986</a>.</div><div><br clear="none"></div><div>You are specifying %D6 -- although the Unicode characters is U+00D6, the UTF-8 octet sequence is 0xC3 0x96. See: <a rel="nofollow" shape="rect" target="_blank" href="http://www.fileformat.info/info/unicode/char/00D6/index.htm">http://www.fileformat.info/info/unicode/char/00D6/index.htm</a></div><div><br clear="none"></div><div>Unless otherwise instructed, <a rel="nofollow" shape="rect" target="_blank" href="https://github.com/perl-catalyst/catalyst-runtime/blob/master/lib/Catalyst/Engine.pm#L579">Catalyst uses UTF-8</a> as the encoding for decoding query parameters -- query parameters are decoded from UTF-8 octets to Perl characters.</div><div><br clear="none"></div><div>As your example showed, if you use invalid UTF-8 sequences then Encode::decode() as used by Catalyst will replace those with the <a rel="nofollow" shape="rect" target="_blank" href="http://www.fileformat.info/info/unicode/char/fffd/index.htm">U+FFFD substitution character</a> "�".</div><div><br clear="none"></div><div>This may or may not be what you want. Personally, I think it's not correct to silently modify user input. You intended to pass "Ölsoße" but ended up with "�lso�e" -- is that really the data you would want to process/store for the request? Seems unlikely.</div><div><br clear="none"></div><div>If "param" is suppose to be passed as textual, UTF-8-encoded octets, and it isn't, then maybe returning a 400 is a better way of handling that. That probably would have helped you see what is wrong in this case.</div><div><br clear="none"></div><div>i.e. use "eval { decode( $default_query_encoding, $str, FB_CROAK | LEAVE_SRC ); }" to catch invalid data and return to the client the "$str" that failed and why.</div><div><br clear="none"></div><div>Of course, it is also possible that you have some query parameters that you want decoded as UTF-8 and some that might represent something else (a raw sequence of bytes), and want more manual control. In that case $c->config->{do_not_decode_query} could be used to bypass the decoding. But then, you must manually decode() yourself.</div></div><div class="yiv7826684112yqt8894549399" id="yiv7826684112yqtfd43463"><div><br clear="none"></div>-- <br clear="none"><div class="yiv7826684112gmail_signature">Bill Moseley<br clear="none"><a rel="nofollow" shape="rect" ymailto="mailto:moseley@hank.org" target="_blank" href="mailto:moseley@hank.org">moseley@hank.org</a></div>
</div></div></div></div></div><br><div class="yqt8894549399" id="yqtfd76020">_______________________________________________<br clear="none">List: <a shape="rect" ymailto="mailto:Catalyst@lists.scsys.co.uk" href="mailto:Catalyst@lists.scsys.co.uk">Catalyst@lists.scsys.co.uk</a><br clear="none">Listinfo: <a shape="rect" href="http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst" target="_blank">http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst</a><br clear="none">Searchable archive: <a shape="rect" href="http://www.mail-archive.com/catalyst@lists.scsys.co.uk/" target="_blank">http://www.mail-archive.com/catalyst@lists.scsys.co.uk/</a><br clear="none">Dev site: <a shape="rect" href="http://dev.catalyst.perl.org/" target="_blank">http://dev.catalyst.perl.org/</a><br clear="none"></div><br><br></div> </div> </div> </div></div></body></html>