[Catalyst] CSV / UTF-8 / Unicode

Craig Chant craig at homeloanpartnership.com
Tue Jul 2 13:34:43 GMT 2013


Hi Alexander,

Well I think I found the problem... the DBI module!

I went back to my original SQL wrapper for Win32::ODBC in my original procedural script application before porting to Catalyst (where I moved to DBI).

All my output to XLS in the old script using Win32::ODBC works perfectly, no funny chars in sight!

So it has to be flaky Windows support in the DBI module to blame wouldn't you say?

So either I refactor my SQL model wrappers under Catalyst to use Win32::ODBC, or I find out why DBI is corrupting my SQL data.

No wonder I was going round in circles trying to UTF-8 my data!

DBI on Windows has always been a thorn in my side especially with the "long read" issues that occur!

I guess I need to head to the Perl forums for DBI support, though it might be nice if MST chimed in as I know he's an expert on all things DBI/DBIC.

Is there a switch or flag or something I am missing to make DBI work properly under DBD::ODBC?

Oh and I did a test with DBI in my old procedural application script and got the same funny characters as I did with DBI under Catalyst, so I think we can safely say it is DBI with the problem and not Catalyst.

Any input greatly received.

Regards,
1DMF


-----Original Message-----
From: Alexander Hartmaier [mailto:alexander.hartmaier at t-systems.at]
Sent: 02 July 2013 13:49
To: catalyst at lists.scsys.co.uk
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

On 2013-07-02 14:20, Craig Chant wrote:
> Hi Alexander,
>
> I have read the DBD::ODBC docs and it seems Unicode is ON by default
>
> " Enabling and Disabling Unicode support
>
> On Windows Unicode support is enabled by default"
>
> So do I assume the data coming from the DB is Unicode encoded already?
>
> if so , why does decode fall over?
>
> I don't seem able to decode or encode without the app crashing?
>
> Do I have non-Unicode chars perhaps?
>
> Craig.
I'd suggest to test it outside of Catalyst with a simple standalone script.
I have neither experience with Perl on Windows nor with ODBC and MSSQL.
Ask on the DBIC list or on IRC, some guys use MSSQL successfully there.

BR Alex

>
> -----Original Message-----
> From: Alexander Hartmaier [mailto:alexander.hartmaier at t-systems.at]
> Sent: 02 July 2013 12:53
> To: catalyst at lists.scsys.co.uk
> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>
> On 2013-07-02 13:45, Craig Chant wrote:
>> Hi Mike,
>>
>> The data is coming from MS SQL 2008 R2, I have checked the DB and the column is defined as data type nvarchar(max) , so is set to Unicode and when viewing data via SQL Studio Manager, the Unicode characters display fine.
>>
>> I am accessing the DB via DBI and have set on the connection string..
>> {'RaiseError' => 1, 'mysql_enable_utf8' => 1}
> MSSQL != MySQL
>> Though I'm not sure if the utf8 setting has any affect when using an MS ODBC driver as the flag implies it's for mysql ?
> No it doesn't, read your DBD's docs.
>> I tried to apply  my $chars = decode('UTF-8', $xls); and I got an error saying you cannot decode a string with wide characters.
>>
>> I don't know what the string is, but when you try to decode you get a wide character error and when you try to encode you get a wide character error.
>>
>> If I don't do anything to it and just output it, no error, but the data is full of junk chars.
>>
>> Where do I go from here?
> Ensure that the data coming out from your database is properly decoded, then make sure it's properly encoded when outputting it.
> Never mess with Perls' internal representation of the data by dis-/enabling the utf-8 bit.
>> Thanks,
>> Craig.
>>
>>
>>
>> -----Original Message-----
>> From: Mike Whitaker [mailto:mike at altrion.org]
>> Sent: 02 July 2013 12:27
>> To: The elegant MVC web framework
>> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>>
>>
>>>       # encode UTF8 octet
>>>        use Encode qw(decode encode);
>>>        my $octets = encode('UTF-8', $xls);
>> OUt of curiousity, where's the data in $xls come from, and what do you know about its encoding and bytes vs characters status?
>> _______________________________________________
>> List: Catalyst at lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/ This Email and any
>> attachments contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority.
>>
>> _______________________________________________
>> List: Catalyst at lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>
>
>
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"
> *"*"*"* T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
> Handelsgericht Wien, FN 79340b
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"
> *"*"*"*
> Notice: This e-mail contains information that is confidential and may be privileged.
> If you are not the intended recipient, please notify the sender and then delete this e-mail immediately.
> *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"
> *"*"*"*
>
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/ This Email and any attachments
> contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority.
>
> _______________________________________________
> List: Catalyst at lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/



_______________________________________________
List: Catalyst at lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
This Email and any attachments contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority.



More information about the Catalyst mailing list