[Dbix-class] inserting utf8 data into varchar columns for a postgres utf8 database

[John Napiorkowski]

--- Adam Paynter <adampaynter at gmail.com> wrote:

> Whoops, "fix" is actually supposed to be "sub fix".
> Silly me.
> 
> On 10/5/06, Adam Paynter <adampaynter at gmail.com>
> wrote:
> >
> > Howdy,
> >
> > We have run into many nightmares at our
> organization with regards to this.
> > We also use PostgreSQL 8. Although there are other
> ways of solving the
> > problem, I typically do something as follows:
> >
> > use Encode qw( _utf8_off _utf8_on from_to );
> >
> > fix {
> >     my $s = shift;
> >     return undef unless defined $s;
> >     _utf8_off( $s );
> >     from_to( $s, 'cp1250', 'utf8' );
> >     _utf8_on( $s );
> >     return $s;
> > }
> >
> > my $nice_string = fix( $ugly_string );
> >
> > Perhaps this will help!
> >
> > - Adam

Yes, that did help.  I also found the root issue was
from some LDIF files being generated by MS Outlook
contained the Latin1 character set.  Looks like I
might be able to fix this with a PerlIO layer against
the IO::File object returned by the Catalyst
$c->request->uploads... method.

The Catalyst Unicode plugin seems to take care of
other submitted parameters, but these files where
uploads.  I can see why by default you wouldn't want
to mess with the characterset of potentially binary
files :)

So actually this wasn't a DBIx issue at all, that's
just where the error message showed up :)

Thanks!

John


> >
> > On 10/4/06, John Napiorkowski <jjn1056 at yahoo.com>
> wrote:
> > >
> > > Okay,
> > >
> > > Sorry this had nothing to do with utf8, but some
> data
> > > that was formated in CHARSET=windows-1252, that
> crazy
> > > windows only format.
> > >
> > > I found a modules called, "Encode::ZapCP1252"
> but that
> > > didn't seem to help me.  It actually seemed to
> just
> > > delete all the values it received without
> changing
> > > anything.
> > >
> > > I could run a regex to clean this out but that's
> > > really ugly.
> > >
> > > Has anyone run into that and found a more
> elegant
> > > solution (besides yelling at the people sending
> me
> > > windows only data?)?
> > >
> > > Thanks!
> > >
> > > John Napiorkowski
> > >
> > > --- John Napiorkowski <jjn1056 at yahoo.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm having trouble inserting data that has
> some utf8
> > > > characters mixed in.  I am using a postgres
> 8.1x
> > > > database and the database was created with
> utf8
> > > > option.
> > > >
> > > > The error I get is:
> > > >
> > > > INSERT INTO ... execute failed: ERROR invalid
> byte
> > > > sequence for encoding "UTF8".
> > > >
> > > > The value it's failing on is "Bjørn Stabell".
> > > >
> > > > I looked at the DBIx::Class::UTF8Columns
> component
> > > > but
> > > > I wasn't sure how that could help me or if it
> could
> > > > help me.
> > > >
> > > > I'm running this under Catalyst and thought I
> was
> > > > doing utf8 correctly, based on the
> (unfortunately
> > > > few)
> > > > examples I could find.
> > > >
> > > > I'm sure this is something simple I should do
> but
> > > > google is not being kind to my inquiries.  How
> have
> > > > the rest of you been dealing with this?
> > > >
> > > > Thanks!
> > > > John Napiorkowski
> > > >
> > > >
> __________________________________________________
> > > > Do You Yahoo!?
> > > > Tired of spam?  Yahoo! Mail has the best spam
> > > > protection around
> > > > http://mail.yahoo.com
> > > >
> > > >
> _______________________________________________
> > > > List:
> > > >
> > >
>
http://lists.rawmode.org/cgi-bin/mailman/listinfo/dbix-class
> > > > Wiki:
> http://dbix-class.shadowcatsystems.co.uk/
> > > > IRC: irc.perl.org#dbix-class
> > > > SVN:
> > > >
> > >
>
http://dev.catalyst.perl.org/repos/bast/trunk/DBIx-Class/
> > > > Searchable Archive:
> > > >
> > >
>
http://www.mail-archive.com/dbix-class@lists.rawmode.org/
> > > >
> > >
> > >
> > >
> __________________________________________________
> > > Do You Yahoo!?
> > > Tired of spam?  Yahoo! Mail has the best spam
> protection around
> > > http://mail.yahoo.com
> > >
> > > _______________________________________________
> > > List:
>
http://lists.rawmode.org/cgi-bin/mailman/listinfo/dbix-class
> > > Wiki: http://dbix-class.shadowcatsystems.co.uk/
> > > IRC: irc.perl.org#dbix-class
> > > SVN:
>
http://dev.catalyst.perl.org/repos/bast/trunk/DBIx-Class/
> > > Searchable Archive:
> > >
>
http://www.mail-archive.com/dbix-class@lists.rawmode.org/
> > >
> >
> >
> > _______________________________________________
> List:
>
http://lists.rawmode.org/cgi-bin/mailman/listinfo/dbix-class
> Wiki: http://dbix-class.shadowcatsystems.co.uk/
> IRC: irc.perl.org#dbix-class
> SVN:
>
http://dev.catalyst.perl.org/repos/bast/trunk/DBIx-Class/
> Searchable Archive:
http://www.mail-archive.com/dbix-class@lists.rawmode.org/


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com