From adsj at novozymes.com Wed Sep 1 11:05:58 2010 From: adsj at novozymes.com (Adam =?iso-8859-1?Q?Sj=F8gren?=) Date: Wed Sep 1 11:06:05 2010 Subject: [Dbix-class] Iteration with ->next versus using ->all, and count Message-ID: <87eiddpvu1.fsf@topper.koldfront.dk> Last week I upgraded to Ubuntu 10.04, and went from DBIx::Class 0.08107 to 0.08115. In an application I noticed a change in iterating using ->next, compared to calling ->all (on a resultset obtained with search_related), that I find a little odd: One returns one record, the other returns three. Here is an attempt at a minimal example (using Postgres and DBIx::Class from git (93d0eb5)): $ createdb cardb $ PGOPTIONS='--client-min-messages=warning' psql cardb --quiet -f init.sql $ DBIC_TRACE=1 ./test.pl DBIx::Class::VERSION: 0.08115 SELECT me.id, me.company FROM fleet me WHERE ( me.id = ? ): '1' SELECT COUNT( * ) FROM car me JOIN driver drivers ON drivers.car_id = me.id WHERE ( me.fleet_id = ? ): '1' Count: 3 Iterator: SELECT drivers.id, drivers.name, drivers.car_id FROM car me LEFT JOIN driver drivers ON drivers.car_id = me.id WHERE ( me.fleet_id = ? ): '1' Christensen All: SELECT drivers.id, drivers.name, drivers.car_id FROM car me LEFT JOIN driver drivers ON drivers.car_id = me.id WHERE ( me.fleet_id = ? ): '1' Christensen Elgaard Magnussen $ cat test.pl #!/usr/bin/perl use warnings; use strict; use CarDB; print 'DBIx::Class::VERSION: ' . $DBIx::Class::VERSION . "\n\n"; my $schema=CarDB->connect('dbi:Pg:dbname=cardb', 'core', 'core', { AutoCommit=>0 }); my $fleet=$schema->resultset('CarDB::Fleet')->find({ id=>1 }); my $drivers=$fleet->cars->search_related('drivers'); print "Count: " . $drivers->count . "\n"; print "\nIterator:\n"; while (my $driver=$drivers->next) { print " " . $driver->name . "\n"; } print "\nAll:\n"; foreach my $driver ($drivers->all) { print " " . $driver->name . "\n"; } $ cat init.sql CREATE TABLE fleet ( id integer NOT NULL, company text NOT NULL, PRIMARY KEY (id) ); CREATE TABLE car ( id integer NOT NULL, plate text NOT NULL, fleet_id integer REFERENCES fleet NOT NULL, PRIMARY KEY (id) ); CREATE TABLE driver ( id integer NOT NULL, name text NOT NULL, car_id integer REFERENCES car NOT NULL, PRIMARY KEY (id) ); INSERT INTO fleet (id, company) VALUES (1, 'Hertz'); INSERT INTO car (id, plate, fleet_id) VALUES (1, 'AAA 101', 1); INSERT INTO car (id, plate, fleet_id) VALUES (2, 'BBB 202', 1); INSERT INTO car (id, plate, fleet_id) VALUES (3, 'CCC 303', 1); INSERT INTO car (id, plate, fleet_id) VALUES (4, 'DDD 404', 1); INSERT INTO driver (id, name, car_id) VALUES (1, 'Christensen', 1); INSERT INTO driver (id, name, car_id) VALUES (2, 'Elgaard', 3); INSERT INTO driver (id, name, car_id) VALUES (3, 'Magnussen', 4); $ cat CarDB.pm package CarDB; use strict; use warnings; use base qw(DBIx::Class::Schema); __PACKAGE__->load_classes(); 1; $ cat CarDB/ Car.pm Driver.pm Fleet.pm $ cat CarDB/Fleet.pm package CarDB::Fleet; use strict; use warnings; use base qw(DBIx::Class); __PACKAGE__->load_components('PK::Auto', 'Core'); __PACKAGE__->table('fleet'); __PACKAGE__->add_columns(qw(id company)); __PACKAGE__->set_primary_key('id'); __PACKAGE__->has_many(cars=>'CarDB::Car', 'fleet_id', { cascade_delete=>0 }); 1; $ cat CarDB/Car.pm package CarDB::Car; use strict; use warnings; use base qw(DBIx::Class); __PACKAGE__->load_components('PK::Auto', 'Core'); __PACKAGE__->table('car'); __PACKAGE__->add_columns(qw(id plate fleet_id)); __PACKAGE__->set_primary_key('id'); __PACKAGE__->belongs_to(fleet=>'CarDB::Fleet', 'fleet_id'); __PACKAGE__->has_many(drivers=>'CarDB::Driver', 'car_id', { cascade_delete=>0 }); 1; $ cat CarDB/Driver.pm package CarDB::Driver; use strict; use warnings; use base qw(DBIx::Class); __PACKAGE__->load_components('PK::Auto', 'Core'); __PACKAGE__->table('driver'); __PACKAGE__->add_columns(qw(id name car_id)); __PACKAGE__->set_primary_key('id'); __PACKAGE__->belongs_to(car=>'CarDB::Car', 'car_id'); 1; $ Also, as seen above, if I call ->count on the resultset, the generated SQL uses JOIN, while ->next and ->all both use LEFT JOIN - I was expecting all three to run the "same" query? Running the same example with DBIx::Class 0.08107 gives this result: $ DBIC_TRACE=1 ./test.pl DBIx::Class::VERSION: 0.08107 SELECT me.id, me.company FROM fleet me WHERE ( me.id = ? ): '1' SELECT COUNT( * ) FROM car me LEFT JOIN driver drivers ON drivers.car_id = me.id WHERE ( me.fleet_id = ? ): '1' Count: 4 Iterator: SELECT drivers.id, drivers.name, drivers.car_id FROM car me LEFT JOIN driver drivers ON drivers.car_id = me.id WHERE ( me.fleet_id = ? ): '1' Christensen Use of uninitialized value in concatenation (.) or string at ./test.pl line 19. Elgaard Magnussen All: SELECT drivers.id, drivers.name, drivers.car_id FROM car me LEFT JOIN driver drivers ON drivers.car_id = me.id WHERE ( me.fleet_id = ? ): '1' Christensen Use of uninitialized value in concatenation (.) or string at ./test.pl line 24. Elgaard Magnussen Issuing rollback() for database handle being DESTROY'd without explicit disconnect(). $ So in 0.08107 ->next and ->all do the same thing (albeit arguably the empty driver shouldn't have been returned). In 0.08115 ->count and ->all return the results I was expecting, but ->next doesn't. Shouldn't it skip the record with no driver? (Or maybe both ->next and ->all should use JOIN, like ->count); I am not quite sure. Maybe it is all driver error? Best regards, Adam -- Adam Sj?gren adsj@novozymes.com From rabbit+dbic at rabbit.us Wed Sep 1 12:50:15 2010 From: rabbit+dbic at rabbit.us (Peter Rabbitson) Date: Wed Sep 1 12:50:22 2010 Subject: [Dbix-class] Iteration with ->next versus using ->all, and count In-Reply-To: <87eiddpvu1.fsf@topper.koldfront.dk> References: <87eiddpvu1.fsf@topper.koldfront.dk> Message-ID: <4C7E4C07.3040106@rabbit.us> Adam Sj?gren wrote: > Last week I upgraded to Ubuntu 10.04, and went from DBIx::Class 0.08107 > to 0.08115. > > In an application I noticed a change in iterating using ->next, compared > to calling ->all (on a resultset obtained with search_related), that I > find a little odd: One returns one record, the other returns three. > > Here is an attempt at a minimal example (using Postgres and DBIx::Class > from git (93d0eb5)): You say you used the latest version from git, yet I don't see the results below, only 0.08107 and 0.08115 (both are pretty old). Additionally there is this: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?p=dbsrgits/DBIx-Class.git;a=blob;f=Changes;h=e9e8654b02c00f39af9b4b07bacb9fca050c7e5f;hb=HEAD#l214 I'd recommend upgrading to the latest official CPAN version (0.08123) and see if this helps. If you need to backport the particular fix to 0.08115 it is located here: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?p=dbsrgits/DBIx-Class.git;a=commitdiff;h=4f9eebe193524afdbe7e4089cd23db666600f3c6;hp=37cb0de173eed31c34a583b1404fcef6f41da0ce Cheers From adsj at novozymes.com Wed Sep 1 13:30:28 2010 From: adsj at novozymes.com (Adam =?iso-8859-1?Q?Sj=F8gren?=) Date: Wed Sep 1 13:30:34 2010 Subject: [Dbix-class] Re: Iteration with ->next versus using ->all, and count In-Reply-To: <4C7E4C07.3040106@rabbit.us> (Peter Rabbitson's message of "Wed, 01 Sep 2010 14:50:15 +0200") References: <87eiddpvu1.fsf@topper.koldfront.dk> <4C7E4C07.3040106@rabbit.us> Message-ID: <871v9dpp57.fsf@topper.koldfront.dk> On Wed, 01 Sep 2010 14:50:15 +0200, Peter wrote: >> Here is an attempt at a minimal example (using Postgres and DBIx::Class >> from git (93d0eb5)): > You say you used the latest version from git, yet I don't see the results > below, only 0.08107 and 0.08115 (both are pretty old). Ah, sorry - I messed up the -I option (I used -I~/blahblah which didn't work, note the missing space) and mistakenly thought that the version in the clone and the Ubuntu package was the same. Trying, for real this time, the latest git it works as expected. Great! > I'd recommend upgrading to the latest official CPAN version (0.08123) and > see if this helps. If you need to backport the particular fix to 0.08115 > it is located here: I will backport the packages from Ubuntu unstable to lucid and use them. Thanks! Best regards, Adam -- Adam Sj?gren adsj@novozymes.com From adsj at novozymes.com Wed Sep 1 14:46:05 2010 From: adsj at novozymes.com (Adam =?iso-8859-1?Q?Sj=F8gren?=) Date: Wed Sep 1 14:46:17 2010 Subject: [Dbix-class] Re: Iteration with ->next versus using ->all, and count References: <87eiddpvu1.fsf@topper.koldfront.dk> <4C7E4C07.3040106@rabbit.us> <871v9dpp57.fsf@topper.koldfront.dk> Message-ID: <87lj7lo72q.fsf@topper.koldfront.dk> On Wed, 01 Sep 2010 15:30:28 +0200, Adam wrote: > On Wed, 01 Sep 2010 14:50:15 +0200, Peter wrote: >> I'd recommend upgrading to the latest official CPAN version (0.08123) and >> see if this helps. If you need to backport the particular fix to 0.08115 >> it is located here: > I will backport the packages from Ubuntu unstable to lucid and use them. > Thanks! Ok, I was a little optimistic there - after backporting libclass-accessor-grouped-perl, libclass-mop-perl, libhash-merge-perl, libmath-base36-perl, libmoose-perl, libmoosex-types-json-perl, libnamespace-clean-perl, libpackage-stash-perl, libsql-translator-perl, and libsql-abstract-perl (phew) to lucid, building libdbix-class-perl failed with a lot of tests not working out. So I ended up backporting the fix to 0.08115; making the patch against 0.08115 was easy with git and your pointer to the merge - thanks! Best regards, Adam -- Adam Sj?gren adsj@novozymes.com From tiziano at graphistudio.com Wed Sep 1 17:09:38 2010 From: tiziano at graphistudio.com (Tiziano Faion) Date: Wed Sep 1 17:09:43 2010 Subject: [Dbix-class] Dinamic join conditions Message-ID: <4C7E88D2.9080007@graphistudio.com> Dear Mailing List, i'm developing an utility to administrate some things in an intranet enviroinment. I've got two tables one with 3 fields : id, seminar_group_id, desc and the other one that contains 3 fields also id, seminar_group_id role_id the second table contains the seminar group id that a role is able to see. what i want to get is a complete list of the groups , and field that return true or false if a user can see the group. i got this by executing this query select a.id, group_name, role_id from seminari_gruppi a left join seminar_2_roles b on ( a.id = b.seminar_group_id AND role_id = $role_id) looking into google i've discovered that this can be done by using the from, and this is what i've done my $allGroups_rs = $c->model('DbsMysql::Seminars::Gruppi')->search(undef, { from => \" (SELECT a.id , group_name, group_desc, testo , img , role_id FROM seminari_gruppi a LEFT JOIN seminar_2_roles b ON a.id = b.seminar_group_id AND b.role_id = $id_role ) me" }); but... how can i access role_id ? as i can do a thing like this push @{$json->{groups}} , { group_id => $r->id, group_desc => $r->group_desc, group_name => $r->group_name, cansee => defined $r->{role_id} ? JSON::XS::true : JSON::XS::false }; Thanks guys! From orasnita at gmail.com Fri Sep 3 18:38:18 2010 From: orasnita at gmail.com (Octavian Rasnita) Date: Fri Sep 3 18:39:24 2010 Subject: [Dbix-class] test1 from gmail Message-ID: Octavian From octavian.rasnita at ssifbroker.ro Fri Sep 3 18:38:57 2010 From: octavian.rasnita at ssifbroker.ro (Octavian Rasnita) Date: Fri Sep 3 18:39:32 2010 Subject: [Dbix-class] test2 - from ssifbroker Message-ID: Octavian From pau4o at kamennn.eu Fri Sep 3 19:58:06 2010 From: pau4o at kamennn.eu (Kamen Naydenov) Date: Fri Sep 3 19:58:12 2010 Subject: [Dbix-class] test2 - from ssifbroker In-Reply-To: References: Message-ID: 2010/9/3 Octavian Rasnita : > > Octavian test from ssifbroker passed From pau4o at kamennn.eu Fri Sep 3 19:58:44 2010 From: pau4o at kamennn.eu (Kamen Naydenov) Date: Fri Sep 3 19:58:48 2010 Subject: [Dbix-class] test1 from gmail In-Reply-To: References: Message-ID: 2010/9/3 Octavian Rasnita : > > Octavian same for test form gmail From dan.horne at redbone.co.nz Mon Sep 6 03:58:01 2010 From: dan.horne at redbone.co.nz (Dan Horne) Date: Mon Sep 6 03:58:09 2010 Subject: [Dbix-class] Date not inflating Message-ID: Hi Consider the following: A config has many sections. A section has many process runs. I want to find the latest process run from a config and section combination: my $rs =3D $scheduler->EpProcessRun->search( { 'ep_config.config_name' =3D> 'my_config', 'ep_section.section_name' =3D> 'my_section', }, { select =3D> [ { max =3D> 'me.date_created' } ], as =3D> [ 'date_created' ], 'join' =3D> {'ep_section' =3D> ['ep_config']}, } ); say $rs->first->get_column('date_created'); This works - but the date is text, whereas I was hoping it would be inflated to a DateTime object (as they are with regular queries). The generated SQL is SELECT MAX(me.date_created) FROM ep_process_run me JOIN ep_section ep_section ON ep_section.section_id =3D me.section_id JOIN ep_config ep_config ON ep_config.config_id =3D ep_section.config_id WHERE ((ep_config.config_name =3D ? AND ep_section.section_name =3D ?)) I also tried my $rs =3D $scheduler->EpProcessRun->search( { 'ep_config.config_name' =3D> 'my_config', 'ep_section.section_name' =3D> 'my_section', }, { 'join' =3D> {'ep_section' =3D> ['ep_config']}, } ); say $rs->get_column('date_created')->max; The generated SQL is exactly the same, and the date isn't inflated. Am I doing something wrong, or have I hit a DBIx::Class limitation? I;m on DBIx:Class v0.08123 on Win32 Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100906/b64= 6bf4b/attachment.htm From neil.lunn at trixan.com Mon Sep 6 04:16:44 2010 From: neil.lunn at trixan.com (Neil Lunn) Date: Mon Sep 6 04:16:58 2010 Subject: [Dbix-class] Date not inflating In-Reply-To: References: Message-ID: <1283746604.3579.550.camel@localhost.localdomain> Try: http://search.cpan.org/~frew/DBIx-Class-0.08123/lib/DBIx/Class/InflateColumn/DateTime.pm On Mon, 2010-09-06 at 15:58 +1200, Dan Horne wrote: > Hi > > Consider the following: A config has many sections. A section has many > process runs. I want to find the latest process run from a config and > section combination: > > my $rs = $scheduler->EpProcessRun->search( > { > 'ep_config.config_name' => 'my_config', > 'ep_section.section_name' => 'my_section', > }, > { > select => [ { max => 'me.date_created' } ], > as => [ 'date_created' ], > 'join' => {'ep_section' => ['ep_config']}, > } > ); > > say $rs->first->get_column('date_created'); > > This works - but the date is text, whereas I was hoping it would be > inflated to a DateTime object (as they are with regular queries). The > generated SQL is > > SELECT MAX(me.date_created) > FROM ep_process_run me > JOIN ep_section ep_section > ON ep_section.section_id = me.section_id > JOIN ep_config ep_config > ON ep_config.config_id = ep_section.config_id > WHERE ((ep_config.config_name = ? AND ep_section.section_name = ?)) > > I also tried > > my $rs = $scheduler->EpProcessRun->search( > { > 'ep_config.config_name' => 'my_config', > 'ep_section.section_name' => 'my_section', > }, > { > 'join' => {'ep_section' => ['ep_config']}, > } > ); > > say $rs->get_column('date_created')->max; > > The generated SQL is exactly the same, and the date isn't inflated. Am > I doing something wrong, or have I hit a DBIx::Class limitation? I;m > on DBIx:Class v0.08123 on Win32 > > Dan > > _______________________________________________ > List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class > IRC: irc.perl.org#dbix-class > SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/ > Searchable Archive: http://www.grokbase.com/group/dbix-class@lists.scsys.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100906/da8db454/attachment.htm From dan.horne at redbone.co.nz Mon Sep 6 04:31:43 2010 From: dan.horne at redbone.co.nz (Dan Horne) Date: Mon Sep 6 04:31:46 2010 Subject: [Dbix-class] Date not inflating In-Reply-To: <1283746604.3579.550.camel@localhost.localdomain> References: <1283746604.3579.550.camel@localhost.localdomain> Message-ID: Hi Which part of the documentation are you referring to? My dates normally inflate, but not when I apply the max function. On 6 September 2010 16:16, Neil Lunn wrote: > > Try: > > > http://search.cpan.org/~frew/DBIx-Class-0.08123/lib/DBIx/Class/InflateCol= umn/DateTime.pm > > > > > On Mon, 2010-09-06 at 15:58 +1200, Dan Horne wrote: > > Hi > > Consider the following: A config has many sections. A section has many > process runs. I want to find the latest process run from a config and > section combination: > > my $rs =3D $scheduler->EpProcessRun->search( > { > 'ep_config.config_name' =3D> 'my_config', > 'ep_section.section_name' =3D> 'my_section', > }, > { > select =3D> [ { max =3D> 'me.date_created' } ], > as =3D> [ 'date_created' ], > 'join' =3D> {'ep_section' =3D> ['ep_config']}, > } > ); > > say $rs->first->get_column('date_created'); > > This works - but the date is text, whereas I was hoping it would be > inflated to a DateTime object (as they are with regular queries). The > generated SQL is > > SELECT MAX(me.date_created) > FROM ep_process_run me > JOIN ep_section ep_section > ON ep_section.section_id =3D me.section_id > JOIN ep_config ep_config > ON ep_config.config_id =3D ep_section.config_id > WHERE ((ep_config.config_name =3D ? AND ep_section.section_name =3D ?)) > > I also tried > > my $rs =3D $scheduler->EpProcessRun->search( > { > 'ep_config.config_name' =3D> 'my_config', > 'ep_section.section_name' =3D> 'my_section', > }, > { > 'join' =3D> {'ep_section' =3D> ['ep_config']}, > } > ); > > say $rs->get_column('date_created')->max; > > The generated SQL is exactly the same, and the date isn't inflated. Am I > doing something wrong, or have I hit a DBIx::Class limitation? I;m on > DBIx:Class v0.08123 on Win32 > > Dan > > _______________________________________________ > List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class > IRC: irc.perl.org#dbix-class > SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/ > Searchable Archive: http://www.grokbase.com/group/dbix-class@lists.scsys.= co.uk > > > > _______________________________________________ > List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class > IRC: irc.perl.org#dbix-class > SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/ > Searchable Archive: > http://www.grokbase.com/group/dbix-class@lists.scsys.co.uk > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100906/551= 36bf2/attachment.htm From neil.lunn at trixan.com Mon Sep 6 05:16:27 2010 From: neil.lunn at trixan.com (Neil Lunn) Date: Mon Sep 6 05:16:39 2010 Subject: [Dbix-class] Date not inflating In-Reply-To: References: <1283746604.3579.550.camel@localhost.localdomain> Message-ID: <1283750187.3579.566.camel@localhost.localdomain> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: TrixanLogo.jpg Type: image/jpeg Size: 2011 bytes Desc: not available Url : http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100906/9bd14115/TrixanLogo-0001.jpg From dan.horne at redbone.co.nz Mon Sep 6 12:56:53 2010 From: dan.horne at redbone.co.nz (Dan Horne) Date: Mon Sep 6 12:56:58 2010 Subject: [Dbix-class] Date not inflating In-Reply-To: <1283750187.3579.566.camel@localhost.localdomain> References: <1283746604.3579.550.camel@localhost.localdomain> <1283750187.3579.566.camel@localhost.localdomain> Message-ID: Skipped content of type multipart/alternative-------------- next part -----= --------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 2011 bytes Desc: not available Url : http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100907/47= 4de205/attachment.jpeg From edencardim at gmail.com Tue Sep 7 18:00:04 2010 From: edencardim at gmail.com (Eden Cardim) Date: Tue Sep 7 18:00:27 2010 Subject: [Dbix-class] Dinamic join conditions In-Reply-To: <4C7E88D2.9080007@graphistudio.com> (Tiziano Faion's message of "Wed, 01 Sep 2010 19:09:38 +0200") References: <4C7E88D2.9080007@graphistudio.com> Message-ID: >>>>> "Tiziano" == Tiziano Faion writes: Tiziano> my $allGroups_rs = Tiziano> $c->model('DbsMysql::Seminars::Gruppi')->search(undef, { Tiziano> from => \" (SELECT a.id , group_name, group_desc, testo , Tiziano> img , role_id FROM seminari_gruppi a LEFT JOIN Tiziano> seminar_2_roles b ON a.id = b.seminar_group_id AND Tiziano> b.role_id = $id_role ) me" }); Tiziano> but... how can i access role_id ? as i can do a thing like Tiziano> this push @{$json->{groups}} , { group_id => $r->id, Tiziano> group_desc => $r->group_desc, group_name => $r->group_name, Tiziano> cansee => defined $r->{role_id} ? JSON::XS::true : Tiziano> JSON::XS::false }; Thanks guys! try: $rs->search({},{ '+select' => [qw(role_id)] }) $row->get_column('role_id'); -- Eden Cardim Need help with your Catalyst or DBIx::Class project? Code Monkey http://www.shadowcat.co.uk/catalyst/ Shadowcat Systems Ltd. Want a managed development or deployment platform? http://blog.edencardim.com/ http://www.shadowcat.co.uk/servers/ From guienator at gmail.com Mon Sep 20 04:08:54 2010 From: guienator at gmail.com (Wei Gui) Date: Mon Sep 20 04:09:02 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level Message-ID: I start use DBIx in my project. and I read this from DBIx::Class::ResultSet document >> Note: Because find_or_create() reads from the database and then possibly inserts based on the result, this method is subject to a race condition. Another process could create a record in the table after the find has completed and before the create has started. To avoid this problem, use find_or_create() inside a transaction. IMO, put find_or_create inside a transaction only works if the transaction level is serializable. But for Oracle whose default transaction level is "read committed", which means race condition in find_or_create may still happen even in a transaction. Can someone confirm it. if it is true, how use DBIx handle it thanks, From moseley at hank.org Mon Sep 20 06:01:00 2010 From: moseley at hank.org (Bill Moseley) Date: Mon Sep 20 06:01:25 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: References: Message-ID: On Sun, Sep 19, 2010 at 9:08 PM, Wei Gui wrote: > I start use DBIx in my project. and I read this from > DBIx::Class::ResultSet document > > >> Note: Because find_or_create() reads from the database and then possib= ly > inserts based on the result, this method is subject to a race condition. > Another process could create a record in the table after the find has > completed and before the create has started. To avoid this problem, use > find_or_create() inside a transaction. > > IMO, put find_or_create inside a transaction only works if the > transaction level is serializable. But for Oracle whose default > transaction level is "read committed", which means race condition in > find_or_create may still happen even in a transaction. > I find that comment in the docs vague, too. Postgresql, likewise, uses read committed as the default. Would a "create_or_find()" method solve that problem? I've got code that attempts to insert first, and on duplicate key violation I then fetch the row. -- = Bill Moseley moseley@hank.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100919/955= 93895/attachment.htm From joerg at braun-senden.de Mon Sep 20 07:32:06 2010 From: joerg at braun-senden.de (Joerg Braun) Date: Mon Sep 20 07:32:11 2010 Subject: [Dbix-class] Howto create multiple rows with INSERT INTO () SELECT FROM Message-ID: <71ff2999c5709ed7435bb3a84d744714@server88.greatnet.de> I can update a bunch of rows with resultset->search(...)->update(...). Is it possible to model with DBIC an SQL like INSERT INTO ( ) SELECT FROM
WHERE ; Thank You for any hints. From rabbit+dbic at rabbit.us Mon Sep 20 09:12:49 2010 From: rabbit+dbic at rabbit.us (Peter Rabbitson) Date: Mon Sep 20 09:12:56 2010 Subject: [Dbix-class] Howto create multiple rows with INSERT INTO () SELECT FROM In-Reply-To: <71ff2999c5709ed7435bb3a84d744714@server88.greatnet.de> References: <71ff2999c5709ed7435bb3a84d744714@server88.greatnet.de> Message-ID: <4C972591.1060803@rabbit.us> Joerg Braun wrote: > I can update a bunch of rows with resultset->search(...)->update(...). > > Is it possible to model with DBIC an SQL like > > INSERT INTO
( ) SELECT FROM
WHERE ; > Not yet, but this is a feature we'd like to have. You can kick-start the process by contributing an extra test to the test suite. Something like: $rs->populate ([ \@column_names, $source_rs->as_query ]); The repo can be found at http://github.com/dbix-class/DBIx-Class git://git.shadowcat.co.uk/dbsrgits/DBIx-Class.git Cheers From rabbit+dbic at rabbit.us Mon Sep 20 09:43:14 2010 From: rabbit+dbic at rabbit.us (Peter Rabbitson) Date: Mon Sep 20 09:43:20 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: References: Message-ID: <4C972CB2.4030008@rabbit.us> Wei Gui wrote: > I start use DBIx in my project. and I read this from > DBIx::Class::ResultSet document > >>> Note: Because find_or_create() reads from the database and then possibly inserts based on the result, this method is subject to a race condition. Another process could create a record in the table after the find has completed and before the create has started. To avoid this problem, use find_or_create() inside a transaction. > > IMO, put find_or_create inside a transaction only works if the > transaction level is serializable. But for Oracle whose default > transaction level is "read committed", which means race condition in > find_or_create may still happen even in a transaction. > > Can someone confirm it. if it is true, how use DBIx handle it > thanks, > find_or_create is just a convenience method, so you don't have to write one yourself. It is not (and can not be made) race-proof. Consider the possibilities: 1) We use a transaction the way we do now - depending on the isolation you use the 2nd racing select will either block, or will continue and then fail on insert. We can not mandate a certain isolation level so eventually it may fail (but it will fail immediatelly) 2) We issue the select with a read-lock. This however means that the lock will persist until the end of the current transaction, which may be coming *thousands* of statements later, which will effectively lock most of your database. So instead of an exception *now*, the user sees a locked database for... a while. 3) We try to create the row first, going in blind. This is a good idea in theory, however it has a nasty side effect that it makes the current transaction no-longer-commitable. So using transactions in a race-prone environment is out as well (the user sees random transactions failing for no apparent reason) Thus DBIC implements 1) which is the most predictable way to guard against races. From billcrawford1970 at gmail.com Mon Sep 20 10:35:22 2010 From: billcrawford1970 at gmail.com (Bill Crawford) Date: Mon Sep 20 10:35:27 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: <4C972CB2.4030008@rabbit.us> References: <4C972CB2.4030008@rabbit.us> Message-ID: On 20 September 2010 10:43, Peter Rabbitson wrote: > 1) We use a transaction the way we do now - depending on the isolation ... > 2) We issue the select with a read-lock. This however means that the ... > 3) We try to create the row first, going in blind. This is a good idea > in theory, however it has a nasty side effect that it makes the current > transaction no-longer-commitable. So using transactions in a race-prone > environment is out as well (the user sees random transactions failing > for no apparent reason) Or you could 4) check whether savepoints are supported, use one, try the insert, and see what happens? From billcrawford1970 at gmail.com Mon Sep 20 10:36:45 2010 From: billcrawford1970 at gmail.com (Bill Crawford) Date: Mon Sep 20 10:36:49 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: References: <4C972CB2.4030008@rabbit.us> Message-ID: On 20 September 2010 11:35, Bill Crawford wrote: > Or you could 4) check whether savepoints are supported, use one, try > the insert, and see what happens? s/you/we/ and, no I have no idea where in the DBIx code this could be done. Sorry for the patronising tone. From rabbit+dbic at rabbit.us Mon Sep 20 13:04:30 2010 From: rabbit+dbic at rabbit.us (Peter Rabbitson) Date: Mon Sep 20 13:04:36 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: References: <4C972CB2.4030008@rabbit.us> Message-ID: <4C975BDE.7060408@rabbit.us> Bill Crawford wrote: > On 20 September 2010 10:43, Peter Rabbitson wrote: > >> 1) We use a transaction the way we do now - depending on the isolation > ... >> 2) We issue the select with a read-lock. This however means that the > ... >> 3) We try to create the row first, going in blind. This is a good idea >> in theory, however it has a nasty side effect that it makes the current >> transaction no-longer-commitable. So using transactions in a race-prone >> environment is out as well (the user sees random transactions failing >> for no apparent reason) > > Or you could 4) check whether savepoints are supported, use one, try > the insert, and see what happens? Savepoints do not help here. An error happening within a savepoint invalidates *the entire transaction*. Test it on your RDBMS, but I am pretty sure this is the case all-around. From billcrawford1970 at gmail.com Mon Sep 20 13:35:33 2010 From: billcrawford1970 at gmail.com (Bill Crawford) Date: Mon Sep 20 13:35:37 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: <4C975BDE.7060408@rabbit.us> References: <4C972CB2.4030008@rabbit.us> <4C975BDE.7060408@rabbit.us> Message-ID: On 20 September 2010 14:04, Peter Rabbitson wrote: > Savepoints do not help here. An error happening within a savepoint > invalidates *the entire transaction*. Test it on your RDBMS, but > I am pretty sure this is the case all-around. Hmm. PG docs say: "Moreover, ROLLBACK TO is the only way to regain control of a transaction block that was put in aborted state by the system due to an error, short of rolling it back completely and starting again." which I thought implied that you can rollback to a savepoint after an error. I'll try it out ... From billcrawford1970 at gmail.com Mon Sep 20 13:54:32 2010 From: billcrawford1970 at gmail.com (Bill Crawford) Date: Mon Sep 20 13:54:36 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: <4C975BDE.7060408@rabbit.us> References: <4C972CB2.4030008@rabbit.us> <4C975BDE.7060408@rabbit.us> Message-ID: On 20 September 2010 14:04, Peter Rabbitson wrote: > Savepoints do not help here. An error happening within a savepoint > invalidates *the entire transaction*. Test it on your RDBMS, but > I am pretty sure this is the case all-around. Work fine, but if you have two clients trying to insert, the second one hangs until the first one commits the whole transaction, not just releases the savepoint. So it would probably be counterproductive anyway :-/ From moseley at hank.org Fri Sep 24 14:11:49 2010 From: moseley at hank.org (Bill Moseley) Date: Fri Sep 24 14:12:11 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: <4C972CB2.4030008@rabbit.us> References: <4C972CB2.4030008@rabbit.us> Message-ID: On Mon, Sep 20, 2010 at 2:43 AM, Peter Rabbitson > wrote: > > Consider the possibilities: > > 1) We use a transaction the way we do now - depending on the isolation > you use the 2nd racing select will either block, or will continue > and then fail on insert. We can not mandate a certain isolation level > so eventually it may fail (but it will fail immediatelly) > > 2) We issue the select with a read-lock. This however means that the > lock will persist until the end of the current transaction, which > may be coming *thousands* of statements later, which will effectively > lock most of your database. So instead of an exception *now*, the user > sees a locked database for... a while. > And the DBA sends a nasty email about the long transaction. > > 3) We try to create the row first, going in blind. This is a good idea > in theory, however it has a nasty side effect that it makes the current > transaction no-longer-commitable. So using transactions in a race-prone > environment is out as well (the user sees random transactions failing > for no apparent reason) > What if DBIC did the create_or_select type of call if not currently in a transaction, otherwise fall back to #1? Would that be generic enough to help with the issue? Frankly, I'm a bit shocked by how often I'm seeing the race condition hit. Obviously, find_or_create is a useful and needed convenience, so making it a bit less likely to fail would also be convenient. -- = Bill Moseley moseley@hank.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.scsys.co.uk/pipermail/dbix-class/attachments/20100924/1b0= 4dbd7/attachment.htm From rabbit+dbic at rabbit.us Fri Sep 24 14:40:58 2010 From: rabbit+dbic at rabbit.us (Peter Rabbitson) Date: Fri Sep 24 14:41:05 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: References: <4C972CB2.4030008@rabbit.us> Message-ID: <4C9CB87A.5040503@rabbit.us> Bill Moseley wrote: > > > On Mon, Sep 20, 2010 at 2:43 AM, Peter Rabbitson > wrote: > > 3) We try to create the row first, going in blind. This is a good idea > in theory, however it has a nasty side effect that it makes the current > transaction no-longer-commitable. So using transactions in a race-prone > environment is out as well (the user sees random transactions failing > for no apparent reason) > > > What if DBIC did the create_or_select type of call if not currently in a > transaction, otherwise fall back to #1? Would that be generic enough to > help with the issue? Certainly it can be done (the "are we in a transaction" state is precisely monitored in the storage object). However there is one more caveat - you can not reliably tell *why* the thing failed (duplicate or something else?). So I guess the proper way would be to: attempt insert if exception - attempt select if nothing found attempt insert again so you propagate the correct exception > Frankly, I'm a bit shocked by how often I'm seeing the race condition > hit. Obviously, find_or_create is a useful and needed convenience, so > making it a bit less likely to fail would also be convenient. Come to #dbix-class to get more info on how to actually write this Cheers From billcrawford1970 at gmail.com Fri Sep 24 15:28:18 2010 From: billcrawford1970 at gmail.com (Bill Crawford) Date: Fri Sep 24 15:28:24 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: <4C9CB87A.5040503@rabbit.us> References: <4C972CB2.4030008@rabbit.us> <4C9CB87A.5040503@rabbit.us> Message-ID: On 24 September 2010 15:40, Peter Rabbitson wrote: > attempt insert > ?if exception - attempt select > ? if nothing found attempt insert again so you propagate the correct > exception while (1) { try to create, return on success try to update, return on success } From rabbit+dbic at rabbit.us Fri Sep 24 17:04:23 2010 From: rabbit+dbic at rabbit.us (Peter Rabbitson) Date: Fri Sep 24 17:04:29 2010 Subject: [Dbix-class] is it safe use find_or_create in "read committed" transaction level In-Reply-To: References: <4C972CB2.4030008@rabbit.us> <4C9CB87A.5040503@rabbit.us> Message-ID: <4C9CDA17.6070103@rabbit.us> Bill Crawford wrote: > On 24 September 2010 15:40, Peter Rabbitson wrote: > >> attempt insert >> if exception - attempt select >> if nothing found attempt insert again so you propagate the correct >> exception > > while (1) { > try to create, return on success > try to update, return on success > } Hehe, except that when you mix in exceptions it will get slightly more involved. Anyway - when you have a moment please give it a shot and throw us a patch. Cheers!