[Dbix-class] Help with Titanic Number of Inserts

[Matt S Trout]

On 24 Jan 2007, at 01:01, Ryan Richt wrote:

> Hey Guys and Gals...
>
> 	We are just starting a DBIx::Class project here at the Washington  
> University Genome Sequencing Center in St. Louis, and  
> __of__course__ we choose DBIx::Class over all that other stuff ;-).
>
> Anyway, we need to make > 112,000,000 inserts per day.  I _know_  
> that DBIx::Class is not concerned with performance, but with the  
> (wonderful) design/interface/features.
>
> However, other than wrapping it in a transaction, do you have any  
> other tips on making this manageable?  Currently this would take  
> 130 DAYS to insert w/o transactions, and something like 30 DAYS  
> with transactions.
>
> (As far as setup goes, this is currently in SQLite and soon to be  
> Oracle once SQLite explodes, testing on a newer dual processor   
> 3gig ram machine, and could deploy on anything from 1000+ node  
> cluster or 50 node cluster of Opertons or 5 x 100gig ram itanium  
> quad processor machines, if that could help us in any way!).

Try grabbing http://dev.catalyst.perl.org/repos/bast/branches/DBIx- 
Class-current and trying populate in void context on that; it should  
use a much, much faster insertion approach.

If you're worried about your production schedule, note I'm planning  
to start shipping 07999_XX releases this week so 08000 should be out  
fairly soon now.

-- 
Matt S Trout, Technical Director, Shadowcat Systems Ltd.
Offering custom development, consultancy and support contracts for  
Catalyst,
DBIx::Class and BAST. Contact mst (at) shadowcatsystems.co.uk for  
details.
+ Help us build a better perl ORM: http://dbix- 
class.shadowcatsystems.co.uk/ +