[Dbix-class] Help with Titanic Number of Inserts

[Matt S Trout]

On 24 Jan 2007, at 01:01, Ryan Richt wrote:

> Hey Guys and Gals...
>
> 	We are just starting a DBIx::Class project here at the Washington  
> University Genome Sequencing Center in St. Louis, and  
> __of__course__ we choose DBIx::Class over all that other stuff ;-).
>
> Anyway, we need to make > 112,000,000 inserts per day.  I _know_  
> that DBIx::Class is not concerned with performance, but with the  
> (wonderful) design/interface/features.
>
> However, other than wrapping it in a transaction, do you have any  
> other tips on making this manageable?  Currently this would take  
> 130 DAYS to insert w/o transactions, and something like 30 DAYS  
> with transactions.
>
> (As far as setup goes, this is currently in SQLite and soon to be  
> Oracle once SQLite explodes, testing on a newer dual processor   
> 3gig ram machine, and could deploy on anything from 1000+ node  
> cluster or 50 node cluster of Opertons or 5 x 100gig ram itanium  
> quad processor machines, if that could help us in any way!).

While I think about it, we've been working towards code that could be  
used for multi-master setups to spread query load - if that's  
something you'd find useful please start a fresh thread covering what  
requirements you'd have for that, I guess you guys would make a good  
test case for this and I can probably spend some time on this under  
the aegis of Shadowcat given the marketing impact of being able to  
mention you guys are using DBIx::Class :)

-- 
Matt S Trout, Technical Director, Shadowcat Systems Ltd.
Offering custom development, consultancy and support contracts for  
Catalyst,
DBIx::Class and BAST. Contact mst (at) shadowcatsystems.co.uk for  
details.
+ Help us build a better perl ORM: http://dbix- 
class.shadowcatsystems.co.uk/ +