[Dbix-class] Re: Help with Titanic Number of Inserts

[leif.eriksen at hpa.com.au]

Keep careful notes - I reckon there's a good article in this...

L 

-----Original Message-----
From: rjr2 at cec.wustl.edu [mailto:rjr2 at cec.wustl.edu] 
Sent: Thursday, 25 January 2007 10:47 AM
To: dbix-class at lists.rawmode.org
Subject: Re: [Dbix-class] Re: Help with Titanic Number of Inserts

Hello Everyone,

Thank you all for your really helpful comments.  Using the batch loading tool of SQLite (someone mentioned the batch loading tools of Oracle and MySQL) we got a 720x speed improvement (from 30 days to 1 hour).

Definitely going to keep the fancy MySQL stuff (partitioned tables, clustering, archive storage engine) in mind.

Tomorrow we will be looking at selects instead of inserts ;-)

Wish us luck and thank you all again!

ryan


On Jan 24, 2007, at 3:47 PM, A. Pagaltzis wrote:

> * Lee Standen <nom at standen.id.au> [2007-01-24 08:50]:
>> Step 1:
>> Ditch SQLite - it's not optimized for large databases (to the best of 
>> my knowledge)
>
> I always hear people saying that, and none of them has ever been able 
> to substantiate it.
>
> What SQLite isn’t designed for is a lot of concurrent access, 
> particularly write access. It copes with huge databases just fine, 
> however.
>
> Regards,
> --
> Aristotle Pagaltzis // <http://plasmasturm.org/>
>
> _______________________________________________
> List: http://lists.rawmode.org/cgi-bin/mailman/listinfo/dbix-class
> Wiki: http://dbix-class.shadowcatsystems.co.uk/
> IRC: irc.perl.org#dbix-class
> SVN: http://dev.catalyst.perl.org/repos/bast/trunk/DBIx-Class/
> Searchable Archive: http://www.mail-archive.com/dbix- 
> class at lists.rawmode.org/


_______________________________________________
List: http://lists.rawmode.org/cgi-bin/mailman/listinfo/dbix-class
Wiki: http://dbix-class.shadowcatsystems.co.uk/
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/trunk/DBIx-Class/
Searchable Archive: http://www.mail-archive.com/dbix-class@lists.rawmode.org/

--
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.410 / Virus Database: 268.17.10/651 - Release Date: 24/01/2007
 

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.410 / Virus Database: 268.17.10/651 - Release Date: 24/01/2007
 
**********************************************************************
IMPORTANT
The contents of this e-mail and its attachments are confidential and intended
solely for the use of the individual or entity to whom they are addressed.  If
you received this e-mail in error, please notify the HPA Postmaster, postmaster at hpa.com.au,
then delete  the e-mail.
This footnote also confirms that this e-mail message has been swept for the
presence of computer viruses by Ironport. Before opening or using any
attachments, check them for viruses and defects.
Our liability is limited to resupplying any affected attachments.
HPA collects personal information to provide and market our services. For more
information about use, disclosure and access see our Privacy Policy at
www.hpa.com.au
**********************************************************************