[Dbix-class] Caching DBIx-Class Fixtures

[luke saunders]

On Wed, Jun 18, 2008 at 1:59 PM, Ovid <publiustemp-dbic at yahoo.com> wrote:
> --- Ash Berlin <ash at cpan.org> wrote:
>
>> First thing I'd suggest is to look at DBIx::Class::Fixtures and see
>> how close it comes to your needs. (I'm guessing you haven't looked at
>
>> it yet judging by your useperl post)
>
> You're right, I didn't know about that.
>
> Unfortunately, I'm having trouble with it.  I have something like this:
>
>  use PIPTest::Setup;
>  use TestPM::All;
>
>  my $setup  = PIPTest::Setup->new;
>  my $schema = $setup->init_db;
>  TestPM::All->load($schema);
>
> That particular snippet loads the 11 fixture classes we have so I
> figured I would test with that.  I managed to successfully get
> something dumped out with this:
>
>  my ($config_dir, $fixture_dir) = qw(
>    t/var/configs t/var/fixtures
>  );
>
>  my $fixtures = DBIx::Class::Fixtures->new({
>      config_dir => $config_dir,
>      debug      => 0,
>  });
>  $fixtures->dump({
>      all       => 1,
>      schema    => $schema,
>      directory => $fixture_dir,
>  });
>
> And that scattered 927 (holy crap!) fixture files around in the
> $config_dir directory.  Right off the bat, this looks problematic.
> With 14,000 tests and growing, I don't want each test program needing
> fixtures to have to read in a separate file for every object in a
> fixture, parse it and execute the code for it.  I/O issues are already
> killing us.  For those (admittedly few) tests which to read in all
> fixtures, I don't want them to have to read in 927 files, parse 'em,
> and shove 'em in the database.  Heck, with only 11 fixture classes, it
> looks like they'll each have to read an average of 84 fixture files to
> build the fixture.  I want them to read *one*.

As Drew said, the normal approach here when using
DBIx::Class::Fixtures is to create a MySQL dump of a pre-populated
database, and then at populate time use the dump rather than the raw
fixtures files (so read one file instead of many). You have to
subclass DBIx::Class::Fixtures in order to achieve this but it's very
simple - we can give you sample code if you need it.

> Then it gets worse.  In reading the populate method, it sort of looks
> like we drop all tables and have to recreate and populate them each
> time fixtures are used.  This model breaks one of our very aggressive
> performance strategies (http://tinyurl.com/55zlsv).  We don't drop
> databases or tables (see the link if you're curious) because it slows
> our test suite down even more.

Ditching the database and then recreating it for each test is the
right thing to do in the general case since previous tests might have
dropped tables, deleted/created rows and so on which would affect
subsequent tests. I see from the article that you have a
static/dynamic split for your tables so that the static tables are
created once then left untouched for all tests, but when/how do you
refresh the dynamic ones?

> I need this to be FAST.  FAST, FAST, FAST.  So I would *much* prefer
> that a single fixture class -- written out as proper DBIx::Class code
> which we already know -- can be marked as "cacheable" and I can cache
> all of its SQL into a file.  So "Test::Fixture::WakingTheDead" could
> generate a test_fixture_waking_the_dead.sql file which gets read in
> once and bypasses all all DBIx::Class code.

I think the main thing is, if you find the ability of
DBIx::Class::Fixtures to specify what data should be included in each
fixture set desirable then you can just customise the rest to suit
your needs.

Cheers,
Luke.