[Catalyst] OT: Testing (renamed, was "Is it normal to have lots of trouble installing catalyst?")

[Brandon Black]

On 2/8/06, apv <apv at sedition.com> wrote:
> On Wednesday, February 8, 2006, at 12:39  PM, Brandon Black wrote:
> > But if there are no effective tests without it, then removing it from
> > the dependencies and making it an optional/skippable test means that
> > the majority of users installing it will not do any effective testing.
> >  Pretty much nobody chooses to go out of their way to get a test to
> > run, at least not among normal users.  Whatever tests don't run
> > automatically just don't get run for most installations.
>
> By extension of this thinking, every possible package that is supported
> should be forced to be installed for testing. SQLite is not the same as
> Oracle
> or Pg or MySQL or ... So they should all be installed or the package
> should
> fail tests.
>
> It seems you're already really in agreement with Perrin's point (which I
> think is entirely correct) that some testing which might actually be
> crucial
> to knowing if the code will run correctly is okay to skip b/c it's
> unfriendly
> (and unreasonable) to force the user through hoops that provide the
> illusion of bullet proof code.
>
> Ideally, the tests should optionally support all supported DB backends
> and
> run against those that user chooses or can be automatically detected.
> I am not implying this is easy or has to happen. And I'm not denigrating
> the fabulous work you're doing on these packages. Just saying that this
> kind of testing would be ideal.
>

True enough.  Part of the problem in this particular case is that for
most DB backends, we require a user-supplied username, password, and
database name, aside from having the actual database vendor software
itself installed, and there doesn't seem to be a way around that.

People *can* test on all installed DB backends, if they set the
environment variables for those things.  SQLite is the exception, as
we can create arbitrary databases in the test directory with no
special input requirements from the user.

Still, testing even a single DB backend (SQLite) provides considerably
more test coverage for the code in question than not testing any
backends at all in the default "no user interaction" case where SQLite
was not a prereq.

-- Brandon