[Catalyst] RE: [sqlite] experiencing SQLite crash (Apache/mod_perl/Catalyst)

[Dami Laurent (PJ)]

>Hi all,
>
>I have a Perl Catalyst app using SQLite as storage. It has been running
>for several months without any problem.
>Now in the last few days we had several crashes of SQLite (file gets
>corrupted). This is just incomprehensible because all development and
>production engineers claim that nothing has changed in 
>application code,
>nor in Catalyst, DBD::Sqlite, Apache, or operating system 
>(Solaris), and
>we can't reproduce the problem in dev. So we don't even know where to
>start our investigations.
>
>Any hints, suggestions, related info, etc. would be welcome.
>
>The production server is running Apache 2.2.8 Solaris / mod_perl 2.04 /
>Perl 5.7.8 / Catalyst 5.7007 / DBD::SQlite 1.13.
>
>Thanks in advance, 
>
>	Laurent Dami
>


For info : we partially understood the problem.

The production server uses a private Apache authentication module
written in C.
That module was accidentally compiled with a debug flag and therefore
was sending
printf(..) statements to STDOUT (probably a very bad idea, instead of
going through Apache logging methods!). 
This went undetected for several months -- the printf were written God
knows where.

What happened in the last few days is that in some occasions the file
descriptor
for accessing SQLite was the same as the stdout used by C printf --
so some logging data was overwriting some blocks inside the SQLite
binary file!
No idea if this sharing of file descriptor is because of Apache or
mod_perl or 
DBD::Sqlite or still something else, nor why this sharing only started a
few
days ago  (maybe a difference in load of the server).

Anyway, the fix is of course to disable debug flag in that C module, so
that
it no longer prints to stdout.

Many thanks to people who responded to my previous call for help.

Best regards,

	Laurent Dami







-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20081016/56567961/attachment.htm