[Catalyst] Managing module regressions.

[Fred Moyer]

I meant to reply to this a month ago but haven't had the time until now.

On Fri, Jun 29, 2012 at 8:24 AM, Bill Moseley <moseley at hank.org> wrote:
> This is a "how do others do this?" post.
>
> In your large (or even not so large) apps I assume at times you experience
> dependency regressions.  My question is do you manage it on a case-by-case
> basis (simply install an older version or fix internally) or is it such a
> significant issue that you have a system for managing modules outside of
> CPAN?

I've been working with a few apps where we haven't experienced a lot
of dependency regressions per se, but have been dealing with several
modules that had major api changes. We have had to spend a significant
amount of time on maintenance in the apps to upgrade the code to
handle these updated module apis.

> The reason I ask is because at work we are considering managing different
> stacks of CPAN and in-house dependencies -- so maybe "dev", "testing", and
> "production/staging" stacks all in separate private CPAN mirrors.  And now
> multiply that times the number of different apps we work on.

I would tend to stay away from this approach, as we have managed
several apps which use different versions of modules. We have a few
different development teams where developers upgrade modules without
any set schedule. Some of them are writing modules which are used by
all the apps, and the unannounced api changes in those modules have
had cascading effects which have slowed down deployment a lot. Only 2
of these developers have modules on CPAN, so I think the others
haven't experienced the real world pain that comes with rapid api
changes and are less cautious with their coding.

My feedback here would be try to stick with a fixed set of modules as
much as possible, no matter how tempting a new module looks.


> My experience is that even with the very large number of dependencies in a
> Catalyst app that it's pretty rare to have a regression.   It happens, sure,
> and when it happens just deal with it.  (And thinking of our own code it's
> typically not a regression in a module but a fix in a module where our code
> was depending on some broken behavior).
>
> So, the question is: does anyone else find it necessary to manage
> dependencies as I described above?  And if so, what is your process?

We looked at implementing Pinto for module version management, but
that effort seemed to have never really taken off. I don't think the
part of the team implementing it dealt with the pain the rest of us
were experiencing on a day to day basis.


> Another argument that is floated around is we don't want to upgrade
> dependencies often because of potential new bugs.  That seems a bit silly to
> me because it's ignoring known bug fixes for the chance that there might
> some unknown new bug.    (Yes, we have apps running Catalyst from 2010!)

This has been a big pain point for us, enough that the build engineer
I work with to deploy the rpms with is now averse to upgrading
modules. The config module we developed in house changed its api 3
times in one year. A couple of the other core modules changed
significantly, causing weeks/months of delays. We had to upgrade to
Catalyst 5.9 and Moose 2.0 recently, and that was quite painful (this
is not a dig at either of those modules, just a reflection of our
upgrade experience).

We ran into this issue with the Moose 2 upgrade - took a few days to
track down. http://freebsd.so14k.com/problems_with_perl_catalyst.shtml

So while we haven't run into unknown new bugs, the cost of upgrading
modules on an ad hoc schedule has been large, and outweighing most
business value brought by the upgraded modules. I'm not saying that
you shouldn't upgrade modules when they are available. But such
upgrades should be scheduled apart from feature releases, and not done
ad hoc. This can be difficult when you have a loosely organized team
and developers want to try out the latest release of a module. Of
course, this is all in the context of an application that generates
revenue and has a user base that expects the app to remain largely bug
free. If I'm developing something that has an alpha user base, I'll
always go for the most recent release.

Also, these experiences are that of a large team developing large apps
to hundreds of servers. The experience of one developer deploying to a
small environment will certainly be different.