People who know me would say that usually I'm the first one to back Red Hat based distro's. I'm a big fan of RPM, spent much of my youth playing with early Red Hat boxes, and usually don't tend to enjoy working with Debian... but this week has been some what of a gigantic black hole of SUCK. What began as a small set of updates and bug fixes has turned into a total nightmare.

A few weeks ago Red Hat released RHEL 5.2, the second maintenance release for their series 5 server. As many people know, these updates don't happen all that often and there are few individual packages released in the interim. The whole point of this is that the applications can be tested and deemed stable before being sent out to a production environment. Unlike Fedora where you receive daily updates of the latest bleeding edge, these releases are supposed to be tried and tested heavily. Then there's nss_ldap 253-12...

Following pretty standard procedures, we updated a few test machines to 5.2 before putting it on anything important, and as it turned out, I'm extremely glad I did! The first thing we noticed after the update was that upon boot up, almost all our services failed to start. What. The. Fuck. Almost nothing would run. Once I was able to get into the machine, just running bash was spewing bad code all over the screen and all our bash scripting was broken. It seemed like bash itself had sold its soul to the devil and then immediately commenced eating babies. Pipes were broken, return codes were in the thousands... absolute insanity.

Of course I immediately started poking around on the web and, WOW what a surprise, bug #448014 over at Red Hat's bugzilla. As it turns out, if you use LDAP, the upgrade from nss_ldap 253-5 => 253-12 completely fries your system. GREAT. What do we run just about everywhere??? Yes, that's right. LDAP. Good one guys! A little more reading in the bug informs us that apparently there has been a major regression in the code, and this issue should be considered a block for the 5.2 release, but it gets even better! They release 5.2 anyways! Awesome.

Needless to say, this problem completely crippled our whole system. Luckily, a patch was upstreamed and packaged by none other than the CentOS crew, but seriously. This should never have made it into a release... especially one designed as a bug *FIX*. Breaking all LDAP users is unacceptable. So, at this point we're out of the woods right?! Wrong.

Soon after patching I discovered that, yes, the immediate symptoms were gone, but that now we could not shutdown or reboot without causing a never ending segfault!! YAY!! :(

It's amazing how no one noticed any of this prior to signing off on nss_ldap 253-12 as stable. For the new problems I opened up bug #454675, but once again, downgrading to 253-5 (the 5.1 release) fixes the problems. *sigh* It's been a never ending bucket of FAIL for Red Hat Enterprise Linux this week, and as far as I'm concerned they need to start getting their shit together. People spend thousands on support contracts only to find out that the bug which was supposed to block a release, as designated by their own staff, was pushed out anyways.

This whole debacle just shows me how much CentOS is totally justified and the proper way to go.