Predicting server hardware failure with mcelog

Have you ever wanted to predict that a piece of hardware in your server was failing before it actually caused the server to crash?

Sure! We all do.

Over the past few months, I have been tracking the correlation between errors logged to the Machine Check Event Log (MCElog) and the hard crash of a server or application running on that server (mostly MySQL). So far, the correlation is about 90%. That is to say, about 9 times out of 10, there will be an error logged to the MCElog before the server actually crashes. It may take days or even weeks between the time of the logged error and the crash, but it will happen. We are now actively monitoring this log and replacing hardware (RAM and CPUs) which show errors before they actually fail which I thought was pretty cool, so I thought I would share how we are doing it.

On Debian, there is a package for the mcelog utility which will allow you to decode and display the kernel messages logged to /dev/mcelog Part of this package is a cron job which outputs the decoded contents of /dev/mcelog to /var/log/mcelog every 5 minutes:

*/5 * * * * root test -x /usr/sbin/mcelog -a ! -e /etc/mcelog-disabled && /usr/sbin/mcelog --ignorenodev --filter >> /var/log/mcelog

We modify this a little bit and add another cron job which rotates this log file on reboot:

@reboot root test -f /var/log/mcelog && mv /var/log/mcelog /var/log/mcelog.0

The reason we do this is because after a reboot, which is most likely a result of the hardware repair, we want to clear the active logfile (monitored by the nagios plugin below), so the alert will clear.  In case, however, the reboot was not part of the hardware maintenance, we still want to have a record of the hardware errors so we move the log file to mcelog.0.

We then have a simple nagios plugin which monitors /var/log/mcelog for errors:

#!/bin/bash

LOGFILE=/var/log/mcelog

if [ ! -f "$LOGFILE" ]
then
	echo "No logfile exists"
	exit 3
else
	ERRORS=$( grep -c "HARDWARE ERROR" /var/log/mcelog )
	if [ $ERRORS -eq 0 ]
	then
		echo "OK: $ERRORS hardware errors found"
		exit 0
	elif [ $ERRORS -gt 0 ]
	then
		echo "WARNING: $ERRORS hardware errors found"
		exit 1
	fi
fi

And thats pretty much it.  In just a few weeks we have caught about a dozen hardware faults before they led to server crashes.

Disclaimer: This only works when running a X86_64 kernel and YMMV.

WordPress Code Repository

We have decided to consolidate all of the small projects we have released into a single subversion repository. Previously these were spread across multiple domains and not very well publicized. We have setup a Trac instance as well to facilitate bug reports. There are 5 projects currently in the repository all of which we have used or are currently using at Automattic. Some of the projects, like Servermattic, are also being used elsewhere. All of these projects are obviously open source and are released under the GPL. Patches and feedback are welcome! We hope to release more of these soon. Thanks to Nikolay and Demitrious who have both contributed to the projects in the repository.

mod_auth_mysql and phpass

With the release of WordPress 2.5, there were some significant changes to the way passwords were stored in the database.  Prior to 2.5, passwords were stored as MD5 hashes.  While simple and easy, there were some security implications, so since 2.5, passwords are now salted and hashed using the phpass encryption library.  At Automattic we like to keep things simple, so we use the WordPress and bbPress user system for external authentication for things such as Trac and Subversion.  This allows us an effective and simple single sign on (SSO) solution for almost everything we do.  Unfortunately, the existing mod_auth_mysql apache module did not have support for the new password format.

Thanks to Nikolay, we now have the best of both worlds.  He has patched mod_auth_mysql to support phpass.  This means you can now have plug and play authentication against your WordPress blog or bbPress forum almost anywhere you can think of.  The patch allows automatic fallback to MD5 in case the user has not yet logged into WordPress and their password is still stored in the old format.  

Once the new module is loaded, you will just need to replace the following line in your apache configuration file.

OLD:AuthMySQLPwEncryption md5
NEW:AuthMySQLPwEncryption phpass

You can download the patched version here. It has been tested with Apache 2.2.3 and MySQL 4.1/5.0