As with any server installation, a massive feature (often overlooked) is the need for good server side logging. That is, the ability to work out what is being used, by whom and for how much. Even more important is to be able to find out what went wrong, in case of any (ahem) bugs.

We already have procedures and systems in place for our Windows installations, but as this is the first time we’ve deployed a massively available server on Linux (as part of Goldfish). As such, we needed to make sure we can backtrack any problems correctly and efficiently.

As a result, I thought I’d dedicate a post to Linux logging facilities, in their most simple state: Enter syslog.

Syslog is the Linux System Logger and is exceedingly easy to use from a C++ perspective: open the log (optional), write to the log and close the log when the application exits (which in our case, is never).

Using C++ syslog:

#include <syslog.h>
openlog(“goldfish”, LOG_PID|LOG_CONS|LOG_NDELAY, LOG_LOCAL0);
…
syslog(LOG_WARNING, “Down to %d free bytes of RAM”, ramLeft);

Notice how in openlog() we specified a facility code of LOCAL0? This becomes important later when we configure a custom log.

On Ubuntu systems (which is what we use for development), you can easily see the log messages in the Log File Viewer (System -> Administration -> Log File Viewer), updating in realtime.

However, to generate a ‘goldfish.log’, a small configuration change has to be made.

In /etc/rsysconf.d/ create a file called ‘goldfish.conf’ and add the following text:

local0.* -/var/log/goldfish.log

This means that anything arriving at the system log using facility code LOCAL0, it should get added to the goldfish.log. Since we know that nothing else is logging toLOCAL0 this means that only goldfish messages arrive there.

Note that we could have chosen any number from 0 to 7, eg LOCAL7, in case of a conflict with some other application.

Note also that the ‘-’ near /var/log/goldfish.log indicates to enable write-behind caching of the log, meaning that the log is not immediately written out to, just periodically. This reduces the hard drive thrashing itself, while it tries to save the logs immediately when a new message is received.

So, you may ask, “Ok, so now we have a log file being generated,” won’t that just fill up the hard drive and then fail in a cataclysmic manner? Not so, there is generally a process called ‘logrotate’ which knows exactly what to do, without any fuss.

You may also ask, “So, how do I watch the log”? Simply use the ‘tail’ command:

tail -f /var/log/goldfish.log