Centralized remote logging can be directly utilized with Linux's built-in syslog logging facilities.

An important part of Linux system administration is dealing with system logs. Even if you aren't watching or reading logs on a daily basis, when you're troubleshooting problems, logs are usually the very first thing you turn to when troubles arise. Having logs in one centralized location on a network simplifies the task of locating them, plus gives you one place to create backups of logs if desired. The default Linux logging facility, syslog, provides you with the basic tools necessary to send logs to a remote server for collection.

The syslog Daemon

The syslog daemon is a standard tool that comes with all Linux flavors. It uses UDP network protocol and runs on port 514. Virtually all log files are found under /var/log on Linux systems, although in rare circumstances you can find logs elsewhere. Red Hat, CentOS, Debian, and SuSE are just a few examples of where configuration and use are virtually identical on all Linux variants. Before you start editing the syslog configuration files, let's spend a few minutes understanding basic syslog terms, something often overlooked. Open up /etc/syslog.conf with your favorite text editor to see the default logging options.

The first field is called a facility and basically is the name of the system process where the log will be coming from. The following lists some facility names with their system process definitions. More do exist, but these are the most commonly seen facilities.

• kern—All kernel-related messages
• authpriv (formerly auth)—Security-related messages
• mail—Mail system messages
• cron—Clock and scheduled job messages
• daemon—System daemon messages
• user—User processes
• local0 to local7—User-defined facilities

The next piece configured directly after the facility name is the priority-level options. This list explains the various levels of priority logging.

• none—Used to disable a facility
• info—Information message
• warning (formerly warn)—Warning messages
• debug—Debug type of message, usually verbose
• crit—Critical messages
• err (formerly error)—Error-related messages
• alert—Immediate action is requested
• emerg (formerly panic)—Extremely problematic messages

The last configuration parameter is the actual location to send the log files to. To send to a file, an option would be listed as something like /var/log/message or /var/log/any_file_name. The asterisk (*) option is used as a placeholder with a meaning of "all." The default authpriv definition listed in /etc/syslog.conf is shown below.

authpriv.*              /var/log/secure

This definition states that the authpriv facility and all priority log-level messages should be sent to the file /var/log/secure. Another more complicated line is shown here.

*.info;mail.none;authpriv.none;cron.none        /var/log/messages

By using the * placeholder and some .none definitions, you can include and exclude facilities and priorities. This statement means that syslog selects all informational messages except those from mail, authpriv, and cron facilities and sends the logs to /var/log/messages.

These are just the basics and are meant to get you started, so if want a lot more detailed information, do some Web searches for syslog. Also, you can visit the syslog Web site, which has some excellent resources.

Enable Remote Logging

Syslog's default configuration has the remote logging features turned off completely. Open up the syslog daemon configuration file at /etc/sysconfig/syslog and add the remote logging feature. The option -r is used to specify remote logging. This option need only be changed on the central syslog server. All of the syslog clients will point to this one server.

SYSLOGD_OPTIONS="-r -m 0"

Once the option is changed, restart the syslog daemon for the changes to take effect. On Red Hat and CentOS systems, the service command is used. On many other systems, the path to the syslog executable can be used.

# service syslog restart

# /etc/init.d/syslog restart

Pointing Clients to the Remote Server

On the client machines from where you want the logs sent to the remote syslog server, open the syslog configuration file at /etc/syslog.conf. Everywhere that there is a path to a file or on every logging facility that you want sent to the remote server, uncomment out the default and add either the IP address or the DNS name of the remote syslog server.

#authpriv.*            /var/log/secure

authpriv.*              @192.168.0.1

After the changes are made, restart the syslog server the same as you did above on the syslog server. Once restarted, the logs that you pointed to the remote syslog server should start to show up on the remote server's file. In the above example, the authpriv facility should have its logs passed on to the IP address of 192.168.0.1. Since SSH runs under this facility, attempt to log in to the client from a new machine, while viewing the logs on the remote syslog server.

On the remote syslog server, run the tail command to view the log in action.

# tail -f /var/log/secure

Now, from a new machine, attempt to log in using SSH to the remote client you just set up to send logs to the remote syslog server.

# ssh user@remote_client

With the log file open on the remote syslog server, you should see some activity based upon either an IP address or a DNS name from where you're logging into. Once you exit the SSH session, you should see a disconnect message as well.

Sep 14 11:26:13 remote_client sshd[9072]: pam_unix(sshd:session): session opened for user root by (uid=0)

Sep 14 11:26:49 remote_client sshd[9072]: pam_unix(sshd:session): session closed for user root

Firewall Rules and Security

You may run into some firewall rules problems from time to time. If you're going to use remote syslog features on a trusted local network, chances are there won't be any firewalls in the road. If, however, there's a firewall on the remote syslog server, you have options to add (by IP addresses or by subnets) to trust the traffic being passed to the server. You should always consult your network administrator to see what the policies are at your location.

If you have a concern about security, you may want to look into rsyslog. The rsyslog tool is the new and improved version of syslog. Its main focus is on reliability and security, and it allows transport of logs over TCP using SSL and TLS security protocols.

Remote Troubleshooting

Something as simple as a centralized place to store logs may seem trivial, but it's very important. Having one place to start troubleshooting can save you time. Plus, if a machine or server has become completely unresponsive, the logs leading up to the crash won't be on the troubled hardware, and instead will be on a remote location for you to view.