108.2 Lesson 1

Because logs are variable data, they are normally found in /var/log. Roughly speaking, they can be classified into system logs and service or program logs.

Let us see some system logs and the information they keep:

Now let us see a few examples of service logs:

To read log files, first ensure you are the root user or have reading permissions on the file. You can use a variety of utilities such as:

As you may have noticed, the output is printed in the following format:

There are a few examples in which logs are not text, but binary files and — consequently — you must use special commands to parse them:

The following process illustrates how a message is written to a log file:

rsyslog configuration file is /etc/rsylog.conf (in some distributions you can also find configuration files in /etc/rsyslog.d/). It is normally divided into three sections: MODULES, GLOBAL DIRECTIVES and RULES. Let us have a look at them by exploring the rsyslog.conf file in our Debian GNU/Linux 10 (buster) host — you can use sudo less /etc/rsyslog.conf to do so.

MODULES includes module support for logging, message capability, and UDP/TCP log reception:

GLOBAL DIRECTIVES allow us to configure a number of things such as logs and log directory permissions:

RULES is where facilities, priorities and actions come in. The settings in this section tell the logging daemon to filter messages according to certain rules and log them or send them where required. To understand these rules, we should first explain the concepts of rsyslog facilities and priorities. Each log message is given a facility number and keyword that are associated with the Linux internal subsystem that produces the message:

Furthermore, each message is assigned a priority level:

Here is an excerpt of rsyslog.conf from our Debian GNU/Linux 10 (buster) system which including some sample rules:

The rule format is as follows: <facility>.<priority> <action>

The <facility>.<priority> selector filters messages to match. Priority levels are hierarchically inclusive, which means rsyslog will match messages of the specified priority and higher. The <action> shows what action to take (where to send the log message). Here are a few examples for clarity:

Regardless of their priority (*), all messages from the auth or authpriv facilities will be sent to /var/log/auth.log.

All messages — irrespective of their priority (*) — from all facilities (*) — discarding those from auth or authpriv (hence the .none suffix) — will be written to /var/log/syslog (the minus sign (-) before the path prevents excessive disk writes). Note the semicolon (;) to split the selector and the comma (,) to concatenate two facilities in the same rule (auth,authpriv).

Messages from the mail facility with a priority level of error or higher (critical, alert or emergency) will be sent to /var/log/mail.err.

Messages from all facilities with the debug priority and no other (=) will be written to /var/log/debug — excluding any messages coming from the auth, authpriv, news and mail facilities (note the syntax: ;\).

To explain this topic we are going to add a new host to our setup. The layout is as follows:

Let us start by configuring the server. First of all, we make sure that rsyslog is up and running:

openSUSE ships with a dedicated configuration file for remote logging: /etc/rsyslog.d/remote.conf. Let us enable receiving messages from clients (remote hosts) via TCP. We must uncomment the lines which load the module and start the TCP server on port 514:

Once this is done, we must restart the rsyslog service and check that the server is listening on port 514:

Next, we should open the ports in the firewall and reload the configuration:

Logs are rotated on a regular basis, which serves two main purposes:

The utility in charge of log rotation (or cycling) is logrotate and its job includes actions such as moving log files to a new name, archiving and/or compressing them, sometimes emailing them to the system administrator and eventually deleting them as they grow old. There are various conventions for naming these rotated log files (adding a suffix with the date to the filename, for example); however, simply adding a suffix with an integer is the common practice:

Let us now explain what will happen in the next log rotation:

Note how — according to logrotate directives that you will see shortly — the three older log files are compressed, whereas the two most recent ones are not. Also, we will be retaining the logs from the last 4-5 weeks. To read messages that are 1 week old, we will consult messages.1 (and so on).

logrotate is run as an automated process or cron job daily through the script /etc/cron.daily/logrotate and reads the configuration file /etc/logrotate.conf. This file includes some global options and is well commented with each option introduced by a brief explanation of its purpose:

As you can see, the configuration files in /etc/logrotate.d for specific packages are also included. These files contain — for the most part — local definitions and specify the logfiles to rotate (remember, local definitions take precedence over global ones, and later definitions override earlier ones). What follows is an excerpt of a definition in /etc/logrotate.d/rsyslog:

As you can see, every directive is separated from its value by whitespace and/or an optional equals sign (=). The lines between postrotate and endscript must appear on lines by themselves, though. The explanation is as follows:

Since the kernel generates several messages before rsyslogd becomes available on boot, a mechanism to register those messages becomes necessary. This is where the kernel ring buffer comes into play. It is a fixed-size data structure and — therefore — as it grows with new messages, the oldest will disappear.

The dmesg command prints the kernel ring buffer. Because of the buffer’s size, this command is normally used in combination with the text filtering utility grep. For instance, to search for messages related to Universal Serial Bus devices: