108.2 Lesson 2

systemd operates on units. A unit is any resource that systemd can manage (e.g. network, bluetooth, etc.). Units — in turn — are governed by unit files. These are plain text files which live in /lib/systemd/system and include the configuration settings — in the form of sections and directives — for a particular resource to be managed. There are a number of unit types: service, mount, automount, swap, timer, device, socket,path, timer, snapshot, slice, scope and target. Thus, every unit file name follows the pattern <resource_name>.<unit_type> (e.g. reboot.service).

A target is a special type of unit which resembles the classic runlevels of SysV Init. This is because a target unit brings together various resources to represent a particular system state (e.g. graphical.target is similar to runlevel 5, etc.). To check the current target in your system, use the systemctl get-default command:

On the other hand, targets and runlevels differ in that the former are mutually inclusive, whereas the latter are not. Thus a target can bring up other targets — which is not possible with runlevels.

journalctl is the utility that you use to query systemd journal. You have to either be root or use sudo to invoke it. If queried without options, it will print the entire journal chronologically (with the oldest entries listed first):

You can make more specific queries by using a number of switches:

The journal allows you to filter log data by different criteria:

Just like how the logger command is used to send messages from the command line to the system log (as we saw in the previous lesson), the systemd-cat command serves a similar — but more well-rounded — purpose with the system journal. It allows us to send standard input (stdin), output (stdout) and error (stderr) to the journal.

If invoked with no parameters, it will send everything it reads from stdin to the journal. Once you are done, press Ctrl+C:.

If it is passed the output of a piped command, this will be sent to the journal too:

If followed by a command, that command’s output will also be sent to the journal — together with stderr (if any):

There is also the possibility of specifiying a priority level with the -p option:

Refer to systemd-cat man page to learn about its other options.

To see the last four lines in the journal:

As mentioned previously, you have three options when it comes to the location of the journal:

The default behaviour is as follows: if /var/log/journal/ does not exist, logs will be saved in a volatile way to a directory in /run/log/journal/ and — therefore — lost at reboot. The name of the directory — the /etc/machine-id — is a newline-terminated, hexadecimal, 32-character, lowercase string:

If you try to read it with less you will get a warning, so instead use the command journalctl:

If /var/log/journal/ exists, logs will be stored persistently there. Should this directory be deleted, systemd-journald would not recreate it but write to /run/log/journal instead. As soon as we create /var/log/journal/ again and restart the daemon, persistent logging will be restablished:

On top of what we have just mentioned, the way the journal daemon deals with log storage can be changed after installation by tweaking its configuration file: /etc/systemd/journald.conf. The key option is Storage= and can have the following values:

For instance, to have systemd-journald create /var/log/journal/ and switch to persistent storage, you would edit /etc/systemd/journald.conf and set Storage=persistent, save the file and restart the daemon with sudo systemctl restart systemd-journald. To ensure the restart went flawlessly, you can always check the daemon’s status:

Logs are saved in journal files whose file names end with .journal or .journal~ and are located in the appropriate directory (/run/log/journal or /var/log/journal as configured). To check how much disk space is currently being occupied by journal files (both archived and active), use the --disk-usage switch:

systemd logs default to a maximum of 10% of the size of the filesystem where they are stored. For instance on a 1GB filesystem they will not take up more than 100MB. Once this limit is reached, old logs will start to disappear to stay near this value.

However, size limit enforcement on stored journal files can be managed by tweaking a series of configuration options in /etc/systemd/journald.conf. These options fall into two categories depending on the filesystem type used: persistent (/var/log/journal) or in-memory (/run/log/journal). The former uses options that are prefixed with the word System and will only apply if persistent logging is properly enabled and once the system is fully booted up. The option names in the latter start with the word Runtime and will apply in the following scenarios:

Apart from size-based deletion and rotation of log messages, systemd-journald also allows for time-based criteria by using the following two options: MaxRetentionSec= and MaxFileSec=. Refer to the manual page of journald.conf for more information about these and other options.

As a system administrator, you may find yourself in a situation where you need to access journal files on the hard drive of a faulty machine through a rescue system (a bootable CD or USB key containing a live Linux distribution).

journalctl looks for the journal files in /var/log/journal/<machine-id>/. Because the machine IDs on the rescue and faulty systems will be different, you must use the following option:

Thus, it is necessary that you mount the faulty system’s rootfs (/dev/sda1) on the rescue system’s filesystem and proceed to read the journal files like so:

Other options that can be useful in this scenario are:

See journalctl man page for more information.

Log data from the journal can be made available to a traditional syslog daemon by: