Linux Professional Institute Learning Logo.
Skip to main content
  • Home
    • All Resources
    • LPI Learning Materials
    • Become a Contributor
    • Publishing Partners
    • Become a Publishing Partner
    • About
    • FAQ
    • Contributors
    • Roadmap
    • Contact
  • LPI.org
108.2 Lesson 2
Topic 105: Shells and Shell Scripting
105.1 Customize and use the shell environment
  • 105.1 Lesson 1
  • 105.1 Lesson 2
  • 105.1 Lesson 3
105.2 Customize or write simple scripts
  • 105.2 Lesson 1
  • 105.2 Lesson 2
Topic 106: User Interfaces and Desktops
106.1 Install and configure X11
  • 106.1 Lesson 1
106.2 Graphical Desktops
  • 106.2 Lesson 1
106.3 Accessibility
  • 106.3 Lesson 1
Topic 107: Administrative Tasks
107.1 Manage user and group accounts and related system files
  • 107.1 Lesson 1
  • 107.1 Lesson 2
107.2 Automate system administration tasks by scheduling jobs
  • 107.2 Lesson 1
  • 107.2 Lesson 2
107.3 Localisation and internationalisation
  • 107.3 Lesson 1
Topic 108: Essential System Services
108.1 Maintain system time
  • 108.1 Lesson 1
  • 108.1 Lesson 2
108.2 System logging
  • 108.2 Lesson 1
  • 108.2 Lesson 2
108.3 Mail Transfer Agent (MTA) basics
  • 108.3 Lesson 1
108.4 Manage printers and printing
  • 108.4 Lesson 1
Topic 109: Networking Fundamentals
109.1 Fundamentals of internet protocols
  • 109.1 Lesson 1
  • 109.1 Lesson 2
109.2 Persistent network configuration
  • 109.2 Lesson 1
  • 109.2 Lesson 2
109.3 Basic network troubleshooting
  • 109.3 Lesson 1
  • 109.3 Lesson 2
109.4 Configure client side DNS
  • 109.4 Lesson 1
Topic 110: Security
110.1 Perform security administration tasks
  • 110.1 Lesson 1
110.2 Setup host security
  • 110.2 Lesson 1
110.3 Securing data with encryption
  • 110.3 Lesson 1
  • 110.3 Lesson 2
  1. Topic 108: Essential System Services
  2. 108.2 System logging
  3. 108.2 Lesson 2

108.2 Lesson 2

Certificate:

LPIC-1

Version:

5.0

Topic:

108 Essential System Services

Objective:

108.2 System logging

Lesson:

2 of 2

Introduction

With the general adoption of systemd by all major distributions, the journal daemon (systemd-journald) has become the standard logging service. In this lesson we will discuss how it operates and how you can use it to do a number of things: query it, filter its information by different criteria, configure its storage and size, delete old data, retrieve its data from a rescue system or filesystem copy and — last but not least — understand its interaction with rsyslogd.

Basics of systemd

First introduced in Fedora, systemd has progressively replaced SysV Init as the de facto system and service manager in most major Linux distributions. Amongst its strenghts are the following:

  • Ease of configuration: unit files as opposed to SysV Init scripts.

  • Versatile management: apart from daemons and processes, it also manages devices, sockets and mount points.

  • Backward compatibility with both SysV Init and Upstart.

  • Parallel loading during boot-up: services are loaded in parallel as opposed to Sysv Init loading them sequentially.

  • It features a logging service called the journal that presents the following advantages:

    • It centralizes all logs in one place.

    • It does not require log rotation.

    • Logs can be disabled, loaded in RAM or made persistent.

Units and Targets

systemd operates on units. A unit is any resource that systemd can manage (e.g. network, bluetooth, etc.). Units — in turn — are governed by unit files. These are plain text files which live in /lib/systemd/system and include the configuration settings — in the form of sections and directives — for a particular resource to be managed. There are a number of unit types: service, mount, automount, swap, timer, device, socket,path, timer, snapshot, slice, scope and target. Thus, every unit file name follows the pattern <resource_name>.<unit_type> (e.g. reboot.service).

A target is a special type of unit which resembles the classic runlevels of SysV Init. This is because a target unit brings together various resources to represent a particular system state (e.g. graphical.target is similar to runlevel 5, etc.). To check the current target in your system, use the systemctl get-default command:

carol@debian:~$ systemctl get-default
graphical.target

On the other hand, targets and runlevels differ in that the former are mutually inclusive, whereas the latter are not. Thus a target can bring up other targets — which is not possible with runlevels.

Note

An explanation of how systemd units work is beyond the scope of this lesson.

The System Journal: systemd-journald

systemd-journald is the system service which takes care of receiving logging information from a variety of sources: kernel messages, simple and structured system messages, standard output and standard error of services as well as audit records from the kernel audit subsystem (for further details see the manual page for systemd-journald). Its mission is that of creating and maintaining a structured and indexed journal.

Its configuration file is /etc/systemd/journald.conf and — as with any other service — you can use the systemctl command to start it, restart it, stop it or — simply — check its status:

root@debian:~# systemctl status systemd-journald
 systemd-journald.service - Journal Service
   Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
   Active: active (running) since Sat 2019-10-12 13:43:06 CEST; 5min ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 178 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/systemd-journald.service
           └─178 /lib/systemd/systemd-journald
(...)

Configuration files of the type journal.conf.d/*.conf — which can include package-specific configurations — are also possible (consult the manual page of journald.conf to learn more).

If enabled, the journal can be stored either persistently on disk or in a volatile manner on a RAM-based filesystem. The journal is not a plain text file, it is binary. Thus, you cannot use text parsing tools such as less or more to read its contents; the journalctl command is used instead.

Querying the Journal Content

journalctl is the utility that you use to query systemd journal. You have to either be root or use sudo to invoke it. If queried without options, it will print the entire journal chronologically (with the oldest entries listed first):

root@debian:~# journalctl
-- Logs begin at Sat 2019-10-12 13:43:06 CEST, end at Sat 2019-10-12 14:19:46 CEST. --
Oct 12 13:43:06 debian kernel: Linux version 4.9.0-9-amd64 (debian-kernel@lists.debian.org) (...)
Oct 12 13:43:06 debian kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-9-amd64 root=UUID=b6be6117-5226-4a8a-bade-2db35ccf4cf4 ro qu
(...)

You can make more specific queries by using a number of switches:

-r

The messages of the journal will be printed in reverse order:

root@debian:~# journalctl -r
-- Logs begin at Sat 2019-10-12 13:43:06 CEST, end at Sat 2019-10-12 14:30:30 CEST. --
Oct 12 14:30:30 debian sudo[1356]: pam_unix(sudo:session): session opened for user root by carol(uid=0)
Oct 12 14:30:30 debian sudo[1356]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/journalctl -r
Oct 12 14:19:53 debian sudo[1348]: pam_unix(sudo:session): session closed for user root
(...)
-f

It will print the most recent journal messages and keep printing new entries as they are appended to the journal — much like tail -f:

root@debian:~# journalctl -f
-- Logs begin at Sat 2019-10-12 13:43:06 CEST. --
(...)
Oct 12 14:44:42 debian sudo[1356]: pam_unix(sudo:session): session closed for user root
Oct 12 14:44:44 debian sudo[1375]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/journalctl -f
Oct 12 14:44:44 debian sudo[1375]: pam_unix(sudo:session): session opened for user root by carol(uid=0)

(...)
-e

It will jump to the end of the journal so that the latest entries are visible within the pager:

root@debian:~# journalctl -e
(...)
Oct 12 14:44:44 debian sudo[1375]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/journalctl -f
Oct 12 14:44:44 debian sudo[1375]: pam_unix(sudo:session): session opened for user root by carol(uid=0)
Oct 12 14:45:57 debian sudo[1375]: pam_unix(sudo:session): session closed for user root
Oct 12 14:48:39 debian sudo[1378]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/journalctl -e
Oct 12 14:48:39 debian sudo[1378]: pam_unix(sudo:session): session opened for user root by carol(uid=0)
-n <value>, --lines=<value>

It will print the value most recent lines (if no <value> is specified, it defaults to 10):

root@debian:~# journalctl -n 5
(...)
Oct 12 14:44:44 debian sudo[1375]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/journalctl -f
Oct 12 14:44:44 debian sudo[1375]: pam_unix(sudo:session): session opened for user root by carol(uid=0)
Oct 12 14:45:57 debian sudo[1375]: pam_unix(sudo:session): session closed for user root
Oct 12 14:48:39 debian sudo[1378]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/journalctl -e
Oct 12 14:48:39 debian sudo[1378]: pam_unix(sudo:session): session opened for user root by carol(uid=0)
-k, --dmesg

Equivalent to using the dmesg command:

root@debian:~# journalctl -k
-- Logs begin at Sat 2019-10-12 13:43:06 CEST, end at Sat 2019-10-12 14:53:20 CEST. --
Oct 12 13:43:06 debian kernel: Linux version 4.9.0-9-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18
Oct 12 13:43:06 debian kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-9-amd64 root=UUID=b6be6117-5226-4a8a-bade-2db35ccf4cf4 ro qu
Oct 12 13:43:06 debian kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Oct 12 13:43:06 debian kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
(...)
Navigating and Searching Through the Journal

You can navigate the journal’s outuput with:

  • PageUp, PageDown and arrow keys to move up, down, left and right.

  • > to go to the end of the output.

  • < to go to the beginning of the output.

You can search for strings both forward and backward from your current view position:

  • Forward search: Press / and enter the string to be searched, then press Enter.

  • Backward search: Press ? and enter the string to be searched, then press Enter.

To navigate through matches in the searches, use N to go to the next occurrence of the match and Shift+N to go to the previous one.

Filtering the Journal Data

The journal allows you to filter log data by different criteria:

Boot number
--list-boots

It lists all available boots. The output consists of three columns; the first one specifies the boot number (0 refers to the current boot, -1 is the previous one, -2 the one prior to the previous one and so on); the second column is the boot ID; the third shows the time stamps:

root@debian:~# journalctl --list-boots
 0 83df3e8653474ea5aed19b41cdb45b78 Sat 2019-10-12 18:55:41 CEST—Sat 2019-10-12 19:02:24 CEST
-b, --boot

It shows all messages from the current boot. To see log messages from previous boots, just add an offset parameter as explained above. For example, to have messages from the previous boot printed, you will type journalctl -b -1. Remember, though, that to recover information from previous logs, persistence of the journal must be enabled (you will learn how to do so in the next section):

root@debian:~# journalctl -b -1
Specifying boot ID has no effect, no persistent journal was found
Priority
-p

Interestingly enough, you can also filter by severity/priority with the -p option:

root@debian:~# journalctl -b -0 -p err
-- No entries --

The journal informs us that — so far — there have not been any messages with a priority of error (or above) from the current boot. Note: -b -0 can be omitted when referring to the current boot.

Note

Refer to the previous lesson for a complete list of all syslog severities (aka priorities).

Time Interval

You can have journalctl print only the messages logged within a specific time frame by using the --since and --until switches. The date specification should follow the format YYYY-MM-DD HH:MM:SS. Midnight will be assumed if we omit the time component. By the same token, if the date is omitted, the current day is assumed. For instance, to see messages logged from 7:00pm to 7:01pm, you will type:

root@debian:~# journalctl --since "19:00:00" --until "19:01:00"
-- Logs begin at Sat 2019-10-12 18:55:41 CEST, end at Sat 2019-10-12 20:10:50 CEST. --
Oct 12 19:00:14 debian systemd[1]: Started Run anacron jobs.
Oct 12 19:00:14 debian anacron[1057]: Anacron 2.3 started on 2019-10-12
Oct 12 19:00:14 debian anacron[1057]: Normal exit (0 jobs run)
Oct 12 19:00:14 debian systemd[1]: anacron.timer: Adding 2min 47.988096s random time.

Likewise you can use a slightly different time specification: "integer time-unit ago". Thus, to see messages logged two minutes ago you will type sudo journalctl --since "2 minutes ago". It is also possible to use + and - to specify times relative to the current time so --since "-2 minutes" and --since "2 minutes ago" are equivalent.

Apart from numeric expressions, you can specify a number of keywords:

yesterday

As of midnight of the day before the current day.

today

As of midnight of the current day.

tomorrow

As of midnight of the day after the current day.

now

The current time.

Let us see all messages since last midnight until today at 21:00pm:

root@debian:~# journalctl --since "today" --until "21:00:00"
-- Logs begin at Sat 2019-10-12 20:45:29 CEST, end at Sat 2019-10-12 21:06:15 CEST. --
Oct 12 20:45:29 debian sudo[1416]:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/bin/systemctl r
Oct 12 20:45:29 debian sudo[1416]: pam_unix(sudo:session): session opened for user root by carol(uid=0)
Oct 12 20:45:29 debian systemd[1]: Stopped Flush Journal to Persistent Storage.
(...)
Note

To learn more about the different syntaxes for time specifications, consult the manual page systemd.time.

Program

To see journal messages related to a specific executable the following syntax is used: journalctl /path/to/executable:

root@debian:~# journalctl /usr/sbin/sshd
-- Logs begin at Sat 2019-10-12 20:45:29 CEST, end at Sat 2019-10-12 21:54:49 CEST. --
Oct 12 21:16:28 debian sshd[1569]: Accepted password for carol from 192.168.1.65 port 34050 ssh2
Oct 12 21:16:28 debian sshd[1569]: pam_unix(sshd:session): session opened for user carol by (uid=0)
Oct 12 21:16:54 debian sshd[1590]: Accepted password for carol from 192.168.1.65 port 34052 ssh2
Oct 12 21:16:54 debian sshd[1590]: pam_unix(sshd:session): session opened for user carol by (uid=0)
Unit

Remember, a unit is any resource handled by systemd and you can filter by them, too.

-u

It shows messages about a specified unit:

root@debian:~# journalctl -u ssh.service
-- Logs begin at Sun 2019-10-13 10:50:59 CEST, end at Sun 2019-10-13 12:22:59 CEST. --
Oct 13 10:51:00 debian systemd[1]: Starting OpenBSD Secure Shell server...
Oct 13 10:51:00 debian sshd[409]: Server listening on 0.0.0.0 port 22.
Oct 13 10:51:00 debian sshd[409]: Server listening on :: port 22.
(...)
Note

To print all loaded and active units, use systemctl list-units; to see all installed unit files use systemctl list-unit-files.

Fields

The journal can also be filtered by specific fields through any of the following syntaxes:

  • <field-name>=<value>

  • _<field-name>=<value>_

  • __<field-name>=<value>

    PRIORITY=

    One of eight possible syslog priority values formatted as a decimal string:

    root@debian:~# journalctl PRIORITY=3
    -- Logs begin at Sun 2019-10-13 10:50:59 CEST, end at Sun 2019-10-13 14:30:50 CEST. --
    Oct 13 10:51:00 debian avahi-daemon[314]: chroot.c: open() failed: No such file or directory

    Note how you could have achieved the same output by using the command sudo journalctl -p err that we saw above.

    SYSLOG_FACILITY=

    Any of the possible facility code numbers formatted as a decimal string. For example, to see all user-level messages:

    root@debian:~# journalctl SYSLOG_FACILITY=1
    -- Logs begin at Sun 2019-10-13 10:50:59 CEST, end at Sun 2019-10-13 14:42:52 CEST. --
    Oct 13 10:50:59 debian mtp-probe[227]: checking bus 1, device 2: "/sys/devices/pci0000:00/0000:00:06.0/usb1/1-1"
    Oct 13 10:50:59 debian mtp-probe[227]: bus: 1, device: 2 was not an MTP device
    Oct 13 10:50:59 debian mtp-probe[238]: checking bus 1, device 2: "/sys/devices/pci0000:00/0000:00:06.0/usb1/1-1"
    Oct 13 10:50:59 debian mtp-probe[238]: bus: 1, device: 2 was not an MTP device
    _PID=

    Show messages produced by a specific process ID. To see all messages produced by systemd, you would type:

    root@debian:~# journalctl _PID=1
    -- Logs begin at Sun 2019-10-13 10:50:59 CEST, end at Sun 2019-10-13 14:50:15 CEST. --
    Oct 13 10:50:59 debian systemd[1]: Mounted Debug File System.
    Oct 13 10:50:59 debian systemd[1]: Mounted POSIX Message Queue File System.
    Oct 13 10:50:59 debian systemd[1]: Mounted Huge Pages File System.
    Oct 13 10:50:59 debian systemd[1]: Started Remount Root and Kernel File Systems.
    Oct 13 10:50:59 debian systemd[1]: Starting Flush Journal to Persistent Storage...
    (...)
    _BOOT_ID

    Based on its boot ID you can single out the messages from a specific boot, for example: sudo journalctl _BOOT_ID=83df3e8653474ea5aed19b41cdb45b78.

    _TRANSPORT

    Show messages received from a specific transport. Possible values are: audit (kernel audit subsystem), driver (generated internally), syslog (syslog socket), journal (native journal protocol), stdout (services' standard output or standard error), kernel (kernel ring buffer — the same as dmesg, journalctl -k or journalctl --dmesg):

    root@debian:~# journalctl _TRANSPORT=journal
    -- Logs begin at Sun 2019-10-13 20:19:58 CEST, end at Sun 2019-10-13 20:46:36 CEST. --
    Oct 13 20:19:58 debian systemd[1]: Started Create list of required static device nodes for the current kernel.
    Oct 13 20:19:58 debian systemd[1]: Starting Create Static Device Nodes in /dev...
    Oct 13 20:19:58 debian systemd[1]: Started Create Static Device Nodes in /dev.
    Oct 13 20:19:58 debian systemd[1]: Starting udev Kernel Device Manager...
    (...)
Combining Fields

Fields are not mutually exclusive so you can use more than one in the same query. However, only messages that match the value of both fields simultaneously will be shown:

root@debian:~# journalctl PRIORITY=3 SYSLOG_FACILITY=0
-- No entries --
root@debian:~# journalctl PRIORITY=4 SYSLOG_FACILITY=0
-- Logs begin at Sun 2019-10-13 20:19:58 CEST, end at Sun 2019-10-13 20:21:55 CEST. --
Oct 13 20:19:58 debian kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration (...)

Unless you use the + separator to combine two expressions in the manner of a logical OR:

root@debian:~# journalctl PRIORITY=3 + SYSLOG_FACILITY=0
-- Logs begin at Sun 2019-10-13 20:19:58 CEST, end at Sun 2019-10-13 20:24:02 CEST. --
Oct 13 20:19:58 debian kernel: Linux version 4.9.0-9-amd64 (debian-kernel@lists.debian.org) (...9
Oct 13 20:19:58 debian kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-9-amd64 root=UUID= (...)
(...)

On the other hand, you can supply two values for the same field and all entries matching either value will be shown:

root@debian:~# journalctl PRIORITY=1
-- Logs begin at Sun 2019-10-13 17:16:24 CEST, end at Sun 2019-10-13 17:30:14 CEST. --
-- No entries --
root@debian:~# journalctl PRIORITY=1 PRIORITY=3
-- Logs begin at Sun 2019-10-13 17:16:24 CEST, end at Sun 2019-10-13 17:32:12 CEST. --
Oct 13 17:16:27 debian connmand[459]: __connman_inet_get_pnp_nameservers: Cannot read /pro
Oct 13 17:16:27 debian connmand[459]: The name net.connman.vpn was not provided by any .se
Note

Journal fields fall in any of the following categories: “User Journal Fields”, “Trusted Journal Fields”, “Kernel Journal Fields”, “Fields on behalf of a different program” and “Address Fields”. For more information on this topic — including a complete list of fields — see the man page for systemd.journal-fields(7).

Manual Entries in the System Journal: systemd-cat

Just like how the logger command is used to send messages from the command line to the system log (as we saw in the previous lesson), the systemd-cat command serves a similar — but more well-rounded — purpose with the system journal. It allows us to send standard input (stdin), output (stdout) and error (stderr) to the journal.

If invoked with no parameters, it will send everything it reads from stdin to the journal. Once you are done, press Ctrl+C:.

carol@debian:~$ systemd-cat
This line goes into the journal.
^C

If it is passed the output of a piped command, this will be sent to the journal too:

carol@debian:~$ echo "And so does this line." | systemd-cat

If followed by a command, that command’s output will also be sent to the journal — together with stderr (if any):

carol@debian:~$ systemd-cat echo "And so does this line too."

There is also the possibility of specifiying a priority level with the -p option:

carol@debian:~$ systemd-cat -p emerg echo "This is not a real emergency."

Refer to systemd-cat man page to learn about its other options.

To see the last four lines in the journal:

carol@debian:~$ journalctl -n 4
(...)
-- Logs begin at Sun 2019-10-20 13:43:54 CEST. --
Nov 13 23:14:39 debian cat[1997]: This line goes into the journal.
Nov 13 23:19:16 debian cat[2027]: And so does this line.
Nov 13 23:23:21 debian echo[2030]: And so does this line too.
Nov 13 23:26:48 debian echo[2034]: This is not a real emergency.
Note

Journal entries with a priority level of emergency will be printed in bold red on most systems.

Persistent Journal Storage

As mentioned previously, you have three options when it comes to the location of the journal:

  • Journaling can be turned off altogether (redirection to other facilities such as the console are still possible, though).

  • Keep it in memory — which makes it volatile — and get rid of the logs with every system reboot. In this scenario, the directory /run/log/journal will be created and used.

  • Make it persistent so that it writes logs to disk. In this case, log messages will go into the /var/log/journal directory.

The default behaviour is as follows: if /var/log/journal/ does not exist, logs will be saved in a volatile way to a directory in /run/log/journal/ and — therefore — lost at reboot. The name of the directory — the /etc/machine-id — is a newline-terminated, hexadecimal, 32-character, lowercase string:

carol@debian:~$ ls /run/log/journal/8821e1fdf176445697223244d1dfbd73/
system.journal

If you try to read it with less you will get a warning, so instead use the command journalctl:

root@debian:~# less /run/log/journal/9a32ba45ce44423a97d6397918de1fa5/system.journal
"/run/log/journal/9a32ba45ce44423a97d6397918de1fa5/system.journal" may be a binary file.  See it anyway?
root@debian:~# journalctl
-- Logs begin at Sat 2019-10-05 21:26:38 CEST, end at Sat 2019-10-05 21:31:27 CEST. --
(...)
Oct 05 21:26:44 debian systemd-journald[1712]: Runtime journal (/run/log/journal/9a32ba45ce44423a97d6397918de1fa5) is 4.9M, max 39.5M, 34.6M free.
Oct 05 21:26:44 debian systemd[1]: Started Journal Service.
(...)

If /var/log/journal/ exists, logs will be stored persistently there. Should this directory be deleted, systemd-journald would not recreate it but write to /run/log/journal instead. As soon as we create /var/log/journal/ again and restart the daemon, persistent logging will be restablished:

root@debian:~# mkdir /var/log/journal/
root@debian:~# systemctl restart systemd-journald
root@debian:~# journalctl
(...)
Oct 05 21:33:49 debian systemd-journald[1712]: Received SIGTERM from PID 1 (systemd).
Oct 05 21:33:49 debian systemd[1]: Stopped Journal Service.
Oct 05 21:33:49 debian systemd[1]: Starting Journal Service...
Oct 05 21:33:49 debian systemd-journald[1768]: Journal started
Oct 05 21:33:49 debian systemd-journald[1768]: System journal (/var/log/journal/9a32ba45ce44423a97d6397918de1fa5) is 8.0M, max 1.1G, 1.1G free.
Oct 05 21:33:49 debian systemd[1]: Started Journal Service.
Oct 05 21:33:49 debian systemd[1]: Starting Flush Journal to Persistent Storage...
(...)
Note

By default, there will be specific journal files for every logged in user, located in /var/log/journal/, so — together with system.journal files — you will also find files of the type user-1000.journal.

On top of what we have just mentioned, the way the journal daemon deals with log storage can be changed after installation by tweaking its configuration file: /etc/systemd/journald.conf. The key option is Storage= and can have the following values:

Storage=volatile

Log data will be stored exclusively in memory — under /run/log/journal/. If not present, the directory will be created.

Storage=persistent

By default log data will be stored on disk  — under /var/log/journal/ —  with a fallback to memory (/run/log/journal/) during early boot stages and if the disk is not writable. Both directories will be created if needed.

Storage=auto

auto is similar to persistent, but the directory /var/log/journal is not created if needed. This is the default.

Storage=none

All log data will be discarded. Forwarding to other targets such as the console, the kernel log buffer, or a syslog socket are still possible, though.

For instance, to have systemd-journald create /var/log/journal/ and switch to persistent storage, you would edit /etc/systemd/journald.conf and set Storage=persistent, save the file and restart the daemon with sudo systemctl restart systemd-journald. To ensure the restart went flawlessly, you can always check the daemon’s status:

root@debian:~# systemctl status systemd-journald
 systemd-journald.service - Journal Service
   Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
   Active: active (running) since Wed 2019-10-09 10:03:40 CEST; 2s ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 1872 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 3558)
   Memory: 1.1M
   CGroup: /system.slice/systemd-journald.service
           └─1872 /lib/systemd/systemd-journald

Oct 09 10:03:40 debian10 systemd-journald[1872]: Journal started
Oct 09 10:03:40 debian10 systemd-journald[1872]: System journal (/var/log/journal/9a32ba45ce44423a97d6397918de1fa5) is 8.0M, max 1.2G, 1.2G free.
Note

The journal files in /var/log/journal/<machine-id>/ or /run/log/journal/<machine-id>/ have the .journal suffix (e.g. system.journal). However, if they are found to be corrupted or the daemon is stopped in an unclean fashion, they will be renamed appending ~ (e.g. system.journal~) and the daemon will start writing to a new, clean file.

Deleting Old Journal Data: Journal Size

Logs are saved in journal files whose file names end with .journal or .journal~ and are located in the appropriate directory (/run/log/journal or /var/log/journal as configured). To check how much disk space is currently being occupied by journal files (both archived and active), use the --disk-usage switch:

root@debian:~# journalctl --disk-usage
Archived and active journals take up 24.0M in the filesystem.

systemd logs default to a maximum of 10% of the size of the filesystem where they are stored. For instance on a 1GB filesystem they will not take up more than 100MB. Once this limit is reached, old logs will start to disappear to stay near this value.

However, size limit enforcement on stored journal files can be managed by tweaking a series of configuration options in /etc/systemd/journald.conf. These options fall into two categories depending on the filesystem type used: persistent (/var/log/journal) or in-memory (/run/log/journal). The former uses options that are prefixed with the word System and will only apply if persistent logging is properly enabled and once the system is fully booted up. The option names in the latter start with the word Runtime and will apply in the following scenarios:

SystemMaxUse=, RuntimeMaxUse=

They control the amount of disk space that can be taken up by the journal. It defaults to 10% of the filesystem size but can be modified (e.g. SystemMaxUse=500M) as long as it does not surpass a maximum of 4GiB.

SystemKeepFree=, RuntimeKeepFree=

They control the amount of disk space that should be left free for other users. It defaults to 15% of the filesystem size but can be modified (e.g. SystemKeepFree=500M) as long as it does not surpass a maximum of 4GiB.

Regarding precedence of *MaxUse and *KeepFree, systemd-journald will satisfy both by using the smaller of the two values. Likewise, bear in mind that only archived (as opposed to active) journal files are deleted.

SystemMaxFileSize=, RuntimeMaxFileSize=

They control the maximum size to which individual journal files can grow. The default is 1/8 of *MaxUse. Size reduction is carried out in a synchronous way and values can be specified in bytes or using K, M, G, T, P, E for Kibibytes, Mebibytes, Gibibyte, Tebibytes, Pebibytes and Exbibytes, respectively.

SystemMaxFiles=, RuntimeMaxFiles=

They establish the maximum number of individual and archived journal files to store (active journal files are not affected). It defaults to 100.

Apart from size-based deletion and rotation of log messages, systemd-journald also allows for time-based criteria by using the following two options: MaxRetentionSec= and MaxFileSec=. Refer to the manual page of journald.conf for more information about these and other options.

Note

Whenever you modify the default behaviour of systemd-journald by uncommenting and editing options in /etc/systemd/journald.conf, you must restart the daemon for the changes to take effect.

Vacuuming the Journal

You can manually clean archived journal files at any time with any of the following three options:

--vacuum-time=

This time-based option will eliminate all messages in journal files with a timestamp older than the specified timeframe. Values must be written with any of the following suffixes: s, m, h, days (or d), months, weeks (or w) and years (or y). For instance, to get rid of all messages in archived journal files that are older than 1 month:

root@debian:~# journalctl --vacuum-time=1m
Deleted archived journal /var/log/journal/7203088f20394d9c8b252b64a0171e08/system@27dd08376f71405a91794e632ede97ed-0000000000000001-00059475764d46d6.journal (16.0M).
Deleted archived journal /var/log/journal/7203088f20394d9c8b252b64a0171e08/user-1000@e7020d80d3af42f0bc31592b39647e9c-000000000000008e-00059479df9677c8.journal (8.0M).
--vacuum-size=

This size-based option will delete archived journal files until they occupy a value below the specified size. Values must be written with any of the following suffixes: K, M, G or T. For instance, to eliminate archived journal files until they are below 100 Mebibits:

root@debian:~# journalctl --vacuum-size=100M
Vacuuming done, freed 0B of archived journals from /run/log/journal/9a32ba45ce44423a97d6397918de1fa5.
--vacuum-files=

This option will take care that no more archived journal files than the specified number remain. The value is an integer. For instance, to limit the number of archived journal files to 10:

root@debian:~# journalctl --vacuum-files=10
Vacuuming done, freed 0B of archived journals from /run/log/journal/9a32ba45ce44423a97d6397918de1fa5.

Vacuuming only removes archived journal files. If you want to get rid of everything (including active journal files), you need to use a signal (SIGUSR2) that requests immediate rotation of the journal files with the --rotate option. Other important signals can be invoked with the following options:

--flush (SIGUSR1)

It requests flushing of journal files from /run/ to /var/ to make the journal persistent. It requires that persistent logging is enabled and /var/ is mounted.

--sync (SIGRTMIN+1)

It is used to request that all unwritten log data be written to disk.

Note

To check the internal consistency of the journal file, use journalctl with the --verify option. You will see a progress bar as the check is done and any possible issues will be shown.

Retrieving Journal Data from a Rescue System

As a system administrator, you may find yourself in a situation where you need to access journal files on the hard drive of a faulty machine through a rescue system (a bootable CD or USB key containing a live Linux distribution).

journalctl looks for the journal files in /var/log/journal/<machine-id>/. Because the machine IDs on the rescue and faulty systems will be different, you must use the following option:

-D </path/to/dir>, --directory=</path/to/dir>

With this option, we specify a directory path where journalctl will search for journal files instead of the default runtime and system locations.

Thus, it is necessary that you mount the faulty system’s rootfs (/dev/sda1) on the rescue system’s filesystem and proceed to read the journal files like so:

root@debian:~# journalctl -D /media/carol/faulty.system/var/log/journal/
-- Logs begin at Sun 2019-10-20 12:30:45 CEST, end at Sun 2019-10-20 12:32:57 CEST. --
oct 20 12:30:45 suse-server kernel: Linux version 4.12.14-lp151.28.16-default (geeko@buildhost) (...)
oct 20 12:30:45 suse-server kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.16-default root=UUID=7570f67f-4a08-448e-aa09-168769cb9289 splash=>
oct 20 12:30:45 suse-server kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
oct 20 12:30:45 suse-server kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
(...)

Other options that can be useful in this scenario are:

-m, --merge

It merges entries from all available journals under /var/log/journal, including remote ones.

--file

It will show the entries in a specific file, for example: journalctl --file /var/log/journal/64319965bda04dfa81d3bc4e7919814a/user-1000.journal.

--root

A directory path meaning the root directory is passed as an argument. journalctl will search there for journal files (e.g. journalctl --root /faulty.system/).

See journalctl man page for more information.

Forwarding Log Data to a Traditional syslog Daemon

Log data from the journal can be made available to a traditional syslog daemon by:

  • Forwarding messages to the socket file /run/systemd/journal/syslog for syslogd to read. This facility is enabled with the ForwardToSyslog=yes option.

  • Having a syslog daemon behaving like journalctl, therefore reading log messages directly from the journal files. In this case, the relevant option is that of Storage; it must have a value other than none.

Note

Likewise, you can forward log messages to other destinations with the following options: ForwardToKMsg (kernel log buffer — kmsg), ForwardToConsole (the system console) or ForwardToWall (all logged-in users via wall). For more information, consult the man page for journald.conf.

Guided Excercises

  1. Assuming you are root, complete the table with the appropriate journalctl command:

    Purpose Command

    Print kernel entries

    Print messages from the second boot starting at beginning of journal

    Print messages from the second boot starting at end of journal

    Print most recent log messages and keep watching for new ones

    Print only new messages since now, and update the output continuously

    Print messages from the previous boot with a priority of warning and in reverse order

  2. The behaviour of the journal daemon concerning storage is mostly controlled by the value of the Storage option in /etc/systemd/journald.conf. Indicate what behaviour is related to what value in the following table:

    Behaviour Storage=auto Storage=none Storage=persistent Storage=volatile

    Log data is thrown away but forwarding is possible.

    Once the system has booted up, log data will be stored under /var/log/journal. If not already present, the directory will be created.

    Once the system has booted up, log data will be stored under /var/log/journal. If not already present, the directory will not be created.

    Log data will be stored under /var/run/journal but will not survive reboots.

  3. As you learned, the journal can be manually vacuumed based on time, size and number of files. Complete the following tasks using journalctl and the appropriate options:

    • Check how much disk space is taken up by journal files:

    • Cut down on the quantity of space reserved for archived journal files and set it to 200MiB:

    • Check on disk space again and explain the results:

Explorational Excercises

  1. What options should you modify in /etc/systemd/journald.conf so that messages are forwarded to /dev/tty5? What values should the options have?

  2. Provide the correct journalctl filter to print the following:

    Purpose Filter + Value

    Print messages belonging to a specific user

    Print messages from a host named debian

    Print messages belonging to a specific group

    Print messages belongin to root

    Based on the executable path, print sudo messages

    Based on the command name, print sudo messages

  3. When filtering by priority, logs with a higher priority than indicated will also be included in the listing; for instance journalctl -p err will print error, critical, alert and emergency messages. However, you can have journalctl show only a specific range. What command would you use to have journalctl print only messages in the warning, error and critical priority levels?

  4. Priority levels can also be specified numerically. Rewrite the command in the previous exercise using the numeric representation of priority levels:

Summary

In this lesson you learned:

  • The advantages of using systemd as a system and service manager.

  • The basics of systemd units and targets.

  • Where systemd-journald gets logging data from.

  • The options you can pass systemctl to control systemd-journald: start, status, restart and stop.

  • Where the journal’s configuration file is located — /etc/systemd/journald.conf — and its main options.

  • How to query the journal in a general way and for specific data by using filters.

  • How to navigate and search through the journal.

  • How to deal with storage of journal files: in memory vs. on disk.

  • How to disable journaling altogether.

  • How to check the disk space taken up by the journal, enforce size limits on stored journal files and clean archived journal files manually (vacumming).

  • How to retrieve journal data from a rescue system.

  • How to forward log data to a traditional syslog daemon.

Commands used in this lesson:

systemctl

Control the systemd system and service manager.

journalctl

Query the systemd journal.

ls

List directory contents.

less

View file contents.

mkdir

Make directories.

Answers to Guided Exercises

  1. Assuming you are root, complete the table with the appropriate journalctl command:

    Purpose Command

    Print kernel entries

    journalctl -k or journalctl --dmesg

    Print messages from the second boot starting at beginning of journal

    journalctl -b 2

    Print messages from the second boot starting at end of journal

    journalctl -b -2 -r or journalctl -r -b -2

    Print most recent log messages and keep watching for new ones

    journalctl -f

    Print only new messages since now, and update the output continuously

    journalctl --since "now" -f

    Print messages from the previous boot with a priority of warning and in reverse order

    journalctl -b -1 -p warning -r

  2. The behaviour of the journal daemon concerning storage is mostly controlled by the value of the Storage option in /etc/systemd/journald.conf. Indicate what behaviour is related to what value in the following table:

    Behaviour Storage=auto Storage=none Storage=persistent Storage=volatile

    Log data is thrown away but forwarding is possible

    x

    Once the system has booted up, log data will be stored under /var/log/journal. If not already present, the directory will be created

    x

    Once the system has booted up, log data will be stored under /var/log/journal. If not already present, the directory will not be created

    x

    Log data will be stored under /var/run/journal but will not survive reboots

    x

  3. As you learned, the journal can be manually vacuumed based on time, size and number of files. Complete the following tasks using journalctl and the appropriate options:

    • Check how much disk space is taken up by journal files:

      journalctl --disk-usage
    • Cut down on the quantity of space reserved for archived journal files and set it to 200MiB:

      journalctl --vacuum-size=200M
    • Check on disk space again and explain the results:

      journalctl --disk-usage

      There is no correlation because --disk-usage shows space occupied by both active and archived journal files whereas --vacuum-size only applies to archived files.

Answers to Explorational Exercises

  1. What options should you modify in /etc/systemd/journald.conf so that messages are forwarded to /dev/tty5? What values should the options have?

    ForwardToConsole=yes
    TTYPath=/dev/tty5
  2. Provide the correct journalctl filter to print the following:

    Purpose Filter + Value

    Print messages belonging to a specific user

    _ID=<user-id>

    Print messages from a host named debian

    _HOSTNAME=debian

    Print messages belonging to a specific group

    _UID=<group-id>

    Print messages belongin to root

    _UID=0

    Based on the executable path, print sudo messages

    _EXE=/usr/bin/sudo

    Based on the command name, print sudo messages

    _COMM=sudo

  3. When filtering by priority, logs with a higher priority than indicated will also be included in the listing; for instance journalctl -p err will print error, critical, alert and emergency messages. However, you can have journalctl show only a specific range. What command would you use to have journalctl print only messages in the warning, error and critical priority levels?

    journalctl -p warning..crit
  4. Priority levels can also be specified numerically. Rewrite the command in the previous exercise using the numeric representation of priority levels:

    journalctl -p 4..2

© 2020 Linux Professional Insitute Inc. All rights reserved. Visit the Learning Materials website: https://learning.lpi.org
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Next Lesson

108.3 Mail Transfer Agent (MTA) basics (108.3 Lesson 1)

Read next lesson

© 2020 Linux Professional Insitute Inc. All rights reserved. Visit the Learning Materials website: https://learning.lpi.org
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

LPI is a non-profit organization.

Linux Professional Institute (LPI) is the global certification standard and career support organization for open source professionals. With more than 200,000 certification holders, it's the world’s first and largest vendor-neutral Linux and open source certification body. LPI has certified professionals in over 180 countries, delivers exams in multiple languages, and has hundreds of training partners.

Our purpose is to enable economic and creative opportunities for everybody by making open source knowledge and skills certification universally accessible.

  • LinkedIn
  • flogo-RGB-HEX-Blk-58 Facebook
  • Twitter
  • Contact Us
  • Privacy and Cookie Policy

Spot a mistake or want to help improve this page? Please let us know.

© Copyright 1999-2020 The Linux Professional Institute Inc. All rights reserved.