Linux Professional Institute Learning Logo.
Skip to main content
  • Home
    • All Resources
    • LPI Learning Materials
    • Become a Contributor
    • Publishing Partners
    • Become a Publishing Partner
    • About
    • FAQ
    • Contributors
    • Roadmap
    • Contact
  • LPI.org
108.2 Lesson 1
Topic 105: Shells and Shell Scripting
105.1 Customize and use the shell environment
  • 105.1 Lesson 1
  • 105.1 Lesson 2
  • 105.1 Lesson 3
105.2 Customize or write simple scripts
  • 105.2 Lesson 1
  • 105.2 Lesson 2
Topic 106: User Interfaces and Desktops
106.1 Install and configure X11
  • 106.1 Lesson 1
106.2 Graphical Desktops
  • 106.2 Lesson 1
106.3 Accessibility
  • 106.3 Lesson 1
Topic 107: Administrative Tasks
107.1 Manage user and group accounts and related system files
  • 107.1 Lesson 1
  • 107.1 Lesson 2
107.2 Automate system administration tasks by scheduling jobs
  • 107.2 Lesson 1
  • 107.2 Lesson 2
107.3 Localisation and internationalisation
  • 107.3 Lesson 1
Topic 108: Essential System Services
108.1 Maintain system time
  • 108.1 Lesson 1
  • 108.1 Lesson 2
108.2 System logging
  • 108.2 Lesson 1
  • 108.2 Lesson 2
108.3 Mail Transfer Agent (MTA) basics
  • 108.3 Lesson 1
108.4 Manage printers and printing
  • 108.4 Lesson 1
Topic 109: Networking Fundamentals
109.1 Fundamentals of internet protocols
  • 109.1 Lesson 1
  • 109.1 Lesson 2
109.2 Persistent network configuration
  • 109.2 Lesson 1
  • 109.2 Lesson 2
109.3 Basic network troubleshooting
  • 109.3 Lesson 1
  • 109.3 Lesson 2
109.4 Configure client side DNS
  • 109.4 Lesson 1
Topic 110: Security
110.1 Perform security administration tasks
  • 110.1 Lesson 1
110.2 Setup host security
  • 110.2 Lesson 1
110.3 Securing data with encryption
  • 110.3 Lesson 1
  • 110.3 Lesson 2
How to get certified
  1. Topic 108: Essential System Services
  2. 108.2 System logging
  3. 108.2 Lesson 1

108.2 Lesson 1

Certificate:

LPIC-1

Version:

5.0

Topic:

108 Essential System Services

Objective:

108.2 System logging

Lesson:

1 of 2

Introduction

Logs can be a system administrator’s best friend. Logs are files (usually text files) where all system and network events are chronologically registered from the moment your system is booted up. Thus, the range of information that can be found in logs includes virtually every aspect of the system: failed authentication attempts, program and service errors, hosts blocked by the firewall, etc. As you can imagine, logs make system administrators' lives a lot easier when it comes to troubleshooting, resource-checking, detection of anomalous behaviour of programs, and so on.

In this lesson we will discuss one of the most common logging facilites currently found in GNU/Linux distributions: rsyslog. We will study the different types of logs that exist, where they are stored, what information they include and how that information can be obtained and filtered. We will also discuss how logs can be kept in centralized servers across IP networks, log rotation and the kernel ring buffer.

System Logging

The moment the kernel and the different processes in your system start running and communicating with one another, a lot of information is generated in the form of messages that are — for the most part — sent to the logs.

Without logging, searching for an event that happened on a server would give system administrators a headache, hence the importance of having a standardized and centralized way of keeping track of any system events. Logs are determinant and telling when it comes to troubleshooting and security and are reliable data sources for understanding system statistics and making trend predictions.

Leaving aside systemd-journald (which we will discuss in the next lesson), logging has traditionally been handled by three main dedicated services: syslog, syslog-ng (syslog new generation) and rsyslog (“the rocket-fast system for log processing”). rsyslog brought along important improvements (such as RELP support) and has become the most popular choice nowadays. Each of these services collects messages from other services and programs and stores them in log files, typically under /var/log. However, some services take care of their own logs (take — for example — the Apache HTTPD web server or the CUPS printing system). Likewise, the Linux kernel uses an in-memory ring buffer for storing its log messages.

Note

RELP stands for Reliable Event Logging Protocol and extends the functionality of the syslog protocol to provide reliable delivery of messages.

Since rsyslog has become the de facto standard logging facility in all major distros, we will focus on it for the present lesson. rsyslog uses a client-server model. The client and the server can live on the same host or in different machines. Messages are sent and received in a particular format and can be kept in centralized rsyslog servers across IP networks. rsyslog’s daemon — rsyslogd — works together with klogd (which manages kernel messages). In the next sections rsyslog and its logging infrastructure will be discussed.

Note

A daemon is a service that runs in the background. Note the final d in daemon names: klogd or rsyslogd.

Log Types

Because logs are variable data, they are normally found in /var/log. Roughly speaking, they can be classified into system logs and service or program logs.

Let us see some system logs and the information they keep:

/var/log/auth.log

Activities related to authentication processes: logged users, sudo information, cron jobs, failed login attempts, etc.

/var/log/syslog

A centralized file for practically all of the logs captured by rsyslogd. Because it includes so much information, logs are distributed across other files according to the configuration supplied in /etc/rsyslog.conf.

/var/log/debug

Debug information from programs.

/var/log/kern.log

Kernel messages.

/var/log/messages

Informative messages which are not related to the kernel but to other services. It is also the default remote client log destination in a centralized log server implementation.

/var/log/daemon.log

Information related to daemons or services running in the background.

/var/log/mail.log

Information related to the email server, e.g. postfix.

/var/log/Xorg.0.log

Information related to the graphics card.

/var/run/utmp and /var/log/wtmp

Successful logins.

/var/log/btmp

Failed login attempts, e.g. brute force attack via ssh.

/var/log/faillog

Failed authentication attempts.

/var/log/lastlog

Date and time of recent user logins.

Now let us see a few examples of service logs:

/var/log/cups/

Directory for logs of the Common Unix Printing System. It commonly includes the following default log files: error_log, page_log and access_log.

/var/log/apache2/ or /var/log/httpd

Directory for logs of the Apache Web Server. It commonly includes the following default log files: access.log, error_log, and other_vhosts_access.log.

/var/log/mysql

Directory for logs of the MySQL Relational Database Management System. It commonly includes the following default log files: error_log, mysql.log and mysql-slow.log.

/var/log/samba/

Directory for logs of the Session Message Block (SMB) protocol. It commonly includes the following default log files: log., log.nmbd and log.smbd.

Note

The exact name and contents of log files may vary across Linux distributions. There are also logs particular to specific distributions such as /var/log/dpkg.log (containing information related to dpkg packages) in Debian GNU/Linux and its derivatives.

Reading Logs

To read log files, first ensure you are the root user or have reading permissions on the file. You can use a variety of utilities such as:

less or more

Pagers that allow viewing and scrolling one page at a time:

root@debian:~# less /var/log/auth.log
Sep 12 18:47:56 debian sshd[441]: Received SIGHUP; restarting.
Sep 12 18:47:56 debian sshd[441]: Server listening on 0.0.0.0 port 22.
Sep 12 18:47:56 debian sshd[441]: Server listening on :: port 22.
Sep 12 18:47:56 debian sshd[441]: Received SIGHUP; restarting.
Sep 12 18:47:56 debian sshd[441]: Server listening on 0.0.0.0 port 22.
Sep 12 18:47:56 debian sshd[441]: Server listening on :: port 22.
Sep 12 18:49:46 debian sshd[905]: Accepted password for carol from 192.168.1.65 port 44296 ssh2
Sep 12 18:49:46 debian sshd[905]: pam_unix(sshd:session): session opened for user carol by (uid=0)
Sep 12 18:49:46 debian systemd-logind[331]: New session 2 of user carol.
Sep 12 18:49:46 debian systemd: pam_unix(systemd-user:session): session opened for user carol by (uid=0)
(...)
zless or zmore

The same as less and more, but used for logs that are compressed with gzip (a common function of logrotate):

root@debian:~# zless /var/log/auth.log.3.gz
Aug 19 20:05:57 debian sudo:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/sbin/shutdown -h now
Aug 19 20:05:57 debian sudo: pam_unix(sudo:session): session opened for user root by carol(uid=0)
Aug 19 20:05:57 debian lightdm: pam_unix(lightdm-greeter:session): session closed for user lightdm
Aug 19 23:50:49 debian systemd-logind[333]: Watching system buttons on /dev/input/event2 (Power Button)
Aug 19 23:50:49 debian systemd-logind[333]: Watching system buttons on /dev/input/event3 (Sleep Button)
Aug 19 23:50:49 debian systemd-logind[333]: Watching system buttons on /dev/input/event4 (Video Bus)
Aug 19 23:50:49 debian systemd-logind[333]: New seat seat0.
Aug 19 23:50:49 debian sshd[409]: Server listening on 0.0.0.0 port 22.
(...)
tail

View the last lines in a file (the default is 10 lines). The power of tail lies — to a great extent — in the -f switch, which will dynamically show new lines as they are appended:

root@suse-server:~# tail -f /var/log/messages
2019-09-14T13:57:28.962780+02:00 suse-server sudo: pam_unix(sudo:session): session closed for user root
2019-09-14T13:57:38.038298+02:00 suse-server sudo:    carol : TTY=pts/0 ; PWD=/home/carol ; USER=root ; COMMAND=/usr/bin/tail -f /var/log/messages
2019-09-14T13:57:38.039927+02:00 suse-server sudo: pam_unix(sudo:session): session opened for user root by carol(uid=0)
2019-09-14T14:07:22+02:00 debian carol: appending new message from client to remote server...
head

View the first lines in a file (the default is 10 lines):

root@suse-server:~# head -5 /var/log/mail
2019-06-29T11:47:59.219806+02:00 suse-server postfix/postfix-script[1732]: the Postfix mail system is not running
2019-06-29T11:48:01.355361+02:00 suse-server postfix/postfix-script[1925]: starting the Postfix mail system
2019-06-29T11:48:01.391128+02:00 suse-server postfix/master[1930]: daemon started -- version 3.3.1, configuration /etc/postfix
2019-06-29T11:55:39.247462+02:00 suse-server postfix/postfix-script[3364]: stopping the Postfix mail system
2019-06-29T11:55:39.249375+02:00 suse-server postfix/master[1930]: terminating on signal 15
grep

Filtering utility which allows you to search for specific strings:

root@debian:~# grep "dhclient" /var/log/syslog
Sep 13 11:58:48 debian dhclient[448]: DHCPREQUEST of 192.168.1.4 on enp0s3 to 192.168.1.1 port 67
Sep 13 11:58:49 debian dhclient[448]: DHCPACK of 192.168.1.4 from 192.168.1.1
Sep 13 11:58:49 debian dhclient[448]: bound to 192.168.1.4 -- renewal in 1368 seconds.
(...)

As you may have noticed, the output is printed in the following format:

  • Timestamp

  • Hostname from which the message originated

  • Name of program/service that generated the message

  • The PID of the program that generated the message

  • Description of the action that took place

There are a few examples in which logs are not text, but binary files and — consequently — you must use special commands to parse them:

/var/log/wtmp

Use who (or w):

root@debian:~# who
root    pts/0        2020-09-14 13:05 (192.168.1.75)
root    pts/1        2020-09-14 13:43 (192.168.1.75)
/var/log/btmp

Use utmpdump or last -f:

root@debian:~# utmpdump /var/log/btmp
Utmp dump of /var/log/btmp
[6] [01287] [    ] [dave     ] [ssh:notty   ] [192.168.1.75        ] [192.168.1.75   ] [2019-09-07T19:33:32,000000+0000]
/var/log/faillog

Use faillog:

root@debian:~# faillog -a | less
Login       Failures Maximum Latest                   On

root            0        0   01/01/70 01:00:00 +0100
daemon          0        0   01/01/70 01:00:00 +0100
bin             0        0   01/01/70 01:00:00 +0100
sys             0        0   01/01/70 01:00:00 +0100
sync            0        0   01/01/70 01:00:00 +0100
games           0        0   01/01/70 01:00:00 +0100
man             0        0   01/01/70 01:00:00 +0100
lp              0        0   01/01/70 01:00:00 +0100
mail            0        0   01/01/70 01:00:00 +0100
(...)
/var/log/lastlog

Use lastlog:

root@debian:~# lastlog | less
Username         Port     From             Latest
root                                       Never logged in
daemon                                     Never logged in
bin                                        Never logged in
sys                                        Never logged in
(...)
sync                                       Never logged in
avahi                                      Never logged in
colord                                     Never logged in
saned                                      Never logged in
hplip                                      Never logged in
carol            pts/1    192.168.1.75     Sat Sep 14 13:43:06 +0200 2019
dave             pts/3    192.168.1.75     Mon Sep  2 14:22:08 +0200 2019
Note

There are also graphical tools for reading log files, for example: gnome-logs and KSystemLog.

How Messages are Turned into Logs

The following process illustrates how a message is written to a log file:

  1. Applications, services and the kernel write messages in special files (sockets and memory buffers), e.g. /dev/log or /dev/kmsg.

  2. rsyslogd gets the information from the sockets or memory buffers.

  3. Depending on the rules found in /etc/rsyslog.conf and/or the files in /etc/ryslog.d/, rsyslogd moves the information to the corresponding log file (typically found in /var/log).

Note

A socket is a special file used to transfer information between different processes. To list all sockets on your system, you can use the command systemctl list-sockets --all.

Facilities, Priorities and Actions

rsyslog configuration file is /etc/rsylog.conf (in some distributions you can also find configuration files in /etc/rsyslog.d/). It is normally divided into three sections: MODULES, GLOBAL DIRECTIVES and RULES. Let us have a look at them by exploring the rsyslog.conf file in our Debian GNU/Linux 10 (buster) host — you can use sudo less /etc/rsyslog.conf to do so.

MODULES includes module support for logging, message capability, and UDP/TCP log reception:

#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

GLOBAL DIRECTIVES allow us to configure a number of things such as logs and log directory permissions:

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

RULES is where facilities, priorities and actions come in. The settings in this section tell the logging daemon to filter messages according to certain rules and log them or send them where required. To understand these rules, we should first explain the concepts of rsyslog facilities and priorities. Each log message is given a facility number and keyword that are associated with the Linux internal subsystem that produces the message:

Number Keyword Description

0

kern

Linux kernel messages

1

user

User-level messages

2

mail

Mail system

3

daemon

System daemons

4

auth, authpriv

Security/Authorization messages

5

syslog

syslogd messages

6

lpr

Line printer subsystem

7

news

Network news subsystem

8

uucp

UUCP (Unix-to-Unix Copy Protocol) subsystem

9

cron

Clock daemon

10

auth, authpriv

Security/Authorization messages

11

ftp

FTP (File Transfer Protocol) daemon

12

ntp

NTP (Network Time Protocol) daemon

13

security

Log audit

14

console

Log alert

15

cron

Clock daemon

16 - 23

local0 through local7

Local use 0 - 7

Furthermore, each message is assigned a priority level:

Code Severity Keyword Description

0

Emergency

emerg, panic

System is unusable

1

Alert

alert

Action must be taken immediately

2

Critical

crit

Critical conditions

3

Error

err, error

Error conditions

4

Warning

warn, warning

Warning conditions

5

Notice

notice

Normal but significant condition

6

Informational

info

Informational messages

7

Debug

debug

Debug-level messages

Here is an excerpt of rsyslog.conf from our Debian GNU/Linux 10 (buster) system which including some sample rules:

###############
#### RULES ####
###############

# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err

#
# Some "catch-all" log files.
#
*.=debug;\
        auth,authpriv.none;\
	news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
	auth,authpriv.none;\
	cron,daemon.none;\
	mail,news.none          -/var/log/messages

The rule format is as follows: <facility>.<priority> <action>

The <facility>.<priority> selector filters messages to match. Priority levels are hierarchically inclusive, which means rsyslog will match messages of the specified priority and higher. The <action> shows what action to take (where to send the log message). Here are a few examples for clarity:

auth,authpriv.*                 /var/log/auth.log

Regardless of their priority (*), all messages from the auth or authpriv facilities will be sent to /var/log/auth.log.

*.*;auth,authpriv.none          -/var/log/syslog

All messages — irrespective of their priority (*) — from all facilities (*) — discarding those from auth or authpriv (hence the .none suffix) — will be written to /var/log/syslog (the minus sign (-) before the path prevents excessive disk writes). Note the semicolon (;) to split the selector and the comma (,) to concatenate two facilities in the same rule (auth,authpriv).

mail.err                        /var/log/mail.err

Messages from the mail facility with a priority level of error or higher (critical, alert or emergency) will be sent to /var/log/mail.err.

*.=debug;\
        auth,authpriv.none;\
	news.none;mail.none     -/var/log/debug

Messages from all facilities with the debug priority and no other (=) will be written to /var/log/debug — excluding any messages coming from the auth, authpriv, news and mail facilities (note the syntax: ;\).

Manual Entries into the System Log: logger

The logger command comes in handy for shell scripting or for testing purposes. logger will append any message it receives to /var/log/syslog (or to /var/log/messages when logging to a remote central log server as you will see later in this lesson):

carol@debian:~$ logger this comment goes into "/var/log/syslog"

To print the last line in /var/log/syslog, use the tail command with the -1 option:

root@debian:~# tail -1 /var/log/syslog
Sep 17 17:55:33 debian carol: this comment goes into /var/log/syslog

rsyslog as a Central Log Server

To explain this topic we are going to add a new host to our setup. The layout is as follows:

Role Hostname OS IP Address

Central Log Server

suse-server

openSUSE Leap 15.1

192.168.1.6

Client

debian

Debian GNU/Linux 10 (buster)

192.168.1.4

Let us start by configuring the server. First of all, we make sure that rsyslog is up and running:

root@suse-server:~# systemctl status rsyslog
 rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-09-17 18:45:58 CEST; 7min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 832 (rsyslogd)
    Tasks: 5 (limit: 4915)
   CGroup: /system.slice/rsyslog.service
           └─832 /usr/sbin/rsyslogd -n -iNONE

openSUSE ships with a dedicated configuration file for remote logging: /etc/rsyslog.d/remote.conf. Let us enable receiving messages from clients (remote hosts) via TCP. We must uncomment the lines which load the module and start the TCP server on port 514:

# ######### Receiving Messages from Remote Hosts ##########
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imtcp.so  # load module
##$UDPServerAddress 10.10.0.1  # force to listen on this IP only
$InputTCPServerRun 514  # Starts a TCP server on selected port

# UDP Syslog Server:
#$ModLoad imudp.so  # provides UDP syslog reception
##$UDPServerAddress 10.10.0.1  # force to listen on this IP only
#$UDPServerRun 514  # start a UDP syslog server at standard port 514

Once this is done, we must restart the rsyslog service and check that the server is listening on port 514:

root@suse-server:~# systemctl restart rsyslog
root@suse-server:~# netstat -nltp | grep 514
[sudo] password for root:
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      2263/rsyslogd
tcp6       0      0 :::514                  :::*                    LISTEN      2263/rsyslogd

Next, we should open the ports in the firewall and reload the configuration:

root@suse-server:~# firewall-cmd --permanent --add-port 514/tcp
success
root@suse-server:~# firewall-cmd --reload
success
Note

With the arrival of openSUSE Leap 15.0, firewalld replaced the classic SuSEFirewall2 completely.

Templates and Filter Conditions

By default, the client’s logs will be written to the server’s /var/log/messages file — together with those of the server themselves. However, we will create a template and a filter condition to have our client’s logs stored in clear-cut directories of their own. To do so, we will add the following to /etc/rsyslog.conf (or /etc/rsyslog.d/remote.conf):

$template RemoteLogs,"/var/log/remotehosts/%HOSTNAME%/%$NOW%.%syslogseverity-text%.log"
if $FROMHOST-IP=='192.168.1.4' then ?RemoteLogs
& stop
Template

The template corresponds to the first line and allows you to specify a format for log names using dynamic file name generation. A template consists of:

  • Template directive ($template)

  • Template name (RemoteLogs)

  • Template text ("/var/log/remotehosts/%HOSTNAME%/%$NOW%.%syslogseverity-text%.log")

  • Options (optional)

Our template is called RemoteLogs and its text consists of a path in /var/log. All of our remote host’s logs will go into the remotehosts directory, where a subdirectory will be created based on the machine’s hostname (%HOSTNAME%). Each filename in this directory will consist of the date (%$NOW%), the severity (aka priority) of the message in text format (%syslogseverity-text%) and the .log suffix. The words between percent signs are properties and allow you access to the contents of the log message (date, priority, etc.). A syslog message has a number of well-defined properties which can be used in templates. These properties are accessed — and can be modified — by the so-called property replacer which implies writing them between percent signs.

Filter Condition

The remaining two lines correspond to the filter condition and its associated action:

  • Expression-Based Filter (if $FROMHOST-IP=='192.168.1.4')

  • Action (then ?RemoteLogs, & stop)

The first line checks the IP address of the remote host sending the log and — if it equals that of our Debian client — it applies the RemoteLogs template. The last line (& stop) guarantees that messages are not being sent simultaneously to /var/log/messages (but only to files in the /var/log/remotehosts directory).

Note

To learn more about templates, properties and rules, you can consult the manual page for rsyslog.conf.

With the configuration updated, we will restart rsyslog again and confirm there is no remotehosts directory in /var/log yet:

root@suse-server:~# systemctl restart rsyslog
root@suse-server:~# ls /var/log/
acpid             chrony     localmessages   pbl.log          Xorg.0.log
alternatives.log  cups       mail            pk_backend_zypp  Xorg.0.log.old
apparmor          firebird   mail.err        samba            YaST2
audit             firewall   mail.info       snapper.log      zypp
boot.log          firewalld  mail.warn       tallylog         zypper.log
boot.msg          krb5       messages        tuned
boot.omsg         lastlog    mysql           warn
btmp              lightdm    NetworkManager  wtmp

The server has now been configured. Next, we will configure the client.

Again, we must ensure that rsyslog is installed and running:

root@debian:~# sudo systemctl status rsyslog
 rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset:
   Active: active (running) since Thu 2019-09-17 18:47:54 CEST; 7min ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 351 (rsyslogd)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/rsyslog.service
           └─351 /usr/sbin/rsyslogd -n

In our sample environment we have implemented name resolution on the client by adding the line 192.168.1.6 suse-server to /etc/hosts. Thus, we can refer to the server by either name (suse-server) or IP address (192.168.1.6).

Our Debian client does not come with a remote.conf file in /etc/rsyslog.d/, so we will apply our configurations in /etc/rsyslog.conf. We will write the following line at the end of the file:

*.* @@suse-server:514

Finally, we restart rsyslog.

root@debian:~# systemctl restart rsyslog

Now, let us go back to our suse-server machine and check for the existence of remotehosts in /var/log:

root@suse-server:~# ls /var/log/remotehosts/debian/
2019-09-17.info.log  2019-09-17.notice.log

We already have two logs inside /var/log/remotehosts as described in our template. To complete this section, we run tail -f 2019-09-17.notice.log on suse-server while we send a log manually from our Debian client and confirm that messages are appended to the log file as expected (the -t option supplies a tag for our message):

root@suse-server:~# tail -f /var/log/remotehosts/debian/2019-09-17.notice.log
2019-09-17T20:57:42+02:00 debian dbus[323]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
2019-09-17T21:01:41+02:00 debian anacron[1766]: Anacron 2.3 started on 2019-09-17
2019-09-17T21:01:41+02:00 debian anacron[1766]: Normal exit (0 jobs run)
carol@debian:~$ logger -t DEBIAN-CLIENT Hi from 192.168.1.4
root@suse-server:~# tail -f /var/log/remotehosts/debian/2019-09-17.notice.log
2019-09-17T20:57:42+02:00 debian dbus[323]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
2019-09-17T21:01:41+02:00 debian anacron[1766]: Anacron 2.3 started on 2019-09-17
2019-09-17T21:01:41+02:00 debian anacron[1766]: Normal exit (0 jobs run)
2019-09-17T21:04:21+02:00 debian DEBIAN-CLIENT: Hi from 192.168.1.4

The Log Rotation Mechanism

Logs are rotated on a regular basis, which serves two main purposes:

  • Prevent older log files from using more disk space than necessary.

  • Keep logs to a manageable length for ease of consultation.

The utility in charge of log rotation (or cycling) is logrotate and its job includes actions such as moving log files to a new name, archiving and/or compressing them, sometimes emailing them to the system administrator and eventually deleting them as they grow old. There are various conventions for naming these rotated log files (adding a suffix with the date to the filename, for example); however, simply adding a suffix with an integer is the common practice:

root@debian:~# ls /var/log/messages*
/var/log/messages  /var/log/messages.1  /var/log/messages.2.gz  /var/log/messages.3.gz  /var/log/messages.4.gz

Let us now explain what will happen in the next log rotation:

  1. messages.4.gz will be deleted and lost.

  2. The content of messages.3.gz will be moved to messages.4.gz.

  3. The content of messages.2.gz will be moved to messages.3.gz.

  4. The content of messages.1 will be moved to messages.2.gz.

  5. The content of messages will be moved to messages.1 and messages will be empty and ready to register new log entries.

Note how — according to logrotate directives that you will see shortly — the three older log files are compressed, whereas the two most recent ones are not. Also, we will be retaining the logs from the last 4-5 weeks. To read messages that are 1 week old, we will consult messages.1 (and so on).

logrotate is run as an automated process or cron job daily through the script /etc/cron.daily/logrotate and reads the configuration file /etc/logrotate.conf. This file includes some global options and is well commented with each option introduced by a brief explanation of its purpose:

carol@debian:~$ sudo less /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

(...)

As you can see, the configuration files in /etc/logrotate.d for specific packages are also included. These files contain — for the most part — local definitions and specify the logfiles to rotate (remember, local definitions take precedence over global ones, and later definitions override earlier ones). What follows is an excerpt of a definition in /etc/logrotate.d/rsyslog:

/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

As you can see, every directive is separated from its value by whitespace and/or an optional equals sign (=). The lines between postrotate and endscript must appear on lines by themselves, though. The explanation is as follows:

rotate 4

Retain 4 weeks worth of logs.

weekly

Rotate log files weekly.

missingok

Do not issue an error message if the log file is missing; just go on to the next one.

notifempty

Do not rotate the log if it is empty.

compress

Compress log files with gzip (default).

delaycompress

Postpone compression of the previous log file to the next rotation cycle (only effective when used in combination with compress). It is useful when a program cannot be told to close its log file and thus might continue writing to the previous log file for some time.

sharedscripts

Related to prerotate and postrotate scripts. To prevent a script from being executed multiple times, run the scripts only once regardless of how many log files match a given pattern (e.g. /var/log/mail/*).The scripts will not be run if none of the logs in the pattern require rotating, though. On top of that, if the scripts exit with errors, any remaining actions will not be executed for any logs.

postrotate

Indicate the beginning of a postrotate script.

invoke-rc.d rsyslog rotate > /dev/null

Use /bin/sh to run invoke-rc.d rsyslog rotate > /dev/null after rotating the logs.

endscript

Indicate the end of the postrotate script.

Note

For a complete list of directives and explanations, consult the manual page for logrotate.conf.

The Kernel Ring Buffer

Since the kernel generates several messages before rsyslogd becomes available on boot, a mechanism to register those messages becomes necessary. This is where the kernel ring buffer comes into play. It is a fixed-size data structure and — therefore — as it grows with new messages, the oldest will disappear.

The dmesg command prints the kernel ring buffer. Because of the buffer’s size, this command is normally used in combination with the text filtering utility grep. For instance, to search for messages related to Universal Serial Bus devices:

root@debian:~# dmesg | grep "usb"
[    1.241182] usbcore: registered new interface driver usbfs
[    1.241188] usbcore: registered new interface driver hub
[    1.250968] usbcore: registered new device driver usb
[    1.339754] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 4.19
[    1.339756] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
(...)

Guided Exercises

  1. What utilities/commands would you use in the following scenarios:

    Purpose and log file Utility

    Read /var/log/syslog.7.gz

    Read /var/log/syslog

    Filter for the word renewal in /var/log/syslog

    Read /var/log/faillog

    Read /var/log/syslog dynamically

  2. Rearrange the following log entries in such a way that they represent a valid log message with the proper structure:

    • debian-server

    • sshd

    • [515]:

    • Sep 13 21:47:56

    • Server listening on 0.0.0.0 port 22

      The correct order is:

  3. What rules would you add to /etc/rsyslog.conf in order to accomplish each of the following:

    • Send all messages from the mail facility and a priority/severity of crit (and above) to /var/log/mail.crit:

    • Send all messages from the mail facility with priorities of alert and emergency to /var/log/mail.urgent:

    • Except for those coming from the cron and ntp facilities, send all messages — irrespective of their facility and priority — to /var/log/allmessages:

    • With all required settings properly configured first, send all messages from the mail facility to a remote host whose IP address is 192.168.1.88 using TCP and specifying the default port:

    • Irrespective of their facility, send all messages with the warning priority (only with the warning priority) to /var/log/warnings preventing excessive writing to the disk:

  4. Consider following stanza from /etc/logrotate.d/samba and explain the different options:

    carol@debian:~$ sudo head -n 11 /etc/logrotate.d/samba
    /var/log/samba/log.smbd {
            weekly
            missingok
            rotate 7
            postrotate
                    [ ! -f /var/run/samba/smbd.pid ] || /etc/init.d/smbd reload > /dev/null
            endscript
            compress
            delaycompress
            notifempty
    }
    Option Meaning

    weekly

    missingok

    rotate 7

    postrotate

    endscript

    compress

    delaycompress

    notifyempty

Explorational Exercises

  1. In section “Templates and Filter Conditions” we used an expression-based filter as a filter condition. Property-based filters are another type of filter unique to rsyslogd. Translate our expression-based filter into a property-based filter:

    Expression-Based Filter Property-Based Filter

    if $FROMHOST-IP=='192.168.1.4' then ?RemoteLogs

  2. omusrmsg is an rsyslog built-in module which facilitates notifying users (it sends log messages to the user terminal). Write a rule to send all emergency messages of all facilities to both root and the regular user carol.

Summary

In this lesson you learned:

  • Logging is crucial for system administration.

  • rsyslogd is the utility in charge of keeping logs neat and orderly.

  • Some services take care of their own logs.

  • Roughly speaking, logs can be classified into system logs and service/program logs.

  • There are a number of utilities which are convenient for log reading: less, more, zless, zmore, grep, head and tail.

  • Most log files are plain-text files; however, there are a small number of binary log files.

  • Log-wise, rsyslogd receives the relevant information from special files (sockets and memory buffers) before processing it.

  • To classify logs, rsyslogd uses rules in /etc/rsyslog.conf or /etc/rsyslog.d/*.

  • Any user may enter their own messages into the system log manually with the logger utility.

  • rsyslog allows you to keep all logs across IP networks in a centralized log server.

  • Templates come in handy for formatting log file names dynamically.

  • The purpose of log rotation is twofold: preventing old logs from using excessive disk space and making consulting logs manageable.

Answers to Guided Exercises

  1. What utilities/commands would you use in the following scenarios:

    Purpose and log file Utility

    Read /var/log/syslog.7.gz

    zmore or zless

    Read /var/log/syslog

    more or less

    Filter for the word renewal in /var/log/syslog

    grep

    Read /var/log/faillog

    faillog -a

    Read /var/log/syslog dynamically

    tail -f

  2. Rearrange the following log entries in such a way that they represent a valid log message with the proper structure:

    • debian-server

    • sshd

    • [515]:

    • Sep 13 21:47:56

    • Server listening on 0.0.0.0 port 22

      The correct order is:

      Sep 13 21:47:56 debian-server sshd[515]: Server listening on 0.0.0.0 port 22
  3. What rules would you add to /etc/rsyslog.conf in order to accomplish each of the following:

    • Send all messages from the mail facility and a priority/severity of crit (and above) to /var/log/mail.crit:

      mail.crit                 /var/log/mail.crit
    • Send all messages from the mail facility with priorities of alert and emergency to /var/log/mail.urgent:

      mail.alert                        /var/log/mail.urgent
    • Except for those coming from the cron and ntp facilities, send all messages — irrespective of their facility and priority — to /var/log/allmessages:

      *.*;cron.none;ntp.none                 /var/log/allmessages
    • With all required settings properly configured first, send all messages from the mail facility to a remote host whose IP address is 192.168.1.88 using TCP and specifying the default port:

      mail.* @@192.168.1.88:514
    • Irrespective of their facility, send all messages with the warning priority (only with the warning priority) to /var/log/warnings preventing excessive writing to the disk:

      *.=warning                        -/var/log/warnings
  4. Consider following stanza from /etc/logrotate.d/samba and explain the different options:

    carol@debian:~$ sudo head -n 11 /etc/logrotate.d/samba
    /var/log/samba/log.smbd {
            weekly
            missingok
            rotate 7
            postrotate
                    [ ! -f /var/run/samba/smbd.pid ] || /etc/init.d/smbd reload > /dev/null
            endscript
            compress
            delaycompress
            notifempty
    }
    Option Meaning

    weekly

    Rotate log files on a weekly basis.

    missingok

    Do not issue an error message if the log is missing; just continue to the next one.

    rotate 7

    Keep 7 weeks worth of backlogs.

    postrotate

    Run the script on the following line after rotating the logs.

    endscript

    Indicate the end of the postrotate script.

    compress

    Compress the logs with gzip.

    delaycompress

    In combination with compress, postpone compression to the next rotation cycle.

    notifyempty

    Do not rotate the log if it is empty.

Answers to Explorational Exercises

  1. In section “Templates and Filter Conditions” we used an expression-based filter as a filter condition. Property-based filters are another type of filter unique to rsyslogd. Translate our expression-based filter into a property-based filter:

    Expression-Based Filter Property-Based Filter

    if $FROMHOST-IP=='192.168.1.4' then ?RemoteLogs

    :fromhost-ip, isequal, "192.168.1.4" ?RemoteLogs

  2. omusrmsg is an rsyslog built-in module which facilitates notifying users (it sends log messages to the user terminal). Write a rule to send all emergency messages of all facilities to both root and the regular user carol.

    *.emerg                        :omusrmsg:root,carol

Linux Professional Insitute Inc. All rights reserved. Visit the Learning Materials website: https://learning.lpi.org
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Next Lesson

108.2 System logging (108.2 Lesson 2)

Read next lesson

Linux Professional Insitute Inc. All rights reserved. Visit the Learning Materials website: https://learning.lpi.org
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

LPI is a non-profit organization.

© 2022 Linux Professional Institute (LPI) is the global certification standard and career support organization for open source professionals. With more than 200,000 certification holders, it's the world’s first and largest vendor-neutral Linux and open source certification body. LPI has certified professionals in over 180 countries, delivers exams in multiple languages, and has hundreds of training partners.

Our purpose is to enable economic and creative opportunities for everybody by making open source knowledge and skills certification universally accessible.

  • LinkedIn
  • flogo-RGB-HEX-Blk-58 Facebook
  • Twitter
  • Contact Us
  • Privacy and Cookie Policy

Spot a mistake or want to help improve this page? Please let us know.

© 1999–2022 The Linux Professional Institute Inc. All rights reserved.