4.3 Lesson 2

Certificate:

Linux Essentials

Version:

1.6

Topic:

4 The Linux Operating System

Objective:

4.3 Where Data is Stored

Lesson:

2 of 2

Introduction

After exploring programs and their configuration files, in this lesson we will learn how commands are executed as processes. Likewise, we will be commenting on the system messaging, the use of the kernel ring buffer and how the arrival of systemd and its journal daemon — journald — changed the ways things had been done so far regarding system logging.

Processes

Every time a user issues a command, a program is run and one or more processes are generated.

Processes exist in a hierarchy. After the kernel is loaded in memory on boot, the first process is initiated which — in turn — starts other processes, which, again, can start other processes. Every process has a unique identifier (PID) and parent process identifier (PPID). These are positive integers that are assigned in sequential order.

Exploring Processes Dynamically: `top`

You can get a dynamic listing of all running processes with the top command:

$ top

top - 11:10:29 up  2:21,  1 user,  load average: 0,11, 0,20, 0,14
Tasks:  73 total,   1 running,  72 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,0 us,  0,3 sy,  0,0 ni, 99,7 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem :  1020332 total,   909492 free,    38796 used,    72044 buff/cache
KiB Swap:  1046524 total,  1046524 free,        0 used.   873264 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
   436 carol     20   0   42696   3624   3060 R  0,7  0,4   0:00.30 top
     4 root      20   0       0      0      0 S  0,3  0,0   0:00.12 kworker/0:0
   399 root      20   0   95204   6748   5780 S  0,3  0,7   0:00.22 sshd
     1 root      20   0   56872   6596   5208 S  0,0  0,6   0:01.29 systemd
     2 root      20   0       0      0      0 S  0,0  0,0   0:00.00 kthreadd
     3 root      20   0       0      0      0 S  0,0  0,0   0:00.02 ksoftirqd/0
     5 root       0 -20       0      0      0 S  0,0  0,0   0:00.00 kworker/0:0H
     6 root      20   0       0      0      0 S  0,0  0,0   0:00.00 kworker/u2:0
     7 root      20   0       0      0      0 S  0,0  0,0   0:00.08 rcu_sched
     8 root      20   0       0      0      0 S  0,0  0,0   0:00.00 rcu_bh
     9 root      rt   0       0      0      0 S  0,0  0,0   0:00.00 migration/0
    10 root       0 -20       0      0      0 S  0,0  0,0   0:00.00 lru-add-drain
    (...)

As we saw above, top can also give us information about memory and CPU consumption of the overall system as well as for each process.

top allows the user some interaction.

By default, the output is sorted by the percentage of CPU time used by each process in descending order. This behavior can be modified by pressing the following keys from within top:

M: Sort by memory usage.
N: Sort by process ID number.
T: Sort by running time.
P: Sort by percentage of CPU usage.

To switch between descending/ascending order just press R.

Tip	A fancier and more user-friendly version of `top` is `htop`. Another — perhaps more exhaustive — alternative is `atop`. If not already installed in your system, I encourage you to use your package manager to install them and give them a try.

A Snapshot of Processes: `ps`

Another very useful command to get information about processes is ps. Whereas top provides dynamic information, that of ps is static.

If invoked without options, the output of ps is quite discrete and relates only to the processes attached to the current shell:

$ ps
  PID TTY          TIME CMD
 2318 pts/0    00:00:00 bash
 2443 pts/0    00:00:00 ps

The displayed information has to do with the process identifier (PID), the terminal in which the process is run (TTY), the CPU time taken by the process (TIME) and the command which started the process (CMD).

A useful switch for ps is -f which shows the full-format listing:

$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
carol     2318  1682  0 08:38 pts/1    00:00:00 bash
carol     2443  2318  0 08:46 pts/1    00:00:00 ps -f

In combination with other switches, -f shows the relationship between parent and child processes:

$ ps -uf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
carol       2318  0.0  0.1  21336  5140 pts/1    Ss   08:38   0:00 bash
carol       2492  0.0  0.0  38304  3332 pts/1    R+   08:51   0:00  \_ ps -uf
carol       1780  0.0  0.1  21440  5412 pts/0    Ss   08:28   0:00 bash
carol       2291  0.0  0.7 305352 28736 pts/0    Sl+  08:35   0:00  \_ emacs index.en.adoc -nw
(...)

Likewise, ps can show the percentage of memory used when invoked with the -v switch:

$ ps -v
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 1163 tty2     Ssl+   0:00      1    67 201224 5576  0.1 /usr/lib/gdm3/gdm-x-session (...)
 (...)

Note	Another visually attractive command that shows the hierarchy of processes is `pstree`. It ships with all major distributions.

Process Information in the `/proc` Directory

We have already seen the /proc filesystem. /proc includes a numbered subdirectory for every running process in the system (the number is the PID of the process):

carol@debian:~# ls /proc
1    108  13   17   21   27   354  41   665  8    9
10   109  14   173  22   28   355  42   7    804  915
103  11   140  18   23   29   356  428  749  810  918
104  111  148  181  24   3    367  432  75   811
105  112  149  19   244  349  370  433  768  83
106  115  15   195  25   350  371  5    797  838
107  12   16   2    26   353  404  507  798  899
(...)

Thus, all the information about a particular process is included within its directory. Let us list the contents of the first process — that whose PID is 1 (the output has been truncated for readability):

# ls /proc/1/
attr        cmdline          environ  io        mem	      ns
autogroup   comm             exe      limits    mountinfo numa_maps
auxv        coredump_filter  fd       loginuid  mounts    oom_adj
...

You can check — for instance — the process executable:

# cat /proc/1/cmdline; echo
/sbin/init

As you can see, the binary that started the hierarchy of processes was /sbin/init.

Note	Commands can be concatenated with the semicolon (`;`). The point in using the `echo` command above is to provide a newline. Try and run simply `cat /proc/1/cmdline` to see the difference.

The System Load

Each process on a system can potentially consume system resources. The so-called system load tries to aggregate the overall load of the system into a single numeric indicator. You can see the current load with the command uptime:

$ uptime
 22:12:54 up 13 days, 20:26,  1 user,  load average: 2.91, 1.59, 0.39

The three last digits indicate the system’s load average for the last minute (2.91), the last five minutes (1.59) and the last fifteen minutes (0.39), respectively.

Each of these numbers indicates how many processes were waiting either for CPU resources or for input/output operations to complete. This means that these processes were ready to run if they had received the respective resources.

System Logging and System Messaging

As soon as the kernel and the processes start executing and communicating with each other, a lot of information is produced. Most of it is sent to files — the so-called log files or, simply, logs.

Without logging, searching for an event that happened on a server would give sysadmins many a headache, hence the importance of having a standardized and centralized way of keeping track of any system events. Besides, logs are determinant and telling when it comes to troubleshooting and security as well as reliable data sources for understanding system statistics and making trend predictions.

Logging with the syslog Daemon

Traditionally, system messages have been managed by the standard logging facility — syslog — or any of its derivatives — syslog-ng or rsyslog. The logging daemon collects messages from other services and programs and stores them in log files, typically under /var/log. However, some services take care of their own logs (take — for example — the Apache HTTPD web server). Likewise, the Linux kernel uses an in-memory ring buffer for storing its log messages.

Log Files in `/var/log`

Because logs are data that varies over time, they are normally found in /var/log.

If you explore /var/log, you will realize that the names of logs are — to a certain degree — quite self-explanatory. Some examples include:

/var/log/auth.log: It stores information about authentication.
/var/log/kern.log: It stores kernel information.
/var/log/syslog: It stores system information.
/var/log/messages: It stores system and application data.

Note	The exact name and contents of log files may vary across Linux distributions.

Accessing Log Files

When exploring log files, remember to be root (if you do not have reading permissions) and use a pager such as less;

# less /var/log/messages
Jun  4 18:22:48 debian liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="285" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jun 29 16:57:10 debian kernel: [    0.000000] Linux version 4.9.0-8-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.130-2 (2018-10-27)
Jun 29 16:57:10 debian kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-8-amd64 root=/dev/sda1 ro quiet

Alternatively, you can use tail with the -f switch to read the most recent messages of the file and dynamically show new lines as they are appended:

# tail -f /var/log/messages
Jul  9 18:39:37 debian kernel: [    2.350572] RAPL PMU: hw unit of domain psys 2^-0 Joules
Jul  9 18:39:37 debian kernel: [    2.512802] input: VirtualBox USB Tablet as /devices/pci0000:00/0000:00:06.0/usb1/1-1/1-1:1.0/0003:80EE:0021.0001/input/input7
Jul  9 18:39:37 debian kernel: [    2.513861] Adding 1046524k swap on /dev/sda5.  Priority:-1 extents:1 across:1046524k FS
Jul  9 18:39:37 debian kernel: [    2.519301] hid-generic 0003:80EE:0021.0001: input,hidraw0: USB HID v1.10 Mouse [VirtualBox USB Tablet] on usb-0000:00:06.0-1/input0
Jul  9 18:39:37 debian kernel: [    2.623947] snd_intel8x0 0000:00:05.0: white list rate for 1028:0177 is 48000
Jul  9 18:39:37 debian kernel: [    2.914805] IPv6: ADDRCONF(NETDEV_UP): enp0s3: link is not ready
Jul  9 18:39:39 debian kernel: [    4.937283] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul  9 18:39:39 debian kernel: [    4.938493] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s3: link becomes ready
Jul  9 18:39:40 debian kernel: [    5.315603] random: crng init done
Jul  9 18:39:40 debian kernel: [    5.315608] random: 7 urandom warning(s) missed due to ratelimiting

You will find the output into the following format:

Timestamp
Hostname from which the message came from
Name of program/service that generated the message
The PID of the program that generated the message
Description of the action that took place

Most log files are written in plain text; however, a few can contain binary data as is the case with /var/log/wtmp — which stores data relevant to successful logins. You can use the file command to determine which is the case:

$ file /var/log/wtmp
/var/log/wtmp: dBase III DBT, version number 0, next free block index 8

These files are normally read using special commands. last is used to interpret the data in /var/log/wtmp:

$ last
carol    tty2         :0               Thu May 30 10:53   still logged in
reboot   system boot  4.9.0-9-amd64    Thu May 30 10:52   still running
carol    tty2         :0               Thu May 30 10:47 - crash  (00:05)
reboot   system boot  4.9.0-9-amd64    Thu May 30 09:11   still running
carol    tty2         :0               Tue May 28 08:28 - 14:11  (05:42)
reboot   system boot  4.9.0-9-amd64    Tue May 28 08:27 - 14:11  (05:43)
carol    tty2         :0               Mon May 27 19:40 - 19:52  (00:11)
reboot   system boot  4.9.0-9-amd64    Mon May 27 19:38 - 19:52  (00:13)
carol    tty2         :0               Mon May 27 19:35 - down   (00:03)
reboot   system boot  4.9.0-9-amd64    Mon May 27 19:34 - 19:38  (00:04)

Note	Similar to `/var/log/wtmp`, `/var/log/btmp` stores information about failed login attempts and the special command to read its contents is `lastb`.

Log Rotation

Log files can grow a lot over a few weeks or months and take up all free disk space. To tackle this, the utility logrotate is used. It implements log rotation or cycling which implies actions such as moving log files to a new name, archiving and/or compressing them, sometimes emailing them to the sysadmin and eventually deleting them as they grow old. The conventions used for naming these rotated log files are diverse (adding a suffix with the date, for example); however, simply adding a suffix with an integer is commonplace:

# ls /var/log/apache2/
access.log  error.log  error.log.1  error.log.2.gz  other_vhosts_access.log

Note how error.log.2.gz has already been compressed with gunzip (hence the .gz suffix).

The Kernel Ring Buffer

The kernel ring buffer is a fixed-size data structure that records kernel boot messages. The function of this buffer — a very important one — is that of logging all the kernel messages produced on boot — when syslog is not yet available. The dmesg command prints the kernel ring buffer (which used to be also stored in /var/log/dmesg). Because of the extension of the ring buffer, this command is normally used in combination with the text filtering utility grep (remember, you must also be root). For instance, to search for boot messages:

$ dmesg | grep boot
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-9-amd64 root=UUID=5216e1e4-ae0e-441f-b8f5-8061c0034c74 ro quiet
[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-9-amd64 root=UUID=5216e1e4-ae0e-441f-b8f5-8061c0034c74 ro quiet
[    0.144986] AppArmor: AppArmor disabled by boot time parameter
(...)

Note	As the kernel ring buffer grows with new messages over time, the oldest ones will fade away.

The System Journal: systemd-journald

As of 2015, systemd replaced SysV Init as a de facto system and service manager in most major Linux distributions. As a consequence, the journal daemon — journald — has become the standard logging component, superseding syslog in most aspects. The data is no longer stored in plain text but in binary form. Thus, the journalctl utility is necessary to read the logs. On top of that, journald is syslog compatible and can be integrated with syslog.

journalctl is the utility that you have to use to query the systemd’s journal. You have to either be root or use sudo to invoke it. If invoked without options, it prints the entire journal:

# journalctl
-- Logs begin at Tue 2019-06-04 17:49:40 CEST, end at Tue 2019-06-04 18:13:10 CEST. --
jun 04 17:49:40 debian systemd-journald[339]: Runtime journal (/run/log/journal/) is 8.0M, max 159.6M, 151.6M free.
jun 04 17:49:40 debian kernel: microcode: microcode updated early to revision 0xcc, date = 2019-04-01
Jun 04 17:49:40 debian kernel: Linux version 4.9.0-8-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) )
Jun 04 17:49:40 debian kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-8-amd64 root=/dev/sda1 ro quiet
(...)

However, if invoked with the -k or --dmesg switches, it will be equivalent to using the dmesg command:

# journalctl -k
[    0.000000] Linux version 4.9.0-9-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-9-amd64 root=UUID=5216e1e4-ae0e-441f-b8f5-8061c0034c74 ro quiet
(...)

Other interesting options for journalctl include:

-b, --boot: It shows boot information.
-u: It shows messages about a specified unit. Roughly, a unit can be defined as any resource handled by systemd. For example journalctl -u apache2.service is used to read messages about the apache2 web server.
-f: It shows most recent journal messages and keeps printing new entries as they are appended to the journal — much like tail -f.

Guided Exercises

Have a look at the following listing of top and answer the following questions:

carol@debian:~$ top

top - 13:39:16 up 31 min,  1 user,  load average: 0.12, 0.15, 0.10
Tasks:  73 total,   2 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.1 us,  0.4 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1020332 total,   698700 free,   170664 used,   150968 buff/cache
KiB Swap:  1046524 total,  1046524 free,        0 used.   710956 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  605 nobody    20   0 1137620 132424  34256 S  6.3 13.0   1:47.24 ntopng
  444 www-data  20   0  364780   4132   2572 S  0.3  0.4   0:00.44 apache2
  734 root      20   0   95212   7004   6036 S  0.3  0.7   0:00.36 sshd
  887 carol     20   0   46608   3680   3104 R  0.3  0.4   0:00.03 top
    1 root      20   0   56988   6688   5240 S  0.0  0.7   0:00.42 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:00.09 ksoftirqd/0
    4 root      20   0       0      0      0 S  0.0  0.0   0:00.87 kworker/0:0
(...)

Which processes have been started by the user carol?
What virtual directory of /proc should you visit to search for data regarding the top command?
What process was run first? How can you tell?
Complete the table specifying in what area of top output the following information is found:

Information about … Summary Area Task Area

Memory

Swap

PID

CPU time

Commands

What command is used to read the following binary logs?
- /var/log/wtmp
- /var/log/btmp
- /run/log/journal/2a7d9730cd3142f4b15e20d6be631836/system.journal
In combination with grep, what commands would you use to find out the following information about your Linux system?
- When the system was last rebooted (wtmp)
- Which hard disks are installed (kern.log)
- When the last login occurred (auth.log)
What two commands would you use to have the kernel ring buffer displayed?
Indicate where the following log messages belong:
- Jul 10 13:37:39 debian dbus[303]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  /var/log/syslog
  
  /var/log/messages
- Jul 10 11:23:58 debian kernel: [ 1.923349] usbhid: USB HID core driver
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  /var/log/syslog
  
  /var/log/messages
- Jul 10 14:02:53 debian sudo: pam_unix(sudo:session): session opened for user root by carol(uid=0)
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  /var/log/syslog
  
  /var/log/messages
- Jul 10 11:23:58 debian NetworkManager[322]: <info> [1562750638.8672] NetworkManager (version 1.6.2) is starting…
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  /var/log/syslog
  
  /var/log/messages
Have journalctl query information about the following units?

Unit Command

ssh

networking

rsyslog

cron

Explorational Exercises

Reconsider the top output from the guided exercises and answer the following questions:
- What two steps would you follow to kill the apache web server?
- In the summary area, how could you display the information about physical memory and swap using progress bars?
- Now, sort the processes by memory usage:
- Now that you have memory information displayed in progress bars and processes sorted by memory usage, save these configurations so that you get them as default next time you use top:
- What file stores top’s configuration settings? Where does it live? How can you check for its existence?
Learn about the command exec in Bash. Try to demonstrate its functionality by starting a Bash session, finding the Bash process with ps, then run exec /bin/sh and search for the process with the same PID again.
Follow these steps to explore kernel events and udev’s dynamic management of devices:
- Hotplug a USB drive into your computer. Run dmesg and pay attention to the last lines. What is the most recent line?
- Bearing in mind the output from the previous command, run ls /dev/sd* and make sure your USB drive appears in the listing. What is the output?
- Now remove the USB drive and run dmesg again. How does the most recent line read?
- Run ls /dev/sd* again and make sure your device disappeared from the listing. What is the output?

Summary

In the context of data storage, the following topics have been discussed in this lesson: process management and system logging and messaging.

Regarding process management, we have learned the following:

Programs generate processes and processes exist in a hierarchy.
Every process has a unique identifier (PID) and a parent process identifier (PPID).
top is a very useful command to dynamically and interactively explore the running processes of the system.
ps can be used to obtain a snapshot of the current running processes in the system.
The /proc directory includes directories for every running process in the system named after their PIDs.
The concept of system load average — which is very useful to check on CPU utilization/overloading.

Concerning system logging, we must remember that:

A log is a file where system events are recorded. Logs are invaluable when it comes to troubleshooting.
Logging has traditionally been handled by special services such as syslog, syslog-ng or rsyslog. Nevertheless, some programs use their own logging daemons.
Because logs are variable data, they are kept in /var and — sometimes — their names can give you a clue about their content (kern.log, auth.log, etc.)
Most logs are written in plain text and can be read with any text editor as long as you have the right permissions. However, a few of them are binary and must be read using special commands.
To avoid problems with disk space, log rotation is carried out by the logrotate utility.
As for the kernel, it uses a circular data structure — the ring buffer — where boot messages are kept (old messages fade away over time).
The system and service manager systemd replaced System V init in virtually all distros with journald becoming the standard logging service.
To read systemd’s journal, the journalctl utility is needed.

Commands used in this lesson:

cat: Concatenate/print file content.
dmesg: Print the kernel ring buffer.
echo: Display a line of text or a newline.
file: Determine file type.
grep: Print lines matching a pattern.
last: Show a listing of last logged in users.
less: Display contents of file one page at a time.
ls: List directory contents.
journalctl: Query the systemd journal.
tail: Display the last lines of a file.

Answers to Guided Exercises

Have a look the following listing of top and answer the following questions:

carol@debian:~$ top

top - 13:39:16 up 31 min,  1 user,  load average: 0.12, 0.15, 0.10
Tasks:  73 total,   2 running,  71 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.1 us,  0.4 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1020332 total,   698700 free,   170664 used,   150968 buff/cache
KiB Swap:  1046524 total,  1046524 free,        0 used.   710956 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
  605 nobody    20   0 1137620 132424  34256 S  6.3 13.0   1:47.24 ntopng
  444 www-data  20   0  364780   4132   2572 S  0.3  0.4   0:00.44 apache2
  734 root      20   0   95212   7004   6036 S  0.3  0.7   0:00.36 sshd
  887 carol     20   0   46608   3680   3104 R  0.3  0.4   0:00.03 top
    1 root      20   0   56988   6688   5240 S  0.0  0.7   0:00.42 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:00.09 ksoftirqd/0
    4 root      20   0       0      0      0 S  0.0  0.0   0:00.87 kworker/0:0
(...)

Which processes have been started by the user carol?

Answer: Only one: top.
What virtual directory of /proc should you visit to search for data regarding the top command?

Answer: /proc/887
What process was run first? How can you tell?

Answer: systemd. Because it is the one with PID #1.
Complete the table specifying in what area of top output the following information is found:

Information about … Summary Area Task Area

Memory

Yes

Yes

Swap

Yes

No

PID

No

Yes

CPU time

Yes

Yes

Commands

No

Yes

What command is used to read the following binary logs?
- /var/log/wtmp
  
  Answer: last
- /var/log/btmp
  
  Answer: lastb
- /run/log/journal/2a7d9730cd3142f4b15e20d6be631836/system.journal
  
  Answer: journalctl
In combination with grep, what commands would you use to find out the following information about your Linux system?
- When the system was last rebooted (wtmp)
  
  Answer: last
- Which hard disk are installed (kern.log)
  
  Answer: less /var/log/kern.log
- When the last login occurred (auth.log)
  
  Answer: less /var/log/auth.log
What two commands would you use to have the kernel ring buffer displayed?

dmesg and journalctl -k (also journalctl --dmesg).
Indicate where the following log messages belong:
- Jul 10 13:37:39 debian dbus[303]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  /var/log/syslog
  
  X
  
  /var/log/messages
- Jul 10 11:23:58 debian kernel: [ 1.923349] usbhid: USB HID core driver
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  X
  
  /var/log/syslog
  
  /var/log/messages
  
  X
  
  Jul 10 14:02:53 debian sudo: pam_unix(sudo:session): session opened for user root by carol(uid=0)
  
  /var/log/auth.log
  
  X
  
  /var/log/kern.log
  
  /var/log/syslog
  
  /var/log/messages
- Jul 10 11:23:58 debian NetworkManager[322]: <info> [1562750638.8672] NetworkManager (version 1.6.2) is starting…
  
  /var/log/auth.log
  
  /var/log/kern.log
  
  /var/log/syslog
  
  /var/log/messages
  
  X
Have journalctl query information about the following units:

Unit Command

ssh

journalctl -u ssh.service

networking

journalctl -u networking.service

rsyslog

journalctl -u rsyslog.service

cron

journalctl -u cron.service

Answers to Explorational Exercises

Reconsider the top output from the guided exercises and answer the following questions:
- What two steps would you follow to kill the apache web server?
  
  First, press k; then provide a kill value.
- In the summary area, how could you display the information about physical memory and swap using progress bars?
  
  By pressing m once or twice.
- Now, sort the processes by memory usage:
  
  M
- Now that you have memory information displayed in progress bars and processes sorted by memory usage, save these configurations so that you get them as default next time you use top:
  
  W
- What file stores top’s configuration settings? Where does it live? How can you check for its existence?
  
  The file is ~/.config/procps/toprc and lives in the user’s home directory (~). Since it is a hidden file (it resides in a directory whose name starts with a dot), we can check for its existence with ls -a (list all files). This file can be generated by pressing kbd:[Shift+W] while in top.
Learn about the command exec in Bash. Try to demonstrate its functionality by starting a Bash session, finding the Bash process with ps, then run exec /bin/sh and search for the process with the same PID again.

exec replaces a process with another command. In the following example we can see that the Bash process is replaced by /bin/sh (instead of /bin/sh becoming a child process):
```
$ echo $$
19877
$ ps auxf | grep 19877 | head -1
carol  19877  0.0  0.0   7448  3984 pts/25   Ss   21:17   0:00  \_ bash
$ exec /bin/sh
sh-5.0$ ps auxf | grep 19877 | head -1
carol  19877  0.0  0.0   7448  3896 pts/25   Ss   21:17   0:00  \_ /bin/sh
```
Follow these steps to explore kernel events and udev’s dynamic management of devices:
- Hotplug a USB drive into your computer. Run dmesg and pay attention to the last lines. What is the most recent line?
  
  You should get something along the lines of [ 1967.700468] sd 6:0:0:0: [sdb] Attached SCSI removable disk.
- Bearing in mind the output from the previous command, run ls /dev/sd* and make sure your USB drive appears in the listing. What is the output?
  
  Depending on the number of devices connected to your system, you should get something like /dev/sda /dev/sda1 /dev/sdb /dev/sdb1 /dev/sdb2. In our case, we find our USB drive (/dev/sdb) and its two partitions (/dev/sdb1 and /dev/sdb2).
- Now remove the USB drive and run dmesg again. How does the most recent line read?
  
  You should get something along the lines of [ 2458.881695] usb 1-9: USB disconnect, device number 6.
- Run ls /dev/sd* again and make sure your device disappeared from the listing. What is the output?
  
  In our case, /dev/sda /dev/sda1.