108.1 Lesson 2

Certificate:

LPIC-1 (102)

Version:

5.0

Topic:

108 Essential System Services

Objective:

108.1 Maintain system time

Lesson:

2 of 2

Introduction

While personal computers are able to keep reasonably accurate time on their own, production computing and network environments requires that very precise time be kept. The most accurate time is measured by reference clocks, which are typically atomic clocks. The modern world has devised a system where all internet-connected computer systems can be synchronised to these reference clocks using what is known as the Network Time Protocol (NTP). A computer system with NTP will be able to synchronise their system clocks to the time provided by reference clocks. If system time and the time as measured on these servers are different, then the computer will speed up or slow down its internal system time incrementally until system time matches network time.

NTP uses a hierarchical structure to disseminate time. Reference clocks are connected to servers at the top of the hierarchy. These servers are Stratum 1 machines and typically are not accessible to the public. Stratum 1 machines are however accessible to Stratum 2 machines, which are accessible to Stratum 3 machines and so on. Stratum 2 servers are accessible to the public, as are any machines lower in the hierarchy. When setting up NTP for a large network, it is good practice to have a small number of computers connect to Stratum 2+ servers, and then have those machines provide NTP to all other machines. In this way, demands on Stratum 2 machines can be minimized.

There are some important terms that come up when discussing NTP. Some of these terms are used in the commands we will use to track the status of NTP on our machines:

Offset: This refers to the absolute difference between system time and NTP time. For example, if the system clock reads 12:00:02 and NTP time reads 11:59:58, then the offset between the two clocks is four seconds.
Step: If the time offset between the NTP provider and a consumer is greater than 128ms, then NTP will perform a single significant change to system time, as opposed to slowing or speeding the system time. This is called stepping.
Slew: Slewing refers to the changes made to system time when the offset between system time and NTP is less than 128ms. If this is the case, then changes will be made gradually. This is referred to as slewing.
Insane Time: If the offset between system time and NTP time is greater than 17 minutes, then the system time is considered insane and the NTP daemon will not introduce any changes to system time. Special steps will have to be taken to bring system time within 17 minutes of proper time.
Drift: Drift refers to the phenomenon where two clocks become out of sync over time. Essentially if two clocks are initially synchronised but then become out of sync over time, then clock drift is occurring.
Jitter: Jitter refers to the amount of drift since the last time a clock was queried. So if the last NTP sync occurred 17 minutes ago, and the offset between the NTP provider and consumer is 3 milliseconds, then 3 milliseconds is the jitter.

Now we will discuss some of the specific ways that Linux implements NTP.

timedatectl

If your Linux distribution uses timedatectl, then by default it implements an SNTP client rather than a full NTP implementation. This is a less complex implementation of network time and means that your machine will not serve NTP to other connected computers.

In this case, SNTP will not work unless the timesyncd service is running. As with all systemd services, we can verify that it is running with:

$ systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Thu 2020-01-09 21:01:50 EST; 2 weeks 1 days ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 1032 (systemd-timesyn)
   Status: "Synchronized to time server for the first time 91.189.89.198:123 (ntp.ubuntu.com)."
    Tasks: 2 (limit: 4915)
   Memory: 3.0M
   CGroup: /system.slice/systemd-timesyncd.service
           └─1032 /lib/systemd/systemd-timesyncd

Jan 11 13:06:18 NeoMex systemd-timesyncd[1032]: Synchronized to time server for the first time 91.189.91.157:123 (ntp.ubuntu.com).
...

The status of timedatectl SNTP synchronisation can be verified using show-timesync:

$ timedatectl show-timesync --all
LinkNTPServers=
SystemNTPServers=
FallbackNTPServers=ntp.ubuntu.com
ServerName=ntp.ubuntu.com
ServerAddress=91.189.89.198
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=34min 8s
NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=2, Precision=-23, RootDelay=8.270ms, RootDispersion=18.432ms, Reference=91EECB0E, OriginateTimestamp=Sat 2020-01-25 18:35:49 EST, ReceiveTimestamp=Sat 2020-01-25 18:35:49 EST, TransmitTimestamp=Sat 2020-01-25 18:35:49 EST, DestinationTimestamp=Sat 2020-01-25 18:35:49 EST, Ignored=no PacketCount=263, Jitter=2.751ms }
Frequency=-211336

This configuration might be adequate for most situations, but as noted before it will be insufficient if one is hoping to synchronise several clients in a network. In this case it is recommended to install a full NTP client.

NTP Daemon

The system time is compared to network time on a regular schedule. For this to work we must have a daemon running in the background. For many Linux systems, the name of this daemon is ntpd. ntpd will allow a machine to not only be a time consumer (that is, able to sync its own clock from an outside source), but also to provide time to other machines.

Let us assume that our computer is systemd-based and it uses systemctl to control daemons. We will install ntp packages using the appropriate package manager and then ensure that our ntpd daemon is running by checking its status:

$ systemctl status ntpd

●  ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-12-06 03:27:21 EST; 7h ago
  Process: 856 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 867 (ntpd)
   CGroup: /system.slice/ntpd.service
           `-867 /usr/sbin/ntpd -u ntp:ntp -g

In some cases it might be required to both start and enable ntpd. On most Linux machines this is accomplished with:

# systemctl enable ntpd && systemctl start ntpd

NTP queries happen on TCP Port 123. If NTP fails, ensure that this port is open and listening.

NTP Configuration

NTP is able to poll several sources and to select the best candidates to use for setting system time. If a network connection is lost, NTP uses previous adjustments from its history to estimate future adjustments.

Depending on your distribution of Linux, the list of network time servers will be stored in different places. Let us assume that ntp is installed on your machine.

The file /etc/ntp.conf contains configuration information about how your system synchronises with network time. This file can be read and modified using vi or nano.

By default, the NTP servers used will be specified in a section like this:

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

The syntax for adding NTP servers looks like this:

server (IP Address)
server server.url.localhost

Server addresses can be IP addresses or URLs if DNS has been properly configured. In this case, the server will always be queried.

A network administrator might also consider using (or setting up) a pool. In this case, we assume that there are several NTP providers, all running NTP daemons and with the same time. When a client queries a pool, a provider is selected at random. This helps to distribute network load among many machines so that no one machine in the pool is handling all NTP queries.

Commonly, /etc/ntp.conf will be populated with a server pool called pool.ntp.org. So for example, server 0.centos.pool.ntp.org is a default NTP pool provided to CentOS machines.

pool.ntp.org

The NTP servers used by default are an open source project. More information can be found at ntppool.org.

Consider if the NTP Pool is appropriate for your use. If business, organization or human life depends on having correct time or can be harmed by it being wrong, you shouldn’t “just get it off the internet”. The NTP Pool is generally very high quality, but it is a service run by volunteers in their spare time. Please talk to your equipment and service vendors about getting local and reliable service setup for you. See also our terms of service. We recommend time servers from Meinberg, but you can also find time servers from End Run, Spectracom and many others.

— ntppool.org

ntpdate

During initial setup, system time and NTP might be seriously de-synchronised. If the offset between system and NTP time is greater than 17 minutes, then the NTP daemon will not make changes to system time. In this scenario manual intervention will be required.

Firstly, if ntpd is running it will be necessary to stop the service. Use systemctl stop ntpd to do so.

Next, use ntpdate pool.ntp.org to perform an initial one-time synchronisation, where pool.ntp.org refers to the IP address or URL of an NTP server. More than one sync may be required.

ntpq

ntpq is a utility for monitoring the status of NTP. Once the NTP daemon has been started and configured, ntpq can be used to check on its status:

$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+37.44.185.42    91.189.94.4      3 u   86  128  377  126.509  -20.398   6.838
+ntp2.0x00.lv    193.204.114.233  2 u   82  128  377  143.885   -8.105   8.478
*inspektor-vlan1 121.131.112.137  2 u   17  128  377  112.878  -23.619   7.959
 b1-66er.matrix. 18.26.4.105      2 u  484  128   10   34.907   -0.811  16.123

In this case -p is for print and it will print a summary of peers. Host addresses can also be returned as IP addresses using -n.

remote: hostname of the NTP provider.
refid: Reference ID of the NTP provider.
st: Stratum of the provider.
when: Number of seconds since the last query.
poll: Number of seconds between queries.
reach: Status ID to indicate whether a server was reached. Successful connections will increase this number by 1.
delay: Time in ms between query and response by the server.
offset: Time in ms between system time and NTP time.
jitter: Offset in ms between system time and NTP in the last query.

ntpq also has an interactive mode, which is accessed when it is run without options or argument. The ? option will return a list of commands that ntpq will recognize.

chrony

chrony is yet another way to implement NTP. It is installed by default on some Linux systems, but is available to download on all major distributions. chronyd is the chrony daemon, and chronyc is the command line interface. It may be required to start and enable chronyd before interacting with chronyc.

If the chrony installation has a default configuration then using the command chronyc tracking will provide information about NTP and system time:

$ chronyc tracking
Reference ID    : 3265FB3D (bras-vprn-toroon2638w-lp130-11-50-101-251-61.dsl.)
Stratum         : 3
Ref time (UTC)  : Thu Jan 09 19:18:35 2020
System time     : 0.000134029 seconds fast of NTP time
Last offset     : +0.000166506 seconds
RMS offset      : 0.000470712 seconds
Frequency       : 919.818 ppm slow
Residual freq   : +0.078 ppm
Skew            : 0.555 ppm
Root delay      : 0.006151616 seconds
Root dispersion : 0.010947504 seconds
Update interval : 129.8 seconds
Leap status     : Normal

This output contains a lot of information, more than what is available from other implementations.

Reference ID: The reference ID and name to which the computer is currently synced.
Stratum: Number of hops to a computer with an attached reference clock.
Ref time: This is the UTC time at which the last measurement from the reference source was made.
System time: Delay of system clock from synchronized server.
Last offset: Estimated offset of the last clock update.
RMS offset: Long term average of the offset value.
Frequency: This is the rate by which the system’s clock would be wrong if chronyd is not correcting it. It is provided in ppm (parts per million).
Residual freq: Residual frequency indicating the difference between the measurements from reference source and the frequency currently being used.
Skew: Estimated error bound of the frequency.
Root delay: Total of the network path delays to the stratum computer, from which the computer is being synced.
Leap status: This is the leap status which can have one of the following values – normal, insert second, delete second or not synchronized.

We can also look at detailed information about the last valid NTP update:

# chrony ntpdata
Remote address  : 172.105.97.111 (AC69616F)
Remote port     : 123
Local address   : 192.168.122.81 (C0A87A51)
Leap status     : Normal
Version         : 4
Mode            : Server
Stratum         : 2
Poll interval   : 6 (64 seconds)
Precision       : -25 (0.000000030 seconds)
Root delay      : 0.000381 seconds
Root dispersion : 0.000092 seconds
Reference ID    : 61B7CE58 ()
Reference time  : Mon Jan 13 21:50:03 2020
Offset          : +0.000491960 seconds
Peer delay      : 0.004312567 seconds
Peer dispersion : 0.000000068 seconds
Response time   : 0.000037078 seconds
Jitter asymmetry: +0.00
NTP tests       : 111 111 1111
Interleaved     : No
Authenticated   : No
TX timestamping : Daemon
RX timestamping : Kernel
Total TX        : 15
Total RX        : 15
Total valid RX  : 15

Finally, chronyc sources will return information about the NTP servers that are used to synchronise time:

$ chronyc sources
210 Number of sources = 0
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================

At the moment, this machine has no sources configured. We can add sources from pool.ntp.org by opening the chrony configuration file. This will usually be located at /etc/chrony.conf. When we open this file, we should see that some servers are listed by default:

210 Number of sources = 0
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
# Most computers using chrony will send measurement requests to one or
# more 'NTP servers'.  You will probably find that your Internet Service
# Provider or company have one or more NTP servers that you can specify.
# Failing that, there are a lot of public NTP servers.  There is a list
# you can access at http://support.ntp.org/bin/view/Servers/WebHome or
# you can use servers from the 3.arch.pool.ntp.org project.

! server 0.arch.pool.ntp.org iburst iburst
! server 1.arch.pool.ntp.org iburst iburst
! server 2.arch.pool.ntp.org iburst iburst

! pool 3.arch.pool.ntp.org iburst

These servers will also serve as a syntax guide when entering our own servers. However, in this case we will simply remove the ! s at the beginning of each line, thus uncommenting out these lines and using the default servers from the pool.ntp.org project.

In addition, in this file we can choose to change the default configuration regarding skew and drift as well as the location of the driftfile and keyfile.

On this machine, we need to make a large initial clock correction. We will choose to uncomment the following line:

! makestep 1.0 3

After making changes to the configuration file, restart the chronyd service and then use chronyc makestep to manually step the system clock:

# chronyc makestep
200 OK

And then use chronyc tracking as before to verify that the changes have taken place.

Guided Exercise

Enter the appropriate term for each definition:

Definition

Term

A computer that will share network time with you

Distance from a reference clock, in hops or steps

Difference between system time and network time

Difference between system time and network time since the last NTP poll

Group of servers that provide network time and share the load between them

Specify which of the commands you would use to output the following values:

Value chronyc tracking timedatectl show-timesync --all ntpq -pn chrony ntpdata chronyc sources

Jitter

Drift

Interval of Poll

Offset

Stratum

IP Address of Provider

Root Delay

You are setting up an enterprise network consisting of a Linux server and several Linux desktops. The server has a static IP address of 192.168.0.101. You decide that the server will connect to pool.ntp.org and then provide NTP time to the desktops. Describe the configuration of the server and of the desktops.
A Linux machine has the incorrect time. Describe the steps you would take to troubleshoot NTP.

Explorational Exercise

Research the differences between SNTP and NTP.

SNTP NTP
Why might a system administrator choose not to use pool.ntp.org?
How would a system administrator choose to join or otherwise contribute to the pool.ntp.org project?

Summary

In this lesson you learned:

What NTP is and why it is important.
Configuring the NTP daemon from the pool.ntp.org project.
Using ntpq to verify NTP configuration.
Using chrony as an alternative NTP service.

Commands used in this lesson:

timedatectl show-timesync --all: Display SNTP information if using timedatectl.
ntpdate <address>: Perform a manual one-time NTP step update.
ntpq -p: Print a history of NTP recent polls. -n will replace URLs with IP addresses.
chronyc tracking: Displays NTP status if using chrony.
chronyc ntpdata: Displays NTP information about the last poll.
chronyc sources: Displays informations about NTP providers.
chronyc makestep: Perform a manual one-time NTP step update if using chrony.

Answers Guided Exercise

Enter the appropriate term for each definition:

Definition

Term

A computer that will share network time with you

Provider

Distance from a reference clock, in hops or steps

Stratum

Difference between system time and network time

Offset

Difference between system time and network time since the last NTP poll

Jitter

Group of servers that provide network time and share the load between them

Pool

Specify which of the commands you would use to output the following values:

Value chronyc tracking timedatectl show-timesync --all ntpq -pn chrony ntpdata chronyc sources

Jitter

Drift

Interval of Poll

X (when column)

Offset

Stratum

IP Address of Provider

Root Delay

You are setting up an enterprise network consisting of a Linux server and several Linux desktops. The server has a static IP address of 192.168.0.101. You decide that the server will connect to pool.ntp.org and then provide NTP time to the desktops. Describe the configuration of the server and of the desktops.

Ensure that the server has an ntpd service running, rather than SNTP. Use pool.ntp.org pools in the /etc/ntp.conf or /etc/chrony.conf file. For each client, specify 192.168.0.101 in each /etc/ntp.conf or /etc/chrony.conf file.
A Linux machine has the incorrect time. Describe the steps you would take to troubleshoot NTP.

First, ensure that the machine is connected to the Internet. Use ping for this. Check that an ntpd or SNTP service is running using systemctl status ntpd or systemctl status systemd-timesyncd. You may see error messages that provide useful information. Finally, use a command such as ntpq -p or chrony tracking to verify if any requests have been made. If the system time is drastically different from network time, it may be that system time is considered “insane” and will not be changed without manual intervention. In this case, use a command from the previous lesson or a command such as ntpdate pool.ntp.org to perform a one-time ntp synchronisation.

Answers to Explorational Exercises

Research the differences between SNTP and NTP.

SNTP

NTP

less accurate

more accurate

requires fewer resources

requires more resources

cannot act as a time provider

can act as a time provider

steps time only

steps or slews time

requests time from a single source

can monitor multiple NTP servers and use the optimal provider

Why might a system administrator choose not to use pool.ntp.org?

From ntppool.org: If it is absolutely crucial to have correct time, you should consider an alternative. Similarly, if your Internet provider has a time server, it is recommended to use that instead.
How would a system administrator choose to join or otherwise contribute to the pool.ntp.org project?

From www.ntppool.org: Your server must have a static IP address and a permanent internet connection. The static IP address must not change at all or at least less than once a year. Beyond that, the bandwidth requirements are modest: 384 - 512 Kbit bandwidth. Stratum 3 or 4 servers are welcome to join.