110.3 Lesson 1

Certificate:

LPIC-1

Version:

5.0

Topic:

110 Security

Objective:

110.3 Securing data with encryption

Lesson:

1 of 2

Introduction

Securing data with encryption is of paramount importance in many aspects of today’s system administration — even more so when it comes to accessing systems remotely. As opposed to insecure solutions such as telnet, rlogin or FTP, the SSH (Secure Shell) protocol was designed with security in mind. Using public key cryptography, it authenticates both hosts and users and encrypts all subsequent information exchange. Furthermore, SSH can be used to establish port tunnels, which — amongst other things — allows for a non-encrypted protocol to transmit data over an encrypted SSH connection. The current, recommended version of the SSH protocol is 2.0. OpenSSH is a free and open source implementation of the SSH protocol.

This lesson will cover basic OpenSSH client configuration as well as the role of OpenSSH server host keys. The concept of SSH port tunnels will also be discussed. We will be using two machines with the following setup:

Machine Role OS IP Address Hostname User

Client

Debian GNU/Linux 10 (buster)

192.168.1.55

debian

carol

Server

openSUSE Leap 15.1

192.168.1.77

halof

ina

Basic OpenSSH Client Configuration and Usage

Although the OpenSSH server and client come in separate packages, you can normally install a metapackage that will provide both at once. To establish a remote session with the SSH server you use the ssh command, specifying the user you want to connect as on the remote machine and the remote machine’s IP address or hostname. The first time you connect to a remote host you will receive a message like this:

carol@debian:~$ ssh ina@192.168.1.77
The authenticity of host '192.168.1.77 (192.168.1.77)' can't be established.
ECDSA key fingerprint is SHA256:5JF7anupYipByCQm2BPvDHRVFJJixeslmppi2NwATYI.
Are you sure you want to continue connecting (yes/no)?

After typing yes and hitting Enter, you will be asked for the remote user’s password. If successfully entered, you will be shown a warning message and then logged in to the remote host:

Warning: Permanently added '192.168.1.77' (ECDSA) to the list of known hosts.
Password:
Last login: Sat Jun 20 10:52:45 2020 from 192.168.1.4
Have a lot of fun...
ina@halof:~>

The messages are quite self-explanatory: as it was the first time that you established a connection to the 192.168.1.77 remote server, its authenticity could not be checked against any database. Thus, the remote server provided an ECDSA key fingerprint of its public key (using the SHA256 hash function). Once you accepted the connection, the public key of the remote server was added to the known hosts database, thus enabling the authentication of the server for future connections. This list of known hosts' public keys is kept in the file known_hosts which lives in ~/.ssh:

ina@halof:~> exit
logout
Connection to 192.168.1.77 closed.
carol@debian:~$ ls .ssh/
known_hosts

Both .ssh and known_hosts were created after the first remote connection was established. ~/.ssh is the default directory for user-specific configuration and authentication information.

Note	You can also use `ssh` to just execute a single command on the remote host and then come back to your local terminal (e.g.: running `ssh ina@halof ls`).

If you are using the same user on both the local and remote hosts, there is no need to specify the username when establishing the SSH connection. For instance, if you are logged in as user carol on debian and wanted to connect to halof also as user carol, you would simply type ssh 192.168.1.77 or ssh halof (if the name can be resolved):

carol@debian:~$ ssh halof
Password:
Last login: Wed Jul  1 23:45:02 2020 from 192.168.1.55
Have a lot of fun...
carol@halof:~>

Now suppose you establish a new remote connection with a host that happens to have the same IP address as halof (a common thing if you use DHCP in your LAN). You will be warned about the possibility of a man-in-the-middle attack:

carol@debian:~$ ssh john@192.168.1.77
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:KH4q3vP6C7e0SEjyG8Wlz9fVlf+jmWJ5139RBxBh3TY.
Please contact your system administrator.
Add correct host key in /home/carol/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/carol/.ssh/known_hosts:1
  remove with:
  ssh-keygen -f "/home/carol/.ssh/known_hosts" -R "192.168.1.77"
ECDSA host key for 192.168.1.77 has changed and you have requested strict checking.
Host key verification failed.

Since you are not dealing with a man-in-the-middle attack, you can safely add the public key fingerprint of the new host to .ssh/known_hosts. As the message indicates, you can first use the command ssh-keygen -f "/home/carol/.ssh/known_hosts" -R "192.168.1.77" to remove the offending key (alternatively, you can go for ssh-keygen -R 192.168.1.77 to delete all keys belonging to 192.168.1.77 from ~/.ssh/known_hosts). Then, you will be able to establish a connection to the new host.

Key-Based Logins

You can set up your SSH client to not provide any passwords at login but use public keys instead. This is the preferred method of connecting to a remote server via SSH, as it is far more secure. The first thing you have to do is create a key pair on the client machine. To do this, you will use ssh-keygen with the -t option specifying the type of encryption you want (Elliptic Curve Digital Signature Algorithm in our case). Then, you will be asked for the path to save the key pair (~/.ssh/ is convenient as well as the default location) and a passphrase. While a passphrase is optional, it is highly recommended to always use one.

carol@debian:~/.ssh$ ssh-keygen -t ecdsa
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/carol/.ssh/id_ecdsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/carol/.ssh/id_ecdsa.
Your public key has been saved in /home/carol/.ssh/id_ecdsa.pub.
The key fingerprint is:
SHA256:tlamD0SaTquPZYdNepwj8XN4xvqmHCbe8g5FKKUfMo8 carol@debian
The key's randomart image is:
+---[ECDSA 256]---+
|      .          |
|     o .         |
|    = o o        |
|     B *         |
|    E B S o      |
|     o & O       |
|      @ ^ =      |
|     *.@ @.      |
|    o.o+B+o      |
+----[SHA256]-----+

Note	When creating the key pair, you can pass `ssh-keygen` the `-b` option to specify the key size in bits (e.g.: `ssh-keygen -t ecdsa -b 521`).

The previous command produced two more files in your ~/.ssh directory:

carol@debian:~/.ssh$ ls
id_ecdsa  id_ecdsa.pub  known_hosts

id_ecdsa: This is your private key.
id_ecdsa.pub: This is your public key.

Note	In asymmetric cryptography (aka public-key cryptography), the public and private keys are mathematically related to one another in such a way that whatever is encrypted by one can only be decrypted by the other.

The next thing you need to do is add your public key to the ~/.ssh/authorized_keys file of the user you want to log in as on the remote host (if the ~/.ssh directory does not already exist, you will have to create it first). You can copy your public key into the remote server in a number of ways: using a USB flash drive, through the scp command — which will transfer the file over using SSH — or by catting out the content of your public key and piping it into ssh like so:

carol@debian:~/.ssh$ cat id_ecdsa.pub |ssh ina@192.168.1.77 'cat >> .ssh/authorized_keys'
Password:

Once your public key has been added to the authorized_keys file on the remote host, you can face two scenarios when trying to establish a new connection:

If you did not provide a passphrase when creating the key pair, you will be logged in automatically. Although convenient, this method can be insecure depending on the situation:
```
carol@debian:~$ ssh ina@192.168.1.77
Last login: Thu Jun 25 20:31:03 2020 from 192.168.1.55
Have a lot of fun...
ina@halof:~>
```
If you provided a passphrase when creating the key pair, you will have to enter it on every connection much in the same way as if it was a password. Apart from the public key, this method adds an extra layer of security in the form of a passphrase and can — therefore — be considered more secure. As far as convenience goes — however — it is is exactly the same as having to enter a password every time you establish a connection. If you don’t use a passphrase and someone manages to obtain your private SSH key file, they would have access to every server on which your public key is installed.
```
carol@debian:~/.ssh$ ssh ina@192.168.1.77
Enter passphrase for key '/home/carol/.ssh/id_ecdsa':
Last login: Thu Jun 25 20:39:30 2020 from 192.168.1.55
Have a lot of fun...
ina@halof:~>
```

There is a way which combines security and convenience, though: using the SSH authentication agent (ssh-agent). The authentication agent needs to spawn its own shell and will hold your private keys — for public key authentication — in memory for the remainder of the session. Let us see how it works in a little bit more detail:

Use ssh-agent to start a new Bash shell:

carol@debian:~/.ssh$ ssh-agent /bin/bash
carol@debian:~/.ssh$

Use the ssh-add command to add your private key to a secure area of memory. If you supplied a passphrase when generating the key pair — which is recommended for extra security — you will be asked for it:
```
carol@debian:~/.ssh$ ssh-add
Enter passphrase for /home/carol/.ssh/id_ecdsa:
Identity added: /home/carol/.ssh/id_ecdsa (carol@debian)
```
Once your identity has been added, you can login to any remote server on which your public key is present without having to type your passphrase again. It is common practice on modern desktops to perform this command upon booting your computer, as it will remain in memory until the computer is shutdown (or the key is unloaded manually).

Let us round this section off by listing the four types of public-key algorithms that can be specified with ssh-keygen:

RSA: Named after its creators Ron Rivest, Adi Shamir and Leonard Adleman, it was published in 1977. It is considered secure and still widely used today. Its minimum key size is 1024 bits (default is 2048).
DSA: The Digital Signature Algorithm has proven to be insecure and it was deprecated as of OpenSSH 7.0. DSA keys must be exaclty 1024 bits in length.
ecdsa: The Elliptic Curve Digital Signature Algorithm is an improvement on DSA and — therefore — considered more secure. It uses elliptic curve cryptography. ECDSA key length is determined by one of the three possible elliptic curve sizes in bits: 256, 384 or 521.
ed25519: It is an implementation of EdDSA — Edwards-curve Digital Signature Algorithm — that uses the stronger 25519 curve. It is considered the most secure of all. All Ed25519 keys have a fixed length of 256 bits.

Note	If invoked with no `-t` specification, `ssh-keygen` will generate an `RSA` key pair by default.

The Role of OpenSSH Server Host Keys

The global configuration directory for OpenSSH lives in the /etc directory:

halof:~ # tree /etc/ssh
/etc/ssh
├── moduli
├── ssh_config
├── ssh_host_dsa_key
├── ssh_host_dsa_key.pub
├── ssh_host_ecdsa_key
├── ssh_host_ecdsa_key.pub
├── ssh_host_ed25519_key
├── ssh_host_ed25519_key.pub
├── ssh_host_rsa_key
├── ssh_host_rsa_key.pub
└── sshd_config

0 directories, 11 files

Apart from moduli and the configuration files for the client (ssh_config) and the server (sshd_config), you will find four key pairs — a key pair for each supported algorithm — that are created when the OpenSSH server is installed. As already noted, the server uses these host keys to identify itself to clients as required. Their name pattern is as follows:

Private keys: ssh_host_ prefix + algorithm + key suffix (e.g.: ssh_host_rsa_key)
Public keys (or public key fingerprints): ssh_host_ prefix + algorithm + key.pub suffix (e.g.: ssh_host_rsa_key.pub)

Note	A fingerprint is created by applying a cryptographic hash function to a public key. As fingerprints are shorter than the keys they refer to, they come in handy to simplify certain key management tasks.

The permissions on the files containing the private keys are 0600 or -rw-------: only readable and writable by the owner (root). On the other hand, all public key files are also readable by members in the owner group and everybody else (0644 or -rw-r—r--):

halof:~ # ls -l /etc/ssh/ssh_host_*
-rw------- 1 root root 1381 Dec 21 20:35 /etc/ssh/ssh_host_dsa_key
-rw-r--r-- 1 root root  605 Dec 21 20:35 /etc/ssh/ssh_host_dsa_key.pub
-rw------- 1 root root  505 Dec 21 20:35 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r-- 1 root root  177 Dec 21 20:35 /etc/ssh/ssh_host_ecdsa_key.pub
-rw------- 1 root root  411 Dec 21 20:35 /etc/ssh/ssh_host_ed25519_key
-rw-r--r-- 1 root root   97 Dec 21 20:35 /etc/ssh/ssh_host_ed25519_key.pub
-rw------- 1 root root 1823 Dec 21 20:35 /etc/ssh/ssh_host_rsa_key
-rw-r--r-- 1 root root  397 Dec 21 20:35 /etc/ssh/ssh_host_rsa_key.pub

You can view the fingerprints of the keys by passing ssh-keygen the -l switch. You must also provide the -f to specify the key file path:

halof:~ # ssh-keygen -l -f /etc/ssh/ssh_host_ed25519_key
256 SHA256:8cnPrinC49ZHc+/9Ai5pV+1JfZ4WBRZhd3rDOsc2zlA root@halof (ED25519)
halof:~ # ssh-keygen -l -f /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:8cnPrinC49ZHc+/9Ai5pV+1JfZ4WBRZhd3rDOsc2zlA root@halof (ED25519)

To view the key fingerprint as well as its random art, just add the -v switch like so:

halof:~ # ssh-keygen -lv -f /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:8cnPrinC49ZHc+/9Ai5pV+1JfZ4WBRZhd3rDOsc2zlA root@halof (ED25519)
+--[ED25519 256]--+
|              +oo|
|             .+o.|
|        .    ..E.|
|         + .  +.o|
|        S +  + *o|
|          ooo Oo=|
|     . . . =o+.==|
|      = o =oo o=o|
|     o.o +o+..o.+|
+----[SHA256]-----+

SSH Port Tunnels

OpenSSH features a very powerful forwarding facility whereby traffic on a source port is tunnelled — and encrypted — through an SSH process which then redirects it to a port on a destination host. This mechanism is known as port tunnelling or port forwarding and has important advantages like the following:

It allows you to bypass firewalls to access ports on remote hosts.
It allows access from the outside to a host on your private network.
It provides encryption for all data exchange.

Roughly speaking, we can differentiate between local and remote port tunnelling.

Local Port Tunnel

You define a port locally to forward traffic to the destination host through the SSH process which sits in between. The SSH process can run on the local host or on a remote server. For instance, if for some reason you wanted to tunnel a connection to www.gnu.org through SSH using port 8585 on your local machine, you would do something like this:

carol@debian:~$ ssh -L 8585:www.gnu.org:80 debian
carol@debian's password:
Linux debian 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
(...)
Last login: Sun Jun 28 13:47:27 2020 from 127.0.0.1

The explanation is as follows: with the -L switch, we specify the local port 8585 to connect to http port 80 on www.gnu.org using the SSH process running on debian — our localhost. We could have written ssh -L 8585:www.gnu.org:80 localhost with the same effect. If you now use a web browser to go to http://localhost:8585, you will be forwarded to www.gnu.org. For demonstration purposes, we will use lynx (the classic, text-mode web browser):

carol@debian:~$ lynx http://localhost:8585
(...)
  * Back to Savannah Homepage
     * Not Logged in
     * Login
     * New User
     * This Page
     * Language
     * Clean Reload
     * Printer Version
     * Search
     * _
(...)

If you wanted to do the exact same thing but connecting through an SSH process running on halof, you would have proceeded like so:

carol@debian:~$ ssh -L 8585:www.gnu.org:80 -Nf ina@192.168.1.77
Enter passphrase for key '/home/carol/.ssh/id_ecdsa':
carol@debian:~$
carol@debian:~$ lynx http://localhost:8585
(...)
  * Back to Savannah Homepage
     * Not Logged in
     * Login
     * New User
     * This Page
     * Language
     * Clean Reload
     * Printer Version
     * Search
     * _
(...)

It is important that you note three details in the command:

Thanks to the -N option we did not login to halof but did the port forwarding instead.
The -f option told SSH to run in the background.
We specified user ina to do the forwarding: ina@192.168.1.77

Remote Port Tunnel

In remote port tunnelling (or reverse port forwarding) the traffic coming on a port on the remote server is forwarded to the SSH process running on your local host, and from there to the specified port on the destination server (which may also be your local machine). For example, say you wanted to let someone from outside your network access the Apache web server running on your local host through port 8585 of the SSH server running on halof (192.168.1.77). You would proceed with the following command:

carol@debian:~$ ssh -R 8585:localhost:80 -Nf ina@192.168.1.77
Enter passphrase for key '/home/carol/.ssh/id_ecdsa':
carol@debian:~$

Now anyone who establishes a connection to halof on port 8585 will see Debian's Apache2 default homepage:

carol@debian:~$ lynx 192.168.1.77:8585
(...)
                                                                 Apache2 Debian Default Page: It works (p1 of 3)
   Debian Logo Apache2 Debian Default Page
   It works!

   This is the default welcome page used to test the correct operation of the Apache2 server after
   installation on Debian systems. If you can read this page, it means that the Apache HTTP server
   installed at this site is working properly. You should replace this file (located at
   /var/www/html/index.html) before continuing to operate your HTTP server.
(...)

Note	There is a third, more complex type of port forwarding which is outside the scope of this lesson: dynamic port forwarding. Instead of interacting with a single port, this type of forwarding uses various TCP communications across a range of ports.

X11 Tunnels

Now that you understand port tunnels, let us round this lesson off by discussing X11 tunnelling (also known as X11forwarding). Through an X11 tunnel, the X Window System on the remote host is forwarded to your local machine. For that, you just need to pass ssh the -X option:

carol@debian:~$ ssh -X ina@halof
...

You can now launch a graphical application such as the firefox web browser with the following result: the app will be run on the remote server, but its display will be forwarded to your local host.

If you start a new SSH session with the -x option instead, X11forwarding will be disabled. Try to start firefox now and you will get an error such as the following:

carol@debian:~$ ssh -x ina@halof
carol@192.168.0.106's password:
(...)
ina@halof:~$ firefox

(firefox-esr:1779): Gtk-WARNING **: 18:45:45.603: Locale not supported by C library.
	Using the fallback 'C' locale.
Error: no DISPLAY environment variable specified

Note	The three configuration directives related to local port forwarding, remote port forwarding and X11 forwarding are `AllowTcpForwarding`,`GatewayPorts` and `X11Forwarding`, respectively. For more information, type `man ssh_config` and/or `man sshd_config`.

Guided Exercises

Logged in as user sonya on your client machine, carry out the following SSH tasks on the remote server halof:
- Execute the command to list the contents of ~/.ssh as user serena on the remote host; then return to your local terminal.
- Login as user serena on the remote host.
- Login as user sonya on the remote host.
- Delete all keys belonging to halof from your local ~/.ssh/known_hosts file.
- On your client machine, create an ecdsa key pair of 256 bits.
- On your client machine, create an ed25519 key pair of 256 bits.
Put the following steps in the right order to establish an SSH connection using the SSH authentication agent:
- On the client, start a new Bash shell for the authentication agent with ssh-agent /bin/bash.
- On the client, create a key pair using ssh-keygen.
- On the client, add your private key to a secure area of memory with ssh-add.
- Add your client’s public key to the ~/.ssh/authorized_keys file of the user you want to login as on the remote host.
- If it does not already exist, create ~/.ssh for the user you want to login as on the server.
- Connect to the remote server.
  
  The correct order is:
  
  Step 1:
  
  Step 2:
  
  Step 3:
  
  Step 4:
  
  Step 5:
  
  Step 6:
Regarding port forwarding, what option and directive is used for the following tunnel types:

Tunnel Type Option Directive

Local

Remote or Reverse

X
Suppose you type the command ssh -L 8888:localhost:80 -Nf ina@halof into the terminal of your client machine. Still on the client machine, you point a browser to http://localhost:8888. What will you get?

Explorational Exercises

Concerning SSH security directives:
- What directive is used in /etc/ssh/sshd_config to enable root logins:
- What directive would you use in /etc/ssh/sshd_config to specify only a local account to accept SSH connections:
When using the same user on both the client and the server, what ssh command can you use to transfer the client’s public key over to the server so that you can login through public key authentication?
Create two local port tunnels in a single command forwarding local unprivileged ports 8080 and 8585 through remote server halof to the websites www.gnu.org and www.melpa.org, respectively. Use user ina on the remote server and do not forget to use the -Nf switches:

Summary

In this lesson we have discussed OpenSSH 2, which uses the Secure Shell protocol to encrypt communications between server and client. You learned:

how to login to a remote server.
how to execute commands remotely.
how to create key pairs.
how to establish key-based logins.
how to use the authentication agent for both extra security and convenience.
the public-key algorithms supported by OpenSSH: RSA, DSA, ecdsa, ed25519.
the role of OpenSSH host keys.
how to create port tunnels: local, remote and X.

The following commands were discussed in this lesson:

ssh: Log into or excute commands on a remote machine.
ssh-keygen: Generate, manage and convert authentication keys.
ssh-agent: OpenSSH authentication agent.
ssh-add: Adds private key identities to the authentication agent.

Answers to Guided Exercises

Logged in as user sonya on your client machine, carry out the following SSH tasks on the remote server halof:
- Execute the command to list the contents of ~/.ssh as user serena on the remote host; then return to your local terminal.
  ssh serena@halof ls .ssh
- Login as user serena on the remote host.
  ssh serena@halof
- Login as user sonya on the remote host.
  ssh halof
- Delete all keys belonging to halof from your local ~/.ssh/known_hosts file.
  ssh-keygen -R halof
- On your client machine, create an ecdsa key pair of 256 bits.
  ssh-keygen -t ecdsa -b 256
- On your client machine, create an ed25519 key pair of 256 bits.
  ssh-keygen -t ed25519

Put the following steps in the right order to establish an SSH connection using the SSH authentication agent:

On the client, start a new Bash shell for the authentication agent with ssh-agent /bin/bash.
On the client, create a key pair using ssh-keygen.
On the client, add your private key to a secure area of memory with ssh-add.
Add your client’s public key to the ~/.ssh/authorized_keys file of the user you want to login as on the remote host.
If it does not already exist, create ~/.ssh for the user you want to login as on the server.

Connect to the remote server.

The correct order is:

Step 1:

On the client, create a key pair using ssh-keygen.

Step 2:

If it does not already exist, create ~/.ssh for the user you want to login as on the server.

Step 3:

Add your client’s public key to the ~/.ssh/authorized_keys file of the user you want to login as on the remote host.

Step 4:

On the client, start a new Bash shell for the authentication agent with ssh-agent /bin/bash.

Step 5:

On the client, add your private key to a secure area of memory with ssh-add.

Step 6:

Connect to the remote server.

Regarding port forwarding, what option and directive is used for the following tunnel types:

Tunnel Type Option Directive

Local

-L

AllowTcpForwarding

Remote or Reverse

-R

GatewayPorts

X

-X

X11Forwarding
Suppose you type the command ssh -L 8888:localhost:80 -Nf ina@halof into the terminal of your client machine. Still on the client machine, you point a browser to http://localhost:8888. What will you get?

The webserver’s homepage of halof, as localhost is understood from the server’s perspective.

Answers to Explorational Exercises

Concerning SSH security directives:
- What directive is used in /etc/ssh/sshd_config to enable root logins:
  
  PermitRootLogin
- What directive would you use in /etc/ssh/sshd_config to specify only a local account to accept SSH connections:
  
  AllowUsers
When using the same user on both the client and the server, what ssh command can you use to transfer the client’s public key over to the server so that you can login through public key authentication?

ssh-copy-id
Create two local port tunnels in a single command forwarding local unprivileged ports 8080 and 8585 through remote server halof to the websites www.gnu.org and www.melpa.org, respectively. Use user ina on the remote server and do not forget to use the -Nf switches:

ssh -L 8080:www.gnu.org:80 -L 8585:www.melpa.org:80 -Nf ina@halof