109.3 Lesson 2

Certificate:

LPIC-1

Version:

5.0

Topic:

109 Networking Fundamentals

Objective:

109.3 Basic network troubleshooting

Lesson:

2 of 2

Introduction

Linux based operating systems have a variety of tools to troubleshoot network problems with. This lesson is going to cover some of the more common ones. At this point you should have a grasp of the OSI or other layered models of networking, IPv4 or IPv6 addressing, and the basics of routing and switching.

The best way to test a network connection is to try to use your application. When that doesn’t work, there are plenty of tools available to help diagnose the problem.

Testing Connections With `ping`

The ping and ping6 commands can be used to send an ICMP echo request to an IPv4 or IPv6 address, respectively. An ICMP echo request sends a small amount of data to the destination address. If the destination address is reachable, it will send an ICMP echo reply message back to the sender with the same data that was sent to it:

$ ping -c 3 192.168.50.2
PING 192.168.50.2 (192.168.50.2) 56(84) bytes of data.
64 bytes from 192.168.50.2: icmp_seq=1 ttl=64 time=0.525 ms
64 bytes from 192.168.50.2: icmp_seq=2 ttl=64 time=0.419 ms
64 bytes from 192.168.50.2: icmp_seq=3 ttl=64 time=0.449 ms

--- 192.168.50.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2006ms
rtt min/avg/max/mdev = 0.419/0.464/0.525/0.047 ms

$ ping6 -c 3 2001:db8::10
PING 2001:db8::10(2001:db8::10) 56 data bytes
64 bytes from 2001:db8::10: icmp_seq=1 ttl=64 time=0.425 ms
64 bytes from 2001:db8::10: icmp_seq=2 ttl=64 time=0.480 ms
64 bytes from 2001:db8::10: icmp_seq=3 ttl=64 time=0.725 ms

--- 2001:db8::10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.425/0.543/0.725/0.131 ms

The -c option is used to specify the number of packets to send. If you omit this option, ping and ping6 will continue to send packets until you stop it, typically with the Ctrl+C keyboard combination.

Just because you can’t ping a host, doesn’t mean you can’t connect to it. Many organizations have firewalls or router access control lists that block everything but the bare minimum needed for their systems to function. This includes ICMP echo request and replies. Since these packets can include arbitrary data, a clever attacker could use them to exfiltrate data.

Tracing Routes

The traceroute and traceroute6 programs can be used to show you the route a packet takes to get to its destination. They do this by sending multiple packets to the destination, incrementing the Time-To-Live (TTL) field of the IP header with each subsequent packet. Each router along the way will respond with a TTL exceeded ICMP message:

$ traceroute 192.168.1.20
traceroute to 192.168.1.20 (192.168.1.20), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  0.396 ms  0.171 ms  0.132 ms
 2  192.168.1.20 (192.168.1.20)  2.665 ms  2.573 ms  2.573 ms
$ traceroute 192.168.50.2
traceroute to 192.168.50.2 (192.168.50.2), 30 hops max, 60 byte packets
 1  192.168.50.2 (192.168.50.2)  0.433 ms  0.273 ms  0.171 ms
$ traceroute6 2001:db8::11
traceroute to 2001:db8::11 (2001:db8::11), 30 hops max, 80 byte packets
 1  2001:db8::11 (2001:db8::11)  0.716 ms  0.550 ms  0.641 ms
$ traceroute 2001:db8::11
traceroute to 2001:db8::11 (2001:db8::11), 30 hops max, 80 byte packets
 1  2001:db8::10 (2001:db8::11)  0.617 ms  0.461 ms  0.387 ms
$ traceroute net2.example.net
traceroute to net2.example.net (192.168.50.2), 30 hops max, 60 byte packets
 1  net2.example.net (192.168.50.2)  0.533 ms  0.529 ms  0.504 ms
$ traceroute6 net2.example.net
traceroute to net2.example.net (2001:db8::11), 30 hops max, 80 byte packets
 1  net2.example.net (2001:db8::11)  0.738 ms  0.607 ms  0.304 ms

By default, traceroute sends 3 UDP packets with junk data to port 33434, incrementing it each time it sends a packet. Each line in the command’s output is a router interface the packet traverses through. The times shown in each line of the output is the round trip time for each packet. The IP address is the address of the router interface in question. If traceroute is able to, it uses the DNS name of the router interface. Sometimes you will see * in place of a time. When this happens, it means that traceroute never received the TTL exceeded message for this packet. When you start seeing this, this often indicates that the last response is the last hop on the route.

If you have access to root, the -I option will set traceroute to use ICMP echo requests instead of UDP packets. This is often more effective than UDP because the destination host is more likely to respond to an ICMP echo request than the UDP packet:

# traceroute -I learning.lpi.org
traceroute to learning.lpi.org (208.94.166.201), 30 hops max, 60 byte packets
 1  047-132-144-001.res.spectrum.com (47.132.144.1)  9.764 ms  9.702 ms  9.693 ms
 2  096-034-094-106.biz.spectrum.com (96.34.94.106)  8.389 ms  8.481 ms  8.480 ms
 3  dtr01hlrgnc-gbe-4-15.hlrg.nc.charter.com (96.34.64.172)  8.763 ms  8.775 ms  8.770 ms
 4  acr01mgtnnc-vln-492.mgtn.nc.charter.com (96.34.67.202)  27.080 ms  27.154 ms  27.151 ms
 5  bbr01gnvlsc-bue-3.gnvl.sc.charter.com (96.34.2.112)  31.339 ms  31.398 ms  31.395 ms
 6  bbr01aldlmi-tge-0-0-0-13.aldl.mi.charter.com (96.34.0.161)  39.092 ms  38.794 ms  38.821 ms
 7  prr01ashbva-bue-3.ashb.va.charter.com (96.34.3.51)  34.208 ms  36.474 ms  36.544 ms
 8  bx2-ashburn.bell.ca (206.126.236.203)  53.973 ms  35.975 ms  38.250 ms
 9  tcore4-ashburnbk_0-12-0-0.net.bell.ca (64.230.125.190)  66.315 ms  65.319 ms  65.345 ms
10  tcore4-toronto47_2-8-0-3.net.bell.ca (64.230.51.22)  67.427 ms  67.502 ms  67.498 ms
11  agg1-toronto47_xe-7-0-0_core.net.bell.ca (64.230.161.114)  61.270 ms  61.299 ms  61.291 ms
12  dis4-clarkson16_5-0.net.bell.ca (64.230.131.98)  61.101 ms  61.177 ms  61.168 ms
13  207.35.12.142 (207.35.12.142)  70.009 ms  70.069 ms  59.893 ms
14  unassigned-117.001.centrilogic.com (66.135.117.1)  61.778 ms  61.950 ms  63.041 ms
15  unassigned-116.122.akn.ca (66.135.116.122)  62.702 ms  62.759 ms  62.755 ms
16  208.94.166.201 (208.94.166.201)  62.936 ms  62.932 ms  62.921 ms

Some organizations block ICMP echo requests and replies. To get around this, you can use TCP. By using a known open TCP port, you can guarantee the destination host will respond. To use TCP, use the -T option along with -p to specify the port. As with ICMP echo requests, you must have access to root to do this:

# traceroute -m 60 -T -p 80  learning.lpi.org
traceroute to learning.lpi.org (208.94.166.201), 60 hops max, 60 byte packets
 1  * * *
 2  096-034-094-106.biz.spectrum.com (96.34.94.106)  12.178 ms  12.229 ms  12.175 ms
 3  dtr01hlrgnc-gbe-4-15.hlrg.nc.charter.com (96.34.64.172)  12.134 ms  12.093 ms  12.062 ms
 4  acr01mgtnnc-vln-492.mgtn.nc.charter.com (96.34.67.202)  31.146 ms  31.192 ms  31.828 ms
 5  bbr01gnvlsc-bue-3.gnvl.sc.charter.com (96.34.2.112)  39.057 ms  46.706 ms  39.745 ms
 6  bbr01aldlmi-tge-0-0-0-13.aldl.mi.charter.com (96.34.0.161)  50.590 ms  58.852 ms  58.841 ms
 7  prr01ashbva-bue-3.ashb.va.charter.com (96.34.3.51)  34.556 ms  37.892 ms  38.274 ms
 8  bx2-ashburn.bell.ca (206.126.236.203)  38.249 ms  36.991 ms  36.270 ms
 9  tcore4-ashburnbk_0-12-0-0.net.bell.ca (64.230.125.190)  66.779 ms  63.218 ms tcore3-ashburnbk_100ge0-12-0-0.net.bell.ca (64.230.125.188)  60.441 ms
10  tcore4-toronto47_2-8-0-3.net.bell.ca (64.230.51.22)  63.932 ms  63.733 ms  68.847 ms
11  agg2-toronto47_xe-7-0-0_core.net.bell.ca (64.230.161.118)  60.144 ms  60.443 ms agg1-toronto47_xe-7-0-0_core.net.bell.ca (64.230.161.114)  60.851 ms
12  dis4-clarkson16_5-0.net.bell.ca (64.230.131.98)  67.246 ms dis4-clarkson16_7-0.net.bell.ca (64.230.131.102)  68.404 ms dis4-clarkson16_5-0.net.bell.ca (64.230.131.98)  67.403 ms
13  207.35.12.142 (207.35.12.142)  66.138 ms  60.608 ms  64.656 ms
14  unassigned-117.001.centrilogic.com (66.135.117.1)  70.690 ms  62.190 ms  61.787 ms
15  unassigned-116.122.akn.ca (66.135.116.122)  62.692 ms  69.470 ms  68.815 ms
16  208.94.166.201 (208.94.166.201)  61.433 ms  65.421 ms  65.247 ms
17  208.94.166.201 (208.94.166.201)  64.023 ms  62.181 ms  61.899 ms

Like ping, traceroute has its limitations. It is possible for firewalls and routers to block the packets sent from or returned to traceroute. If you have root access, there are options that can help you get accurate results.

Finding MTUs With `tracepath`

The tracepath command is similar to traceroute. The difference is it tracks Maximum Transmission Unit (MTU) sizes along the path. The MTU is either a configured setting on a network interface or hardware limitation of the largest protocol data unit that it can transmit or receive. The tracepath program works the same way as traceroute in that it increments the TTL with each packet. It differs by sending a very large UDP datagram. It is almost inevitable for the datagram to be larger than the device with the smallest MTU along the route. When the packet reaches this device, the device will typically respond with a destination unreachable packet. The ICMP destination unreachable packet has a field for the MTU of the link it would send the packet on if it were able. tracepath then sends all subsequent packets with this size:

$ tracepath 192.168.1.20
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.0.2.2                                              0.321ms
 1:  10.0.2.2                                              0.110ms
 2:  192.168.1.20                                          2.714ms reached
     Resume: pmtu 1500 hops 2 back 64

Unlike traceroute, you must explicitly use tracepath6 for IPv6:

$ tracepath 2001:db8::11
tracepath: 2001:db8::11: Address family for hostname not supported
$ tracepath6 2001:db8::11
 1?: [LOCALHOST]                        0.027ms pmtu 1500
 1:  net2.example.net                                      0.917ms reached
 1:  net2.example.net                                      0.527ms reached
     Resume: pmtu 1500 hops 1 back 1

The output is similar to traceroute. The advantage of tracepath is on the last line it outputs the smallest MTU on the entire link. This can be useful for troubleshooting connections that can’t handle fragments.

As with the previous troubleshooting tools, there is the potential for equipment to block your packets.

Creating Arbitrary Connections

The nc program, known as netcat, can send or receive arbitrary data over a TCP or UDP network connection. The following examples should make its functionality clear.

Here is an example of setting up a listener on port 1234:

$ nc -l 1234
LPI Example

The output of LPI Example appears after the example below, which is setting up a netcat sender to send packets to net2.example.net on port 1234. The -l option is used to specify that you wish for nc to receive data instead of send it:

$ nc net2.example.net 1234
LPI Example

Press Ctrl+C on either system to stop the connection.

Netcat works with both IPv4 and IPv6 addresses. It works with both TCP and UDP. It can even be used to setup a crude remote shell.

Warning

Note that not every installation of nc supports the -e switch. Be sure to review the man pages for your installation for security information about this option as well as alternative methods to execute commands on a remote system.

$ hostname
net2
$ nc -u -e /bin/bash -l 1234

The -u option is for UDP. -e instructs netcat to send everything it receives to standard input of the executable following it. In this example, /bin/bash.

$ hostname
net1
$ nc -u net2.example.net 1234
hostname
net2
pwd
/home/emma

Notice how the hostname command output matched that of the listening host and the pwd command output a directory?

Viewing Current Connections and Listeners

The netstat and ss programs can be used to view the status of your current listeners and connections. As with ifconfig, netstat is a legacy tool. Both netstat and ss have similar output and options. Here some options available to both programs:

-a: Shows all sockets.
-l: Shows listening sockets.
-p: Shows the process associated with the connection.
-n: Prevents name lookups for both ports and addresses.
-t: Shows TCP connections.
-u: Shows UDP connections.

The examples below show the output of a commonly used set of options for both programs:

# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      892/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1141/master
tcp6       0      0 :::22                   :::*                    LISTEN      892/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1141/master
udp        0      0 0.0.0.0:68              0.0.0.0:*                           692/dhclient
# ss -tulnp
# ss -tulnp
Netid  State      Recv-Q Send-Q      Local Address:Port                     Peer Address:Port
udp    UNCONN     0      0                       :68                                  *:                   users:(("dhclient",pid=693,fd=6))
tcp    LISTEN     0      128                     :22                                  *:                   users:(("sshd",pid=892,fd=3))
tcp    LISTEN     0      100             127.0.0.1:25                                  :                   users:(("master",pid=1099,fd=13))
tcp    LISTEN     0      128                  [::]:22                               [::]:*                   users:(("sshd",pid=892,fd=4))
tcp    LISTEN     0      100                 [::1]:25                               [::]:*                   users:(("master",pid=1099,fd=14))

The Recv-Q column is the number of packets a socket has received but not passed off to its program. The Send-Q column is the number of packets a socket has sent that have not been acknowledged by the receiver. The rest of the columns are self explanatory.

Guided Exercises

What command(s) would you use to send an ICMP echo to learning.lpi.org?
How could you determine the route to 8.8.8.8?
What command would show you if any processes are listening on TCP port 80?
How could you find which process is listening on a port?
How could you determine the max MTU of a network path?

Explorational Exercises

How could you use netcat to send an HTTP request to a web server?
What are a few reasons pinging a host can fail?
Name a tool you could use to see network packets reaching or leaving a Linux host?
How can you force traceroute to use a different interface?
Is it possible for traceroute to report MTUs?

Summary

Networking is usually configured by a system’s startup scripts or a helper such as NetworkManager. Most distributions have tools that will edit the startup script configuration files for you. Consult your distribution’s documentation for details.

Being able to manually configure networking allows you to troubleshoot more effectively. It is useful in minimal environments used for things like restoring from backups or migrating to new hardware.

The utilities covered in this section have more functionality than covered in this lesson. It would be worthwhile to skim through the man page of each to familiarize yourself with the options available. The ss and ip commands are the modern way of doing things, while the rest that are covered, while still in common use, are considered legacy tools.

The best way to get familiar with the tools covered is practice. Using a computer with a modest amount of RAM, it is possible to setup a virtual network lab using virtual machines that you can practice with. Three virtual machines are enough to get comfortable with the tools listed.

The commands covered in this lesson include:

ping and ping6: Used to transmit ICMP packets to a remote host to test a network connection’s availability.
traceroute and traceroute6: Used to trace a path through a network to determine a network’s connectivity.
tracepath and tracepath6: Used to trace a path through a network as well as determine MTU sizes along a route.
nc: Used to set up arbitrary connections on a network for testing connectivity, as well as querying a network for available services and devices.
netstat: Legacy command used to determine a system’s open network connections and statistics.
ss: Modern command used to determine a system’s open network connections and statistics.

Answers to Guided Exercises

What command(s) would you use to send an ICMP echo to learning.lpi.org?

You would use ping or ping6:
```
$ ping learning.lpi.org
```
or
```
$ ping6 learning.lpi.org
```
How could you determine the route to 8.8.8.8?

By using the tracepath or traceroute commands.
```
$ tracepath 8.8.8.8
```
or
```
$ traceroute 8.8.8.8
```
What command would show you if any processes are listening on TCP port 80?

With ss:
```
$ ss -ln | grep ":80"
```
With netstat:
```
$ netstat -ln | grep ":80"
```
While not listed as a requirement for the exam, you can also use lsof:
```
# lsof -Pi:80
```
How could you find which process is listening on a port?

Again, there are multiple ways to do this. You could use lsof in the same manner as the previous answer, replacing the port number. You could also use netstat or ss with the -p option. Remember, netstat is considered a legacy tool.
```
# netstat -lnp | grep ":22"
```
The same options that work with netstat also work with ss:
```
# ss -lnp | grep ":22"
```
How could you determine the max MTU of a network path?

By using the tracepath command:
```
$ tracepath somehost.example.com
```

Answers to Explorational Exercises

How could you use netcat to send an HTTP request to a web server?

By entering the HTTP request line, any headers, and a blank line into the terminal:

$ nc learning.lpi.org 80
GET /index.html HTTP/1.1
HOST: learning.lpi.org

HTTP/1.1 302 Found
Location: https://learning.lpi.org:443/index.html
Date: Wed, 27 May 2020 22:54:46 GMT
Content-Length: 5
Content-Type: text/plain; charset=utf-8

Found

What are a few reasons pinging a host can fail?

There are a number of possible reasons. Here are some:
- The remote host is down.
- A router ACL is blocking your ping.
- The remote host’s firewall is blocking your ping.
- You may be using an incorrect host name or address.
- Your name resolution is returning an incorrect address.
- Your machine’s network configuration is incorrect.
- Your machine’s firewall is blocking it.
- The remote host’s network configuration is incorrect.
- Your machine’s interface(s) are disconnected.
- The remote machine’s interface(s) are disconnected.
- A network component such as a switch, cable, or router between your machine and the remote’s is no longer functioning.
Name a tool you could use to see network packets reaching or leaving a Linux host?

Both tcpdump and wireshark can be used.

How can you force traceroute to use a different interface?

By using the -i option:

$ traceroute -i eth2 learning.lpi.org
traceroute -i eth2 learning.lpi.org
traceroute to learning.lpi.org (208.94.166.201), 30 hops max, 60 byte packets
...

Is it possible for traceroute to report MTUs?

Yes, with the --mtu option:

# traceroute -I --mtu  learning.lpi.org
traceroute to learning.lpi.org (208.94.166.201), 30 hops max, 65000 byte packets
 1  047-132-144-001.res.spectrum.com (47.132.144.1)  9.974 ms F=1500  10.476 ms  4.743 ms
 2  096-034-094-106.biz.spectrum.com (96.34.94.106)  8.697 ms  9.963 ms  10.321 ms
...