109.3 Lesson 2
Certificate: |
LPIC-1 |
---|---|
Version: |
5.0 |
Topic: |
109 Networking Fundamentals |
Objective: |
109.3 Basic network troubleshooting |
Lesson: |
2 of 2 |
Introduction
Linux based operating systems have a variety of tools to troubleshoot network problems with. This lesson is going to cover some of the more common ones. At this point you should have a grasp of the OSI or other layered models of networking, IPv4 or IPv6 addressing, and the basics of routing and switching.
The best way to test a network connection is to try to use your application. When that doesn’t work, there are plenty of tools available to help diagnose the problem.
Testing Connections With ping
The ping
and ping6
commands can be used to send an ICMP echo request to an IPv4 or IPv6 address, respectively. An ICMP echo request sends a small amount of data to the destination address. If the destination address is reachable, it will send an ICMP echo reply message back to the sender with the same data that was sent to it:
$ ping -c 3 192.168.50.2 PING 192.168.50.2 (192.168.50.2) 56(84) bytes of data. 64 bytes from 192.168.50.2: icmp_seq=1 ttl=64 time=0.525 ms 64 bytes from 192.168.50.2: icmp_seq=2 ttl=64 time=0.419 ms 64 bytes from 192.168.50.2: icmp_seq=3 ttl=64 time=0.449 ms --- 192.168.50.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2006ms rtt min/avg/max/mdev = 0.419/0.464/0.525/0.047 ms
$ ping6 -c 3 2001:db8::10 PING 2001:db8::10(2001:db8::10) 56 data bytes 64 bytes from 2001:db8::10: icmp_seq=1 ttl=64 time=0.425 ms 64 bytes from 2001:db8::10: icmp_seq=2 ttl=64 time=0.480 ms 64 bytes from 2001:db8::10: icmp_seq=3 ttl=64 time=0.725 ms --- 2001:db8::10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.425/0.543/0.725/0.131 ms
The -c
option is used to specify the number of packets to send. If you omit this option, ping
and ping6
will continue to send packets until you stop it, typically with the Ctrl+C keyboard combination.
Just because you can’t ping a host, doesn’t mean you can’t connect to it. Many organizations have firewalls or router access control lists that block everything but the bare minimum needed for their systems to function. This includes ICMP echo request and replies. Since these packets can include arbitrary data, a clever attacker could use them to exfiltrate data.
Tracing Routes
The traceroute
and traceroute6
programs can be used to show you the route a packet takes to get to its destination. They do this by sending multiple packets to the destination, incrementing the Time-To-Live (TTL) field of the IP header with each subsequent packet. Each router along the way will respond with a TTL exceeded ICMP message:
$ traceroute 192.168.1.20 traceroute to 192.168.1.20 (192.168.1.20), 30 hops max, 60 byte packets 1 10.0.2.2 (10.0.2.2) 0.396 ms 0.171 ms 0.132 ms 2 192.168.1.20 (192.168.1.20) 2.665 ms 2.573 ms 2.573 ms $ traceroute 192.168.50.2 traceroute to 192.168.50.2 (192.168.50.2), 30 hops max, 60 byte packets 1 192.168.50.2 (192.168.50.2) 0.433 ms 0.273 ms 0.171 ms $ traceroute6 2001:db8::11 traceroute to 2001:db8::11 (2001:db8::11), 30 hops max, 80 byte packets 1 2001:db8::11 (2001:db8::11) 0.716 ms 0.550 ms 0.641 ms $ traceroute 2001:db8::11 traceroute to 2001:db8::11 (2001:db8::11), 30 hops max, 80 byte packets 1 2001:db8::10 (2001:db8::11) 0.617 ms 0.461 ms 0.387 ms $ traceroute net2.example.net traceroute to net2.example.net (192.168.50.2), 30 hops max, 60 byte packets 1 net2.example.net (192.168.50.2) 0.533 ms 0.529 ms 0.504 ms $ traceroute6 net2.example.net traceroute to net2.example.net (2001:db8::11), 30 hops max, 80 byte packets 1 net2.example.net (2001:db8::11) 0.738 ms 0.607 ms 0.304 ms
By default, traceroute
sends 3 UDP packets with junk data to port 33434, incrementing it each time it sends a packet. Each line in the command’s output is a router interface the packet traverses through. The times shown in each line of the output is the round trip time for each packet. The IP address is the address of the router interface in question. If traceroute
is able to, it uses the DNS name of the router interface. Sometimes you will see *
in place of a time. When this happens, it means that traceroute
never received the TTL exceeded message for this packet. When you start seeing this, this often indicates that the last response is the last hop on the route.
If you have access to root
, the -I
option will set traceroute
to use ICMP echo requests instead of UDP packets. This is often more effective than UDP because the destination host is more likely to respond to an ICMP echo request than the UDP packet:
# traceroute -I learning.lpi.org traceroute to learning.lpi.org (208.94.166.201), 30 hops max, 60 byte packets 1 047-132-144-001.res.spectrum.com (47.132.144.1) 9.764 ms 9.702 ms 9.693 ms 2 096-034-094-106.biz.spectrum.com (96.34.94.106) 8.389 ms 8.481 ms 8.480 ms 3 dtr01hlrgnc-gbe-4-15.hlrg.nc.charter.com (96.34.64.172) 8.763 ms 8.775 ms 8.770 ms 4 acr01mgtnnc-vln-492.mgtn.nc.charter.com (96.34.67.202) 27.080 ms 27.154 ms 27.151 ms 5 bbr01gnvlsc-bue-3.gnvl.sc.charter.com (96.34.2.112) 31.339 ms 31.398 ms 31.395 ms 6 bbr01aldlmi-tge-0-0-0-13.aldl.mi.charter.com (96.34.0.161) 39.092 ms 38.794 ms 38.821 ms 7 prr01ashbva-bue-3.ashb.va.charter.com (96.34.3.51) 34.208 ms 36.474 ms 36.544 ms 8 bx2-ashburn.bell.ca (206.126.236.203) 53.973 ms 35.975 ms 38.250 ms 9 tcore4-ashburnbk_0-12-0-0.net.bell.ca (64.230.125.190) 66.315 ms 65.319 ms 65.345 ms 10 tcore4-toronto47_2-8-0-3.net.bell.ca (64.230.51.22) 67.427 ms 67.502 ms 67.498 ms 11 agg1-toronto47_xe-7-0-0_core.net.bell.ca (64.230.161.114) 61.270 ms 61.299 ms 61.291 ms 12 dis4-clarkson16_5-0.net.bell.ca (64.230.131.98) 61.101 ms 61.177 ms 61.168 ms 13 207.35.12.142 (207.35.12.142) 70.009 ms 70.069 ms 59.893 ms 14 unassigned-117.001.centrilogic.com (66.135.117.1) 61.778 ms 61.950 ms 63.041 ms 15 unassigned-116.122.akn.ca (66.135.116.122) 62.702 ms 62.759 ms 62.755 ms 16 208.94.166.201 (208.94.166.201) 62.936 ms 62.932 ms 62.921 ms
Some organizations block ICMP echo requests and replies. To get around this, you can use TCP. By using a known open TCP port, you can guarantee the destination host will respond. To use TCP, use the -T
option along with -p
to specify the port. As with ICMP echo requests, you must have access to root
to do this:
# traceroute -m 60 -T -p 80 learning.lpi.org traceroute to learning.lpi.org (208.94.166.201), 60 hops max, 60 byte packets 1 * * * 2 096-034-094-106.biz.spectrum.com (96.34.94.106) 12.178 ms 12.229 ms 12.175 ms 3 dtr01hlrgnc-gbe-4-15.hlrg.nc.charter.com (96.34.64.172) 12.134 ms 12.093 ms 12.062 ms 4 acr01mgtnnc-vln-492.mgtn.nc.charter.com (96.34.67.202) 31.146 ms 31.192 ms 31.828 ms 5 bbr01gnvlsc-bue-3.gnvl.sc.charter.com (96.34.2.112) 39.057 ms 46.706 ms 39.745 ms 6 bbr01aldlmi-tge-0-0-0-13.aldl.mi.charter.com (96.34.0.161) 50.590 ms 58.852 ms 58.841 ms 7 prr01ashbva-bue-3.ashb.va.charter.com (96.34.3.51) 34.556 ms 37.892 ms 38.274 ms 8 bx2-ashburn.bell.ca (206.126.236.203) 38.249 ms 36.991 ms 36.270 ms 9 tcore4-ashburnbk_0-12-0-0.net.bell.ca (64.230.125.190) 66.779 ms 63.218 ms tcore3-ashburnbk_100ge0-12-0-0.net.bell.ca (64.230.125.188) 60.441 ms 10 tcore4-toronto47_2-8-0-3.net.bell.ca (64.230.51.22) 63.932 ms 63.733 ms 68.847 ms 11 agg2-toronto47_xe-7-0-0_core.net.bell.ca (64.230.161.118) 60.144 ms 60.443 ms agg1-toronto47_xe-7-0-0_core.net.bell.ca (64.230.161.114) 60.851 ms 12 dis4-clarkson16_5-0.net.bell.ca (64.230.131.98) 67.246 ms dis4-clarkson16_7-0.net.bell.ca (64.230.131.102) 68.404 ms dis4-clarkson16_5-0.net.bell.ca (64.230.131.98) 67.403 ms 13 207.35.12.142 (207.35.12.142) 66.138 ms 60.608 ms 64.656 ms 14 unassigned-117.001.centrilogic.com (66.135.117.1) 70.690 ms 62.190 ms 61.787 ms 15 unassigned-116.122.akn.ca (66.135.116.122) 62.692 ms 69.470 ms 68.815 ms 16 208.94.166.201 (208.94.166.201) 61.433 ms 65.421 ms 65.247 ms 17 208.94.166.201 (208.94.166.201) 64.023 ms 62.181 ms 61.899 ms
Like ping
, traceroute
has its limitations. It is possible for firewalls and routers to block the packets sent from or returned to traceroute
. If you have root
access, there are options that can help you get accurate results.
Finding MTUs With tracepath
The tracepath
command is similar to traceroute
. The difference is it tracks Maximum Transmission Unit (MTU) sizes along the path. The MTU is either a configured setting on a network interface or hardware limitation of the largest protocol data unit that it can transmit or receive. The tracepath
program works the same way as traceroute
in that it increments the TTL with each packet. It differs by sending a very large UDP datagram. It is almost inevitable for the datagram to be larger than the device with the smallest MTU along the route. When the packet reaches this device, the device will typically respond with a destination unreachable packet. The ICMP destination unreachable packet has a field for the MTU of the link it would send the packet on if it were able. tracepath
then sends all subsequent packets with this size:
$ tracepath 192.168.1.20 1?: [LOCALHOST] pmtu 1500 1: 10.0.2.2 0.321ms 1: 10.0.2.2 0.110ms 2: 192.168.1.20 2.714ms reached Resume: pmtu 1500 hops 2 back 64
Unlike traceroute
, you must explicitly use tracepath6
for IPv6:
$ tracepath 2001:db8::11 tracepath: 2001:db8::11: Address family for hostname not supported $ tracepath6 2001:db8::11 1?: [LOCALHOST] 0.027ms pmtu 1500 1: net2.example.net 0.917ms reached 1: net2.example.net 0.527ms reached Resume: pmtu 1500 hops 1 back 1
The output is similar to traceroute
. The advantage of tracepath
is on the last line it outputs the smallest MTU on the entire link. This can be useful for troubleshooting connections that can’t handle fragments.
As with the previous troubleshooting tools, there is the potential for equipment to block your packets.
Creating Arbitrary Connections
The nc
program, known as netcat, can send or receive arbitrary data over a TCP or UDP network connection. The following examples should make its functionality clear.
Here is an example of setting up a listener on port 1234
:
$ nc -l 1234 LPI Example
The output of LPI Example
appears after the example below, which is setting up a netcat sender to send packets to net2.example.net
on port 1234
. The -l
option is used to specify that you wish for nc
to receive data instead of send it:
$ nc net2.example.net 1234 LPI Example
Press Ctrl+C on either system to stop the connection.
Netcat works with both IPv4 and IPv6 addresses. It works with both TCP and UDP. It can even be used to setup a crude remote shell.
Warning
|
Note that not every installation of |
$ hostname net2 $ nc -u -e /bin/bash -l 1234
The -u
option is for UDP. -e
instructs netcat to send everything it receives to standard input of the executable following it. In this example, /bin/bash
.
$ hostname net1 $ nc -u net2.example.net 1234 hostname net2 pwd /home/emma
Notice how the hostname
command output matched that of the listening host and the pwd
command output a directory?
Viewing Current Connections and Listeners
The netstat
and ss
programs can be used to view the status of your current listeners and connections. As with ifconfig
, netstat
is a legacy tool. Both netstat
and ss
have similar output and options. Here some options available to both programs:
-a
-
Shows all sockets.
-l
-
Shows listening sockets.
-p
-
Shows the process associated with the connection.
-n
-
Prevents name lookups for both ports and addresses.
-t
-
Shows TCP connections.
-u
-
Shows UDP connections.
The examples below show the output of a commonly used set of options for both programs:
# netstat -tulnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 892/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1141/master tcp6 0 0 :::22 :::* LISTEN 892/sshd tcp6 0 0 ::1:25 :::* LISTEN 1141/master udp 0 0 0.0.0.0:68 0.0.0.0:* 692/dhclient # ss -tulnp # ss -tulnp Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port udp UNCONN 0 0 :68 *: users:(("dhclient",pid=693,fd=6)) tcp LISTEN 0 128 :22 *: users:(("sshd",pid=892,fd=3)) tcp LISTEN 0 100 127.0.0.1:25 : users:(("master",pid=1099,fd=13)) tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=892,fd=4)) tcp LISTEN 0 100 [::1]:25 [::]:* users:(("master",pid=1099,fd=14))
The Recv-Q
column is the number of packets a socket has received but not passed off to its program. The Send-Q
column is the number of packets a socket has sent that have not been acknowledged by the receiver. The rest of the columns are self explanatory.
Guided Exercises
-
What command(s) would you use to send an ICMP echo to
learning.lpi.org
? -
How could you determine the route to
8.8.8.8
? -
What command would show you if any processes are listening on TCP port 80?
-
How could you find which process is listening on a port?
-
How could you determine the max MTU of a network path?
Explorational Exercises
-
How could you use netcat to send an HTTP request to a web server?
-
What are a few reasons pinging a host can fail?
-
Name a tool you could use to see network packets reaching or leaving a Linux host?
-
How can you force
traceroute
to use a different interface? -
Is it possible for
traceroute
to report MTUs?
Summary
Networking is usually configured by a system’s startup scripts or a helper such as NetworkManager. Most distributions have tools that will edit the startup script configuration files for you. Consult your distribution’s documentation for details.
Being able to manually configure networking allows you to troubleshoot more effectively. It is useful in minimal environments used for things like restoring from backups or migrating to new hardware.
The utilities covered in this section have more functionality than covered in this lesson. It would be worthwhile to skim through the man page of each to familiarize yourself with the options available. The ss
and ip
commands are the modern way of doing things, while the rest that are covered, while still in common use, are considered legacy tools.
The best way to get familiar with the tools covered is practice. Using a computer with a modest amount of RAM, it is possible to setup a virtual network lab using virtual machines that you can practice with. Three virtual machines are enough to get comfortable with the tools listed.
The commands covered in this lesson include:
ping
andping6
-
Used to transmit ICMP packets to a remote host to test a network connection’s availability.
traceroute
andtraceroute6
-
Used to trace a path through a network to determine a network’s connectivity.
tracepath
andtracepath6
-
Used to trace a path through a network as well as determine MTU sizes along a route.
nc
-
Used to set up arbitrary connections on a network for testing connectivity, as well as querying a network for available services and devices.
netstat
-
Legacy command used to determine a system’s open network connections and statistics.
ss
-
Modern command used to determine a system’s open network connections and statistics.
Answers to Guided Exercises
-
What command(s) would you use to send an ICMP echo to
learning.lpi.org
?You would use
ping
orping6
:$ ping learning.lpi.org
or
$ ping6 learning.lpi.org
-
How could you determine the route to
8.8.8.8
?By using the
tracepath
ortraceroute
commands.$ tracepath 8.8.8.8
or
$ traceroute 8.8.8.8
-
What command would show you if any processes are listening on TCP port 80?
With
ss
:$ ss -ln | grep ":80"
With
netstat
:$ netstat -ln | grep ":80"
While not listed as a requirement for the exam, you can also use
lsof
:# lsof -Pi:80
-
How could you find which process is listening on a port?
Again, there are multiple ways to do this. You could use
lsof
in the same manner as the previous answer, replacing the port number. You could also usenetstat
orss
with the-p
option. Remember,netstat
is considered a legacy tool.# netstat -lnp | grep ":22"
The same options that work with
netstat
also work withss
:# ss -lnp | grep ":22"
-
How could you determine the max MTU of a network path?
By using the
tracepath
command:$ tracepath somehost.example.com
Answers to Explorational Exercises
-
How could you use netcat to send an HTTP request to a web server?
By entering the HTTP request line, any headers, and a blank line into the terminal:
$ nc learning.lpi.org 80 GET /index.html HTTP/1.1 HOST: learning.lpi.org HTTP/1.1 302 Found Location: https://learning.lpi.org:443/index.html Date: Wed, 27 May 2020 22:54:46 GMT Content-Length: 5 Content-Type: text/plain; charset=utf-8 Found
-
What are a few reasons pinging a host can fail?
There are a number of possible reasons. Here are some:
-
The remote host is down.
-
A router ACL is blocking your ping.
-
The remote host’s firewall is blocking your ping.
-
You may be using an incorrect host name or address.
-
Your name resolution is returning an incorrect address.
-
Your machine’s network configuration is incorrect.
-
Your machine’s firewall is blocking it.
-
The remote host’s network configuration is incorrect.
-
Your machine’s interface(s) are disconnected.
-
The remote machine’s interface(s) are disconnected.
-
A network component such as a switch, cable, or router between your machine and the remote’s is no longer functioning.
-
-
Name a tool you could use to see network packets reaching or leaving a Linux host?
Both
tcpdump
andwireshark
can be used. -
How can you force
traceroute
to use a different interface?By using the
-i
option:$ traceroute -i eth2 learning.lpi.org traceroute -i eth2 learning.lpi.org traceroute to learning.lpi.org (208.94.166.201), 30 hops max, 60 byte packets ...
-
Is it possible for
traceroute
to report MTUs?Yes, with the
--mtu
option:# traceroute -I --mtu learning.lpi.org traceroute to learning.lpi.org (208.94.166.201), 30 hops max, 65000 byte packets 1 047-132-144-001.res.spectrum.com (47.132.144.1) 9.974 ms F=1500 10.476 ms 4.743 ms 2 096-034-094-106.biz.spectrum.com (96.34.94.106) 8.697 ms 9.963 ms 10.321 ms ...