3.2 Lesson 1

Certificate:

Linux Essentials

Version:

1.6

Topic:

3 The Power of the Command Line

Objective:

3.2 Searching and Extracting Data from Files

Lesson:

1 of 2

Introduction

In this lab we will be focusing on redirecting or transmitting information from one source to another with the help of specific tools. The Linux command line redirects the information through specific standard channels. The standard input (stdin or channel 0) of the command is considered to be the keyboard and the standard output (stdout or channel 1) is considered the screen. There is also another channel that is meant to redirect error output (stderr or channel 2) of a command or a program’s error messages. The input and/or output can be redirected.

When running a command, sometimes we want to transmit certain information to the command or redirect the output to a specific file. Each of these functionalities will be discussed in the next two sections.

I/O Redirection

I/O redirection enables the user to redirect information from or to a command by using a text file. As described earlier, the standard input, output and error output can be redirected, and the information can be taken from text files.

Redirecting Standard Output

To redirect standard output to a file, instead of the screen, we need to use the > operator followed by the name of the file. If the file doesn’t exist, a new one will be created, otherwise, the information will overwrite the existing file.

In order to see the contents of the file that we just created, we can use the cat command. By default, this command displays the contents of a file on the screen. Consult the manual page to find out more about its functionalities.

The example below demonstrates the functionality of the operator. In the first instance, a new file is created containing the text “Hello World!”:

$ echo "Hello World!" > text
$ cat text
Hello World!

In the second invocation, the same file is overwritten with the new text:

$ echo "Hello!" > text
$ cat text
Hello!

If we want to add new information at the end of the file, we need to use the >> operator. This operator also creates a new file if it cannot find an existing one.

The first example shows the addition of the text. As it can be seen, the new text was added on the following line:

$ echo "Hello to you too!" >> text
$ cat text
Hello!
Hello to you too!

The second example demonstrates that a new file will be created:

$ echo "Hello to you too!" >> text2
$ cat text2
Hello to you too!

Redirecting Standard Error

In order to redirect just the error messages, a user will need to employ the 2> operator followed by the name of the file in which the errors will be written. If the file doesn’t exist, a new one will be created, otherwise the file will be overwritten.

As explained, the channel for redirecting the standard error is channel 2. When redirecting the standard error, the channel must be specified, contrary to the other standard output where channel 1 is set by default. For example, the following command searches for a file or directory named games and only writes the error into the text-error file, while displaying the standard output on the screen:

$ find /usr games 2> text-error
/usr
/usr/share
/usr/share/misc
---------Omitted output----------
/usr/lib/libmagic.so.1.0.0
/usr/lib/libdns.so.81
/usr/games
$ cat text-error
find: `games': No such file or directory

Note	For more information about the `find` command, consult its man page.

For example, the following command will run without errors, therefore no information will be written in the file text-error:

$ sort /etc/passwd 2> text-error
$ cat text-error

As well as the standard output, the standard error can also be appended to a file with the 2>> operator. This will add the new error at the end of the file. If the file doesn’t exist, a new one will be created. The first example shows the addition of the new information into the file, whereas the second example shows that the command creates a new file where an existing one can’t be found with the same name:

$ sort /etc 2>> text-error
$ cat text-error
sort: read failed: /etc: Is a directory

$ sort /etc/shadow 2>> text-error2
$ cat text-error2
sort: open failed: /etc/shadow: Permission denied

Using this type of redirection, only the error messages will be redirected to the file, the normal output will be written on the screen or go through standard output or stdout.

There is one particular file that technically is a bit bucket (a file that accepts input and doesn’t do anything with it): /dev/null. You can redirect any irrelevant information that you might not want displayed or redirected into an important file, as shown in the example below:

$ sort /etc 2> /dev/null

Redirecting Standard Input

This type of redirection is used to input data to a command, from a specified file instead of a keyboard. In this case the < operator is used as shown in the example:

$ cat < text
Hello!
Hello to you too!

Redirecting standard input is usually used with commands that don’t accept file arguments. The tr command is one of them. This command can be used to translate file contents by modifying the characters in a file in specific ways, like deleting any particular character from a file, the example below shows the deletion of the character l:

$ tr -d "l" < text
Heo!
Heo to you too!

For more information, consult the man page of tr.

Here Documents

Unlike the output redirections, the << operator acts in a different way compared to the other operators. This input stream is also called here document. Here document represents the block of code or text which can be redirected to the command or the interactive program. Different types of scripting languages, like bash, sh and csh are able to take input directly from the command line, without using any text files.

As can be seen in the example below, the operator is used to input data into the command, while the word after doesn’t specify the file name. The word is interpreted as the delimiter of the input and it will not be taken in consideration as content, therefore cat will not display it:

$ cat << hello
> hey
> ola
> hello
hey
ola

Consult the man page of the cat command to find more information.

Combinations

The first combination that we will explore combines the redirection of the standard output and standard error output to the same file. The &> and &>> operators are used, & representing the combination of channel 1 and channel 2. The first operator will overwrite the existing contents of the file and the second one will append or add the new information at the end of the file. Both operators will enable the creation of the new file if it doesn’t exist, just like in the previous sections:

$ find /usr admin &> newfile
$ cat newfile
/usr
/usr/share
/usr/share/misc
---------Omitted output----------
/usr/lib/libmagic.so.1.0.0
/usr/lib/libdns.so.81
/usr/games
find: `admin': No such file or directory
$ find /etc/calendar &>> newfile
$ cat newfile
/usr
/usr/share
/usr/share/misc
---------Omitted output----------
/usr/lib/libmagic.so.1.0.0
/usr/lib/libdns.so.81
/usr/games
find: `admin': No such file or directory
/etc/calendar
/etc/calendar/default

Let’s take a look at an example using the cut command:

$ cut -f 3 -d "/" newfile
$ cat newfile

share
share
share
---------Omitted output----------
lib
games
find: `admin': No such file or directory
calendar
calendar
find: `admin': No such file or directory

The cut command cuts specified fields from the input file by using the -f option, the 3rd field in our case. In order for the command to find the field, a delimiter needs to be specified as well with the -d option. In our case the delimiter will be the / character.

To find more about the cut command, consult its man page.

Command Line Pipes

Redirection is mostly used to store the result of a command, to be processed by a different command. This type of intermediate process can become very tedious and complicated if you want the data to go through multiple processes. In order to avoid this, you can link the command directly via pipes. In other words, the first command’s output automatically becomes the second command’s input. This connection is made by using the | (vertical bar) operator:

$ cat /etc/passwd | less
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
:

In the example above, the less command after the pipe operator modifies the way that the file is displayed. The less command displays the text file allowing the user to scroll up and down a line at the time. less is also used by default to display the man pages, as discussed in the previous lessons.

It is possible to use multiple pipes at the same time. The intermediate commands that receive input then change it and produce output are called filters. Let’s take the ls -l command and try to count the number of words from the first 10 lines of the output. In order to do this, we will have to use the head command that by default displays the first 10 lines of a file and then count the words using the wc command:

$ ls -l | head | wc -w
10

As mentioned earlier, by default, head only displays the first 10 lines of the text file specified. This behaviour can be modified by using specific options. Check the command’s man page to find more.

There is another command that displays the end of a file: tail. By default, this command selects the last 10 lines and displays them, but as head the number can also be modified. Check tail’s man page for more details.

Note	The `-f` option can show the last lines of a file while it’s being updated. This feature can become very useful when monitoring a file like `syslog` for ongoing activity.

The wc (word count) command counts by default the lines, words and bytes of a file. As shown in the exercise, the -w option causes the command to only count the words within the selected lines. The most common options that you can use with this command are: -l, which specifies to the command to only count the lines, and -c, which is used to count only the bytes. More variations and options of the command, as well as more information about wc can be found within the command’s man page.

Guided Exercises

List the contents of your current directory, including the ownership and permissions, and redirect the output to a file called contents.txt within your home directory.
Sort the contents of the contents.txt file from your current directory and append it to the end of a new file named contents-sorted.txt.
Display the last 10 lines of the /etc/passwd file and redirect it to a new file in your user’s Documents directory.
Count the number of words within the contents.txt file and append the output to the end of a file field2.txt in your home directory. You will need to use both input and output redirection.
Display the first 5 lines of the /etc/passwd file and sort the output reverse alphabetically.
Using the previously created contents.txt file, count the number of characters of the last 9 lines.
Count the number of files called test within the /usr/share directory and its subdirectories. Note: each line output from the find command represents a file.

Explorational Exercises

Select the second field of the contents.txt file and redirect the standard output and error output to another file called field1.txt.
Using the input redirection operator and the tr command, delete the dashes (-) from the contents.txt file.
What is the biggest advantage of only redirecting errors to a file?
Replace all recurrent spaces within the alphabetically sorted contents.txt file with a single space.
In one command line, eliminate the recurrent spaces (as done in the previous exercise), select the ninth field and sort it reverse alphabetically and non-case sensitive. How many pipes did you have to use?

Summary

In this lab you learned:

Types of redirection
How to use the redirection operators
How to use pipes to filter command output

Commands used in this lesson:

cut: Removes sections from each line of a file.
cat: Displays or concatenates files.
find: Searches for files in a directory hierarchy.
less: Displays a file, allowing the user to scroll one line at the time.
more: Displays a file, a page at the time.
head: Displays the first 10 lines of a file.
tail: Displays the last 10 lines of a file.
sort: Sorts files.
wc: Counts by default the lines, words or bytes of a file.

Answers to Guided Exercises

List the contents of your current directory, including the ownership and permissions, and redirect the output to a file called contents.txt within your home directory.
```
$ ls -l > contents.txt
```
Sort the contents of the contents.txt file from your current directory and append it to the end of a new file named contents-sorted.txt.
```
$ sort contents.txt >> contents-sorted.txt
```
Display the last 10 lines of the /etc/passwd file and redirect it to a new file in the your user’s Documents directory.
```
$ tail /etc/passwd > Documents/newfile
```
Count the number of words within the contents.txt file and append the output to the end of a file field2.txt in your home directory. You will need to use both input and output redirection.
```
$ wc < contents.txt >> field2.txt
```
Display the first 5 lines of the /etc/passwd file and sort the output reverse alphabetically.
```
$ head -n 5 /etc/passwd | sort -r
```
Using the previously created contents.txt file, count the number of characters of the last 9 lines.
```
$ tail -n 9 contents.txt | wc -c
531
```
Count the number of files called test within the /usr/share directory and its subdirectories. Note: each line output from the find command represents a file.
```
$ find /usr/share -name test | wc -l
125
```

Answers to Explorational Exercises

Select the second field of the contents.txt file and redirect the standard output and error output to another file called field1.txt.
```
$ cut -f 2 -d " " contents.txt &> field1.txt
```
Using the input redirection operand and the tr command, delete the dashes (-) from the contents.txt file.
```
$ tr -d "-" < contents.txt
```
What is the biggest advantage of only redirecting errors to a file?

Only redirecting errors to a file can help with keeping a log file that is monitored frequently.
Replace all recurrent spaces within the alphabetically sorted contents.txt file with a single space.
```
$ sort contents.txt | tr -s " "
```
In one command line, eliminate the recurrent spaces (as done in the previous exercise), select the ninth field and sort it reverse alphabetically and non-case sensitive. How many pipes did you have to use?
```
$ cat contents.txt | tr -s " " | cut -f 9 -d " " | sort -fr
```
The exercise uses 3 pipes, one for each filter.