103.4 Lesson 2

Unlike redirects, with pipes data flows from left to right in the command line and the target is another process, not a filesystem path, file descriptor or Here document. The pipe character | tells the shell to start all distinct commands at the same time and to connect the output of the previous command to the input of the following command, left to right. For example, instead of using redirects, the content of the file /proc/cpuinfo sent to the standard output by cat can be piped to the stdin of wc with the following command:

In the absence of a path to a file, wc counts the number of lines, words and characters it receives on its stdin, as is the case in the example. Many pipes can be present in a compound command. In the following example, two pipes are used:

The content of file /proc/cpuinfo produced by cat /proc/cpuinfo was piped to the command grep 'model name', which then selects only the lines containing the term model name. The machine running the example has many CPUs, so there are repeated lines with model name. The last pipe connects grep 'model name' to uniq, which is responsible for skipping any line equal to the previous one.

Pipes can be combined with redirects in the same command line. The previous example can be rewritten to a simpler form:

The input redirect for grep is not strictly necessary as grep accepts a file path as argument, but the example demonstrates how to build such combined commands.

Pipes and redirects are exclusive, that is, one source can be mapped to only one target. Yet, it is possible to redirect an output to a file and still see it on the screen with program tee. To do it, the first program sends its output to the stdin of tee and a file name is provided to the latter to store the data:

The output of the last program in the chain, generated by uniq, is displayed and stored in the file cpu_model.txt. To not overwrite the content of the provided file but to append data to it, the option -a must be provided to tee.

Only the standard output of a process is captured by a pipe. Let’s say you must to go through a long compilation process on the screen and at the same time save both the standard output and the standard error to a file for later inspection. Assuming your current directory does not have a Makefile, the following command will output an error:

Although shown on the screen, the error message generated by make was not captured by tee and the file log.txt was created empty. A redirect needs to be done before a pipe can capture the stderr:

In this example the stderr of make was redirected to the stdout, so tee was able to capture it with a pipe, display it on the screen and save it in the file log.txt. In cases like this, it may be useful to save the error messages for later inspection.

Another method to capture the output of a command is command substitution. By placing a command inside backquotes, Bash replaces it with its standard output. The following example shows how to use the stdout of a program as an argument to another program:

The output of the program date, the current date formatted as year-month-day, was used as an argument to create a directory with mkdir. An identical result is obtained by using $() instead of backquotes:

The same method can be used to store the output of a command as a variable:

The command uname -o outputs the generic name of the current operating system, which was stored in the session variable OS. To assign the output of a command to a variable is very useful in scripts, making it possible to store and evaluate the data in many distinct ways.

Depending on the output generated by the replaced command, the builtin command substitution may not be appropriate. A more sophisticated method to use the output of a program as the argument of another program employs an intermediate called xargs. The program xargs uses the contents it receives via stdin to run a given command with the contents as its argument. The following example shows xargs running the program identify with arguments provided by program find:

The program identify is part of ImageMagick, a set of command-line tools to inspect, convert and edit most image file types. In the example, xargs took all paths listed by find and put them as arguments to identify, which then shows the information for each file formatted as required by the option -format. The files found by find in the example are images containing the distribution logo in a Debian filesystem. -format is a parameter to identify, not to xargs.

Option -n 1 requires xargs to run the given command with only one argument at a time. In the example’s case, instead of passing all paths found by find as a list of arguments to identify, using xargs -n 1 would execute command identify for each path separately. Using -n 2 would execute identify with two paths as arguments, -n 3 with three paths as arguments and so on. Similarly, when xargs process multiline contents — as is the case with input provided by find — the option -L can be used to limit how many lines will be used as arguments per command execution.

If the paths have space characters, it is important to run find with the option -print0. This option instructs find to use a null character between each entry so the list can be correctly parsed by xargs (the output was suppressed):

The option -0 tells xargs the null character should be used as the separator. That way the file paths given by find are correctly parsed even if they have blank or other special characters in it. The previous example shows how to use the command du to find out the disk usage of every file found and then sort the results by size. The output was suppressed for conciseness. Note that for each search criteria it is necessary to place the -print0 option for find.

By default, xargs places the arguments of the executed command last. To change that behavior, the option -I should be used:

In the last example, every file found by find is moved to the current directory. As the source path(s) must be informed to mv before the target path, a substitution term is given to the option -I of xargs which is then appropriately placed alongside mv. By using the null character as separator, it is not necessary to enclose the substitution term with quotes.