105.2 Lesson 2
Certificate: |
LPIC-1 |
---|---|
Version: |
5.0 |
Topic: |
105 Shells and Shell Scripting |
Objective: |
105.2 Customize or write simple scripts |
Lesson: |
2 of 2 |
Introduction
Shell scripts are generally intended to automate operations related to files and directories, the same operations that could be performed manually at the command line. Yet, the coverage of shell scripts are not only restricted to a user’s documents, as the configuration and interaction with many aspects of a Linux operating system are also accomplished through script files.
The Bash shell offers many helpful builtin commands to write shell scripts, but the full power of these scripts relies on the combination of Bash builtin commands with the many command line utilities available on a Linux system.
Extended Tests
Bash as a script language is mostly oriented to work with files, so the Bash builtin command test
has many options to evaluate the properties of filesystem objects (essentially files and directories). Tests that are focused on files and directories are useful, for example, to verify if the files and directories required to perform a particular task are present and can be read. Then, associated with an if conditional construct, the proper set of actions are executed if the test is successful.
The test
command can evaluate expressions using two different syntaxes: test expressions can be given as an argument to command test
or they can be placed inside square brackets, where command test
is given implicitly. Thus, the test to evaluate if /etc
is a valid directory can be written as test -d /etc
or as [ -d /etc]
:
$ test -d /etc $ echo $? 0 $ [ -d /etc ] $ echo $? 0
As confirmed by the exit status codes in the special variable $?
— a value of 0 means the test was successful — both forms evaluated /etc
as a valid directory. Assuming the path to a file or directory was stored in the variable $VAR
, the following expressions can be used as arguments to test
or inside square brackets:
-a "$VAR"
-
Evaluate if the path in
VAR
exists in the filesystem and it is a file. -b "$VAR"
-
Evaluate if the path in
VAR
is a special block file. -c "$VAR"
-
Evaluate if the path in
VAR
is a special character file. -d "$VAR"
-
Evaluate if the path in
VAR
is a directory. -e "$VAR"
-
Evaluate if the path in
VAR
exists in the filesystem. -f "$VAR"
-
Evaluate if the path in
VAR
exists and it is a regular file. -g "$VAR"
-
Evaluate if the path in
VAR
has the SGID permission. -h "$VAR"
-
Evaluate if the path in
VAR
is a symbolic link. -L "$VAR"
-
Evaluate if the path in
VAR
is a symbolic link (like-h
). -k "$VAR"
-
Evaluate if the path in
VAR
has the sticky bit permission. -p "$VAR"
-
Evaluate if the path in
VAR
is a pipe file. -r "$VAR"
-
Evaluate if the path in
VAR
is readable by the current user. -s "$VAR"
-
Evaluate if the path in
VAR
exists and it is not empty. -S "$VAR"
-
Evaluate if the path in
VAR
is a socket file. -t "$VAR"
-
Evaluate if the path in
VAR
is open in a terminal. -u "$VAR"
-
Evaluate if the path in
VAR
has the SUID permission. -w "$VAR"
-
Evaluate if the path in
VAR
is writable by the current user. -x "$VAR"
-
Evaluate if the path in
VAR
is executable by the current user. -O "$VAR"
-
Evaluate if the path in
VAR
is owned by the current user. -G "$VAR"
-
Evaluate if the path in
VAR
belongs to the effective group of the current user. -N "$VAR"
-
Evaluate if the path in
VAR
has been modified since the last time it was accessed. "$VAR1" -nt "$VAR2"
-
Evaluate if the path in
VAR1
is newer than the path inVAR2
, according to their modification dates. "$VAR1" -ot "$VAR2"
-
Evaluate if the path in
VAR1
is older thanVAR2
. "$VAR1" -ef "$VAR2"
-
This expression evaluates to True if the path in
VAR1
is a hardlink toVAR2
.
It is recommended to use the double quotes around a tested variable because, if the variable happens to be empty, it could cause a syntax error for the test
command. The test options require an operand argument and an unquoted empty variable would cause an error due to a missing required argument. There are also tests for arbitrary text variables, described as follows:
-z "$TXT"
-
Evaluate if variable
TXT
is empty (zero size). -n "$TXT"
ortest "$TXT"
-
Evaluate if variable
TXT
is not empty. "$TXT1" = "$TXT2"
or"$TXT1" == "$TXT2"
-
Evaluate if
TXT1
andTXT2
are equal. "$TXT1" != "$TXT2"
-
Evaluate if
TXT1
andTXT2
are not equal. "$TXT1" < "$TXT2"
-
Evaluate if
TXT1
comes beforeTXT2
, in alphabetical order. "$TXT1" > "$TXT2"
-
Evaluate if
TXT1
comes afterTXT2
, in alphabetical order.
Distinct languages may have different rules for alphabetical ordering. To obtain consistent results, regardless of the localization settings of the system where the script is being executed, it is recommended to set the environment variable LANG
to C
, as in LANG=C
, before doing operations involving alphabetical ordering. This definition will also keep the system messages in the original language, so it should be used only within script scope.
Numerical comparisons have their own set of test options:
$NUM1 -lt $NUM2
-
Evaluate if
NUM1
is less thanNUM2
. $NUM1 -gt $NUM2
-
Evaluate if
NUM1
is greater thanNUM2
. $NUM1 -le $NUM2
-
Evaluate if
NUM1
is less or equal toNUM2
. $NUM1 -ge $NUM2
-
Evaluate if
NUM1
is greater or equal toNUM2
. $NUM1 -eq $NUM2
-
Evaluate if
NUM1
is equal toNUM2
. $NUM1 -ne $NUM2
-
Evaluate if
NUM1
is not equal toNUM2
.
All tests can receive the following modifiers:
! EXPR
-
Evaluate if the expression
EXPR
is false. EXPR1 -a EXPR2
-
Evaluate if both
EXPR1
andEXPR2
are true. EXPR1 -o EXPR2
-
Evaluate if at least one of the two expressions are true.
Another conditional construct, case
, can be seen as a variation of the if construct. The instruction case
will execute a list of given commands if a specified item — the content of a variable, for example — can be found in a list of items separated by pipes (the vertical bar |
) and terminated by )
. The following sample script shows how the case
construct can be used to indicate the corresponding software packaging format for a given Linux distribution:
#!/bin/bash DISTRO=$1 echo -n "Distribution $DISTRO uses " case "$DISTRO" in debian | ubuntu | mint) echo -n "the DEB" ;; centos | fedora | opensuse ) echo -n "the RPM" ;; *) echo -n "an unknown" ;; esac echo " package format."
Each list of patterns and associated commands must be terminated with ;;
, ;&
, or ;;&
. The last pattern, an asterisk, will match if no other previous pattern corresponded beforehand. The esac
instruction (case backwards) terminates the case
construct. Assuming the previous sample script was named script.sh
and it is executed with opensuse
as the first argument, the following output will be generated:
$ ./script.sh opensuse Distribution opensuse uses the RPM package format.
Tip
|
Bash has an option called |
The searched item and the patterns undergo tilde expansion, parameter expansion, command substitution and arithmetic expansion. If the searched item is specified with quotes, they will be removed before matching is attempted.
Loop Constructs
Scripts are often used as a tool to automate repetitive tasks, performing the same set of commands until a stop criteria is verified. Bash has three loop instructions — for
, until
and while
— designed for slightly distinct loop constructions.
The for
construct walks through a given list of items — usually a list of words or any other space-separated text segments — executing the same set of commands on each one of those items. Before each iteration, the for
instruction assigns the current item to a variable, which can then be used by the enclosed commands. The process is repeated until there are no more items left. The syntax of the for
construct is:
for VARNAME in LIST do COMMANDS done
VARNAME
is an arbitrary shell variable name and LIST
is any sequence of separated terms. The valid delimiting characters splitting items in the list are defined by the IFS
environment variable, which are the characters space, tab and newline by default. The list of commands to be executed is delimited by the do
and done
instructions, so commands can occupy as much lines as needed.
In the following example, the for
command will take each item from the provided list — a sequence of numbers — and assign it to the NUM
variable, one item at a time:
#!/bin/bash for NUM in 1 1 2 3 5 8 13 do echo -n "$NUM is " if [ $(( $NUM % 2 )) -ne 0 ] then echo "odd." else echo "even." fi done
In the example, a nested if
construct is used in conjunction with an arithmetic expression to evaluate if the number in the current NUM
variable is even or odd. Assuming the previous sample script was named script.sh
and it is in the current directory, the following output will be generated:
$ ./script.sh 1 is odd. 1 is odd. 2 is even. 3 is odd. 5 is odd. 8 is even. 13 is odd.
Bash also supports an alternative format to for
constructs, with the double parenthesis notation. This notation resembles the for
instruction syntax from the C programming language and it is particularly useful for working with arrays:
#!/bin/bash SEQ=( 1 1 2 3 5 8 13 ) for (( IDX = 0; IDX < ${#SEQ[*]}; IDX++ )) do echo -n "${SEQ[$IDX]} is " if [ $(( ${SEQ[$IDX]} % 2 )) -ne 0 ] then echo "odd." else echo "even." fi done
This sample script will generate the exact same output as the previous example. However, instead of using the NUM
variable to store one item at a time, the IDX
variable is employed to track the current array index in ascending order, starting from 0 and continuously adding to it while it is under the number of items in the SEQ
array. The actual item is retrieved from its array position with ${SEQ[$IDX]}
.
In the same fashion, the until
construct executes a command sequence until a test command — like the test
command itself — terminates with status 0 (success). For example, the same loop structure from the previous example can be implemented with until
as follows:
#!/bin/bash SEQ=( 1 1 2 3 5 8 13 ) IDX=0 until [ $IDX -eq ${#SEQ[*]} ] do echo -n "${SEQ[$IDX]} is " if [ $(( ${SEQ[$IDX]} % 2 )) -ne 0 ] then echo "odd." else echo "even." fi IDX=$(( $IDX + 1 )) done
until
constructs may require more instructions than for
constructs, but it can be more suitable to non-numerical stop criteria provided by test
expressions or any other command. It is important to include actions that ensure a valid stop criteria, like the increment of a counter variable, otherwise the loop may run indefinitely.
The instruction while
is similar to the instruction until
, but while
keeps repeating the set of commands if the test command terminates with status 0 (success). Therefore, the instruction until [ $IDX -eq ${#SEQ[*]} ]
from the previous example is equivalent to while [ $IDX -lt ${#SEQ[*]} ]
, as the loop should repeat while the array index is less than the total of items in the array.
A More Elaborate Example
Imagine a user wants to periodically sync a collection of their files and directories with another storage device, mounted at an arbitrary mount point in the filesystem, and a full featured backup system is considered overkill. Since this is an activity intended to be performed periodically, it is a good application candidate for automating with a shell script.
The task is straight forward: sync every file and directory contained in a list, from an origin directory informed as the first script argument to a destination directory informed as the second script argument. To make it easier to add or remove items from the list, it will be kept in a separated file, ~/.sync.list
, one item per line:
$ cat ~/.sync.list Documents To do Work Family Album .config .ssh .bash_profile .vimrc
The file contains a mixture of files and directories, some with blank spaces in their names. This is a suitable scenario for the builtin Bash command mapfile
, which will parse any given text content and create an array variable from it, placing each line as an individual array item. The script file will be named sync.sh
, containing the following script:
#!/bin/bash set -ef # List of items to sync FILE=~/.sync.list # Origin directory FROM=$1 # Destination directory TO=$2 # Check if both directories are valid if [ ! -d "$FROM" -o ! -d "$TO" ] then echo Usage: echo "$0 <SOURCEDIR> <DESTDIR>" exit 1 fi # Create array from file mapfile -t LIST < $FILE # Sync items for (( IDX = 0; IDX < ${#LIST[*]}; IDX++ )) do echo -e "$FROM/${LIST[$IDX]} \u2192 $TO/${LIST[$IDX]}"; rsync -qa --delete "$FROM/${LIST[$IDX]}" "$TO"; done
The first action the script does is to redefine two shell parameters with command set
: option -e
will exit execution immediately if a command exits with a non-zero status and option -f
will disable file name globbing. Both options can be shortened as -ef
. This is not a mandatory step, but helps diminish the probability of unexpected behaviour.
The actual application oriented instructions of the script file can be divided in three parts:
-
Collect and check script parameters
The
FILE
variable is the path to the file containing the list of items to be copied:~/.sync.list
. VariablesFROM
andTO
are the origin and destination paths, respectively. Since these last two parameters are user provided, they go through a simple validation test performed by theif
construct: if any of the two is not a valid directory — assessed by the test[ ! -d "$FROM" -o ! -d "$TO" ]
— the script will show a brief help message and then terminate with an exit status of 1. -
Load list of files and directories
After all the parameters are defined, an array containing the list of items to be copied is created with the command
mapfile -t LIST < $FILE
. Option-t
ofmapfile
will remove the trailing newline character from each line before including it in the array variable namedLIST
. The contents of the file indicated by the variableFILE
—~/.sync.list
— is read via input redirection. -
Perform the copy and inform the user
A
for
loop using double parenthesis notation traverses the array of items, with theIDX
variable keeping track of the index increment. Theecho
command will inform the user of each item being copied. The escaped unicode character —\u2192
— for the right arrow character is present in the output message, so the-e
option of commandecho
must be used. Commandrsync
will selectively copy only the modified file pieces from the origin, thus its use is recommended for such tasks.rsync
options-q
and-a
, condensed in-qa
, will inhibitrsync
messages and activate the archive mode, where all file properties are preserved. Option--delete
will makersync
delete an item in the destination that does not exist in the origin anymore, so it should be used with care.
Assuming all the items in the list exist in the home directory of user carol
, /home/carol
, and the destination directory /media/carol/backup
points to a mounted external storage device, the command sync.sh /home/carol /media/carol/backup
will generate the following output:
$ sync.sh /home/carol /media/carol/backup /home/carol/Documents → /media/carol/backup/Documents /home/carol/"To do" → /media/carol/backup/"To do" /home/carol/Work → /media/carol/backup/Work /home/carol/"Family Album" → /media/carol/backup/"Family Album" /home/carol/.config → /media/carol/backup/.config /home/carol/.ssh → /media/carol/backup/.ssh /home/carol/.bash_profile → /media/carol/backup/.bash_profile /home/carol/.vimrc → /media/carol/backup/.vimrc
The example also assumes the script is executed by root or by user carol
, as most of the files would be unreadable by other users. If script.sh
is not inside a directory listed in the PATH
environment variable, then it should be specified with its full path.
Guided Exercises
-
How could command
test
be used to verify if the file path stored in the variableFROM
is newer than a file whose path is stored in the variableTO
? -
The following script should print a number sequence from 0 to 9, but instead it indefinitely prints 0. What should be done in order to get the expected output?
#!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ] do echo $COUNTER done
-
Suppose a user wrote a script that requires a sorted list of usernames. The resulting sorted list is presented as the following on his computer:
carol Dave emma Frank Grace henry
However, the same list is sorted as the following on his colleague’s computer:
Dave Frank Grace carol emma henry
What could explain the differences between the two sorted lists?
Explorational Exercises
-
How could all of the script’s command line arguments be used to initialize a Bash array?
-
Why is it that, counter intuitively, the command
test 1 > 2
evaluates as true? -
How would a user temporarily change the default field separator to the newline character only, while still being able to revert it to its original content?
Summary
This lesson takes a deeper look at the tests available for the test
command and at other conditional and loop constructs, necessary to write more elaborate shell scripts. A simple file synchronization script is given as an example of a practical shell script application. The lesson goes through the following steps:
-
Extended tests for the
if
andcase
conditional constructs. -
Shell loop constructs:
for
,until
andwhile
. -
Iterating through arrays and parameters.
The commands and procedures addressed were:
test
-
Perform a comparison between items supplied to the command.
if
-
A logic construct used in scripts to evaluate something as either true or false, then branch command execution based on results.
case
-
Evaluate several values against a single variable. Script command execution is then carried out depending on the
case
command’s result. for
-
Repeat execution of a command based on a given criteria.
until
-
Repeat the execution of a command until an expression evaluates to false.
while
-
Repeat the execution of a command while a given expression evaluates to true.
Answers to Guided Exercises
-
How could command
test
be used to verify if the file path stored in the variableFROM
is newer than a file whose path is stored in the variableTO
?The command
test "$FROM" -nt "$TO"
will return a status code of 0 if the file in theFROM
variable is newer than the file in theTO
variable. -
The following script should print a number sequence from 0 to 9, but instead it indefinitely prints 0. What should be done in order to get the expected output?
#!/bin/bash COUNTER=0 while [ $COUNTER -lt 10 ] do echo $COUNTER done
The
COUNTER
variable should be incremented, which could be done with the arithmetic expressionCOUNTER=$(( $COUNTER + 1 ))
, to eventually reach the stop criteria and end the loop. -
Suppose a user wrote a script that requires a sorted list of usernames. The resulting sorted list is presented as the following in his computer:
carol Dave emma Frank Grace henry
However, the same list is sorted as the following in his colleague computer:
Dave Frank Grace carol emma henry
What could explain the differences between the two sorted lists?
The sorting is based on the current system’s locale. To avoid the inconsistencies, sorting tasks should be performed with the
LANG
environment variable set toC
.
Answers to Explorational Exercises
-
How could all of the script’s command line arguments be used to initialize a Bash array?
The commands
PARAMS=( $* )
orPARAMS=( "$@" )
will create an array calledPARAMS
with all the arguments. -
Why is it that, counter intuitively, the command
test 1 > 2
evaluates as true?Operator
>
is intended to be used with string tests, no numerical tests. -
How would a user temporarily change the default field separator to the newline character only, while still being able to revert it to its original content?
A copy of the
IFS
variable can be stored in another variable:OLDIFS=$IFS
. Then the new line separator is defined withIFS=$'\n'
and the IFS variable can be reverted back withIFS=$OLDIFS
.