103.7 Lesson 2

One of the most common uses of grep is to facilitate the inspection of long files, using the regular expression as a filter applied to each line. It can be used to show only the lines starting with a certain term. For example, grep can be used to investigate a configuration file for kernel modules, listing only option lines:

The pipe | character can be employed to redirect the output of a command directly to grep's input. The following example uses a bracket expression to select lines from fdisk -l output, starting with Disk /dev/sda or Disk /dev/sdb:

The mere selection of lines with matches may not be appropriate for a particular task, requiring adjustments to grep's behavior through its options. For example, option -c or --count tells grep to show how many lines had matches:

The option can be placed before or after the regular expression. Other important grep options are:

Although activated by default when multiple file paths are given as input, the option -H is not activated for single files. That may be critical in special situations, like when grep is called directly by find, for instance:

In this example, find lists every file under /usr/share/doc then passes each one to grep, which in turn performs a case-insensitive search for 3d modeling inside the file. The pipe to cut is there just to limit output length to 100 columns. Note, however, that there is no way of knowing from which file the lines came from. This issue is solved by adding -H to grep:

Now it is possible to identify the files where each match was found. To make the listing even more informative, leading and trailing lines can be added to lines with matches:

The option -1 instructs grep to include one line before and one line after when it finds a line with a match. These extra lines are called context lines and are identified in the output by a minus sign after the file name. The same result can be obtained with -C 1 or --context=1 and other context line quantities may be indicated.

There are two complementary programs to grep: egrep and fgrep. The program egrep is equivalent to the command grep -E, which incorporates extra features other than the basic regular expressions. For example, with egrep it is possible to use extended regular expression features, like branching:

In this example either 3D modeling or 3D printing will match the expression, case-insensitive. To display only the parts of a text stream that match the expression used by egrep, use the -o option.

The program fgrep is equivalent to grep -F, that is, it does not parse regular expressions. It is useful in simple searches where the goal is to match a literal expression. Therefore, special characters like the dollar sign and the dot will be taken literally and not by their meanings in a regular expression.

The purpose of the sed program is to modify text-based data in a non-interactive way. It means that all the editing is made by predefined instructions, not by arbitrarily typing directly into a text displayed on the screen. In modern terms, sed can be understood as a template parser: given a text as input, it places custom content at predefined positions or when it finds a match for a regular expression.

Sed, as the name implies, is well suited for text streamed through pipelines. Its basic syntax is sed -f SCRIPT when editing instructions are stored in the file SCRIPT or sed -e COMMANDS to execute COMMANDS directly from the command line. If neither -f or -e are present, sed uses the first non-option parameter as the script file. It is also possible to use a file as the input just by giving its path as an argument to sed.

sed instructions are composed of a single character, possibly preceded by an address or followed by one or more options, and are applied to each line at a time. Addresses can be a single line number, a regular expression, or a range of lines. For example, the first line of a text stream can be deleted with 1d, where 1 specifies the line where the delete command d will be applied. To clarify sed 's usage, take the output of the command factor `seq 12`, which returns the prime factors for numbers 1 to 12:

Deleting the first line with sed is accomplished by 1d:

A range of lines can be specified with a separating comma:

More than one instruction can be used in the same execution, separated by semicolons. In this case, however, it is important to enclose them with parenthesis so the semicolon is not interpreted by the shell:

In this example, two deletion instructions were executed, first on lines ranging from 1 to 7 and then on line 11. An address can also be a regular expression, so only lines with a match will be affected by the instruction:

The regular expression 2.* matches with any occurrence of the number 2 anywhere after a colon, causing the deletion of lines corresponding to numbers with 2 as a factor. With sed, anything placed between slashes (/) is considered a regular expression and by default all basic RE is supported. For example, sed -e "/^#/d" /etc/services shows the contents of the file /etc/services without the lines beginning with # (comment lines).

The delete instruction d is only one of the many editing instructions provided by sed. Instead of deleting a line, sed can replace it with a given text:

The instruction c REMOVED simply replaces a line with the text REMOVED. In the example’s case, every line with a substring matching the regular expression *:.*2.* is affected by instruction c REMOVED. Instruction a TEXT copies text indicated by TEXT to a new line after the line with a match. The instruction r FILE does the same, but copies the contents of the file indicated by FILE. Instruction w does the opposite of r, that is, the line will be appended to the indicated file.

By far the most used sed instruction is s/FIND/REPLACE/, which is used to replace a match to the regular expression FIND with text indicated by REPLACE. For example, the instruction s/hda/sda/ replaces a substring matching the literal RE hda with sda. Only the first match found in the line will be replaced, unless the flag g is placed after the instruction, as in s/hda/sda/g.

A more realistic case study will help to illustrate sed's features. Suppose a medical clinic wants to send text messages to its customers, reminding them of their scheduled appointments for the next day. A typical implementation scenario relies on a professional instant message service, which provides an API to access the system responsible for delivering the messages. These messages usually originate from the same system that runs the application controlling customer’s appointments, triggered by a specific time of the day or some other event. In this hypothetical situation, the application could generate a file called appointments.csv containing tabulated data with all the appointments for the next day, then used by sed to render the text messages from a template file called template.txt. CSV files are a standard way of export data from database queries, so sample appointments could be given as follows:

The first line holds the labels for each column, which will be used to match the tags inside the sample template file:

The less than < and greater than > signs were put around labels just to help identify them as tags. The following Bash script parses all enqueued appointments using template.txt as the message template:

An actual production script would also handle authentication, error checking and logging, but the example has basic functionality to start with. The first instructions executed by sed are applied only to the first line — the address 1 in 1s/^"//;1s/","/\n/g;1s/"$//p — to remove the leading and trailing quotes — 1s/^"// and 1s/"$// — and to replace field separators with a newline character: 1s/","/\n/g. Only the first line is needed for loading column names, so non-matching lines will be suppressed by option -n, requiring flag p to be placed after the last sed command to print the matching line. The tags are then stored in the TAGS variable as a Bash array. Another Bash array variable is created by the command mapfile to store the lines containing the appointments in the array variable ROWS.

A for loop is employed to process each appointment line found in ROWS. Then, quotes and separators in the appointment — the appointment is in variable ${ROWS[$r]} used as a here string — are replaced by sed, similarly to the commands used to load the tags. The separated values for the appointment are then stored in the array variable VALS, where array subscripts 0, 1 and 2 correspond to values for NAME, TIME and PHONE.

Finally, a nested for loop walks through the TAGS array and replaces each tag found in the template with its corresponding value in VALS. The MSG variable holds a copy of the rendered template, updated by the substitution command s/<${TAGS[$c]}>/${VALS[$c]}/g on every loop pass through TAGS.

This results in a rendered message like: "Hey Carol, don’t forget your appointment tomorrow at 11am." The rendered message can then be sent as a parameter through a HTTP request with curl, as a mail message or any other similar method.

Commands grep and sed can be used together when more complex text mining is required. That is usually the case when parsing HTML files. For example, the CIA website (https://www.cia.gov/) publishes detailed information on every country (https://www.cia.gov/library/publications/resources/the-world-factbook/docs/one_page_summaries.html), a resource often used by travelers to gather information about their destinations. However, the summary information for each country is stored in an individual PDF file linked in a HTML file intended to be accessed by a web browser. By using sed and grep, a visitor wishing to gather information on many countries can avoid the need to manually click on each individual country to save its summary file.

To make things easier to read, first the external URL pointing to the HTML page may be stored in a variable:

Upon some inspection of the HTML file, it can be seen that every link to summary PDF files are uniform. Regardless of country, all links point to a file terminated by summary.pdf in an anchor tag, as in <a target=<blank" href="../attachments/summaries/BR-summary.pdf">Brazil</a>. Therefore, a sed substitution command with back references can be used to properly extract the PDF link and the country name:

The regular expression <href="\([^"]<\)">\([^>]<\)>.< matches the desired line and sets two back-references, which in turn are used to replace the entire line with \2\t\1. The \t character represents a horizontal tab space, useful as a field separator when using command cut:

Now the download link to the PDF can be easily assembled from the URL and PDF variables:

Commands like curl and wget copy remote files through HTTP. The suggested steps can be put together in a script, much like in the medical clinic example, to automate the download process. By using a conversion tool, like pdftotext, it’s also possible to parse the contents of the PDF.