shell scripting
Linux

Shell scripting the awesome guide part4

In the previous post, we talked about parameters and options in detail, today we will talk about something very important in shell scripting which are Input, Output, and Redirection.

So far, you’ve seen two methods for displaying the output from your shell scripts

  • Displaying output on the screen
  • Redirecting output to a file

So how Linux handles input and output?

Standard File Descriptors

Everything is a file in Linux and that includes input and output.

Each process is allowed to have up to nine open file descriptors at a time. The bash shell reserves the first three file descriptors 0, 1, 2

0              STDIN                  Standard input

1              STDOUT              Standard output

2              STDERR              Standard error

These three special file descriptors handle the input and output from your shell script.

You need to fully understand these three because they are like the backbones of your shell scripting. So we are going to describe every one of them in detail.

STDIN

This stands for the standard input to the shell. For a terminal, the standard input is the keyboard.

When you use the input redirect symbol (<) in shell scripting, Linux replaces the standard input file descriptor with the file referenced. It reads the file and sends the data just as if it were typed on the keyboard. No magic.

Many bash commands accept input from STDIN If no files are specified on the command line like the cat command.

When you type the cat command in the command line without anything, it accepts input from STDIN. Any line you type, the cat command prints that line to the screen.

STDOUT

This stands for the standard output for the shell. The standard output is the screen.

Most bash commands direct their output to the STDOUT file descriptor by default which is the screen.

You can also append data to a file. You can do this using the >> symbol.

So if we have a file contains data, we can append other data to it using this symbol like this:

pwd >> myfile

The output generated by pwd is appended to myfile without deleting the existed content.

shell-scripting-append

The following command tries to redirect the output to a file using > symbol.

ls l xfile > myfile

shell-scripting-redirect-error

I have no file called xfile on my PC, and that generates an error. The shell doesn’t redirect the error message to the output redirection file, but the error message appeared on the screen and this is the third type of file descriptors.

STDERR

This file descriptor standard error output of the shell.

By default, the STDERR file descriptor points to the same place as the STDOUT file descriptor that’s why when an error occurs you see the error on the screen.

So you need to redirect the errors to a log file instead of viewing it on the screen.

Redirecting Errors

As we mentioned, the STDERR file descriptor is set to the value 2. We can redirect the errors by placing the file descriptor before the redirection symbol like this:

ls -l xfile 2>myfile

cat ./myfile

shell-scripting-redirect-error-to-file

As you can see, the error now is in the file and nothing on the screen.

Redirecting Errors and Normal Output

In shell scripting, if you want to redirect both errors and the normal output, you need to precede each with the appropriate file descriptor for the data you want to redirect like this:

ls l myfile xfile anotherfile 2> errorcontent 1> correctcontent

shell-scripting-redirect-error-and-data

The shell redirects the normal output of the ls command that should go to STDOUT to the correctcontent file using the 1> symbol. And error messages that should go to STDERR were redirected to the errorcontent file using the 2> symbol.

You can redirect both STDERR and STDOUT to the same file, use &> symbol like this:

ls l myfile xfile anotherfile &> content

shell-scripting-redirect-all-to-file

All error and standard output are redirected to file named content.

Redirecting Output in Scripts

There are two methods for redirecting output in shell scripting:

  • Temporarily redirection
  • Permanently redirection

Temporary redirections

You can redirect any individual output line to STDERR. You can use the output redirection symbol to redirect the output to the STDERR file descriptor and you must precede the file descriptor number with an ampersand (&) like this:

You can use the output redirection symbol to redirect the output to the STDERR file descriptor and you must precede the file descriptor number with an ampersand (&) like this:

shell-scripting-temp-redirection

So if we run it, we will see both lines printed normally because as we know STDERR output to STDOUT.

You can redirect STDERR like this:

./myscript 2> myfile

shell-scripting-redirect-error-to-file

Shell scripting is Awesome! The text that was sent to STDOUT appears on the screen, while the echo statement which sends output to STDERR is redirected to the output file.

Permanent redirections

If you have lots of data that need to be redirected in your script, it would be hard to redirect every echo statement. Instead, you can redirect to a specific file descriptor for the duration of the script execution by using the exec command like this:

shell-scripting-redirect-all-to-file

If we look at the file called outfile, we will see the output of echo lines.

You can also redirect the STDOUT in the middle of a script like this:

shell-scripting-permenant-redirection

The exec command redirects any output going to STDERR to the file myerror, then the script uses the echo statement to display a few lines to STDOUT which is the screen.

After that, the exec command is used again to redirect STDOUT to the myfile file and finally, we redirect the error from within the echo statement to go to STDERR which in this case is myerror file.

Redirecting Input in Scripts

You can use the same technique you’ve learned to redirect the output to redirect input. The exec command enables you to redirect STDIN from a file like this:

exec 0< myfile

This command tells the shell to take the input from a file called myfile instead of STDIN and here is an example:

shell-scripting-redirect-input

Shell scripting is easy.

I showed you in the previous post how to use the read command to read data entered from the keyboard by a user. By redirecting STDIN to a file, when the read command tries to read from STDIN, it returns data from the file instead of the keyboard.

Some Linux system administrators use this technique to read the log files for processing and we will discuss more ways to read the log on the upcoming posts in a professional way.

Creating Your Own Redirection

When you redirect input and output in your shell script, you’re not limited to the three default file descriptors. As I mentioned that you could have up to nine open file descriptors in the shell. The other six file descriptors from 3 through 8 and are available for you to use as either input or output redirection. You can assign any of these file descriptors to a file and then use them in your shell scripts.

You can assign a file descriptor for output by using the exec command and here’s an example how to do that:

shell-scripting-create-redirection

Creating Input File Descriptors

You can redirect input file descriptors in shell scripting exactly the same way as output file descriptors. Save the STDIN file descriptor location to another file descriptor before redirecting it to a file.

When you finish reading the file, you can restore STDIN to its original location:

shell-scripting-create-input-file-descriptor

In this example, file descriptor 6 is used to hold the location as STDIN. The shell script then redirects STDIN to a file. All the input for the read command comes from the redirected STDIN, which is now the input file.

After all the lines have been read, the shell script returns STDIN to its original location. And the shell script makes sure that STDIN is back to normal by using another read command, and now it is waiting for your keyboard input.

Closing File Descriptors

The shell automatically closes the file descriptors when the script exits. There are situations you need to manually close a file descriptor before the end of the script. To close a file descriptor, redirect it to the special symbol &- like this:

exec 3>&-

 

shell-scripting-closing-file-desciptor

As you can see, it gives error bad file descriptor because it is no longer exist.

Note: be careful when closing file descriptors. If you open the same output file later in your shell script, the shell replaces the existing file with a new file. That means if you output any data, it will overwrite the existing file.

Listing Open File Descriptors

The lsof command lists all the open file descriptors on the entire Linux system.

On many Linux systems like Fedora, the lsof command is located under /usr/sbin.

This command is very useful, it displays information about the currently opened files on the Linux system. This includes all the running processes on background, as well as any user accounts logged into the system.

This command has a lot of options so I think I will make a special post about it later, but let’s take the important parameters that we need.

-p, allows you to specify a process ID.

-d, allows you to specify the file descriptor numbers to display.

To get the current PID of the process, you can use the special environment variable $$, which the shell sets to the current PID.

The -a option is used to perform a Boolean AND of the results of the other two options to produce the following:

lsof -a -p $$ -d 0,1,2

shell-scripting-list-opened-desciptors

The name of the output file is the device name of the terminal.

Now, let’s take a look at the results of the lsof command from inside a script.

shell-scripting-list-custom-descriptors

The shell script creates three file descriptors, two for output (3 and 6) and one for input (7).

And you can see the pathname for the files used in the file descriptors.

Suppressing Command Output

Sometimes you don’t want to see any output, this often occurs if you’re running a script as a background process (we will discuss how to make your shell script run in the background in the next post).

We redirect the output to the black hole which is /dev/null.

For example, we can suppress errors like this:

ls -al badfile anotherfile 2> /dev/null

And this idea is also used when you want to truncate a file without deleting it completely.

cat /dev/null > myfile

Now you understand the input, output, how to redirect them, how to create your own file descriptor, and redirect to it. This is very important in shell scripting.

I hope you enjoy it. keep coming back.

Thank you.