shell scripting
Linux

Shell Scripting Part4 – Input, Output, and Redirection

In the previous post, we talked about parameters and options in detail, today we will talk about something very important in shell scripting which are Input, Output, and Redirection.

You can display the output from your shell scripts in two ways:

  • Display the output on the screen.
  • Redirect the output to a file.

So how Linux handles input and output?

Standard File Descriptors

Everything is a file in Linux and that includes input and output.

Each process can have up to nine open file descriptors at a time. The first three file descriptors 0, 1, 2 are reserved for the bash shell.

0              STDIN

1              STDOUT

2              STDERR

You can handle input and output from your shell script using the above file descriptors.

You need to fully understand these three because they are like the backbones of your shell scripting. So we are going to describe every one of them in detail.

STDIN

This stands for the standard input to the shell. For a terminal, the standard input is the keyboard.

By using the input redirect symbol (<), Linux substitutes the standard input file descriptor with the file referenced. It reads the file and sends the data just as if it were typed on the keyboard. No magic.

If no file specified, some bash commands will accept input from STDIN like the cat command.

When you type the cat command in the command line without anything, it accepts input from STDIN. Any line you type, the cat command prints that line to the screen.

STDOUT

This stands for the standard output for the shell. The standard output is the screen.

Most Linux commands deliver their output to the STDOUT file descriptor which is the screen by default.

You can also append data to a file. You can do this using the >> symbol.

So if we have a file contains data, we can append other data to it using this symbol like this:

pwd >> myfile

The output generated by pwd is appended to myfile without deleting the existed content.

shell-scripting-append

The following command tries to redirect the output to a file using > symbol.

ls l xfile > myfile

shell-scripting-redirect-error

I have no file called xfile on my PC, and that generates an error, this error message appeared on the screen and this is the third type of file descriptors.

STDERR

This file descriptor is the standard error output of the shell.

When an error occurs, you see the error on the screen, this is because the STDERR file descriptor and STDOUT file descriptor referring to the same place.

So you need to redirect the errors to a log file instead of viewing it on the screen.

Redirecting Errors

As we mentioned, the STDERR file descriptor is set to the value 2. We can redirect the errors by placing the file descriptor before the redirection symbol like this:

ls -l xfile 2>myfile

cat ./myfile

shell-scripting-redirect-error-to-file

As you can see, the error now is in the file and nothing on the screen.

Redirecting Errors and Normal Output

To redirect errors and the normal output, you have to precede each with the proper file descriptor for the data you want to redirect like this:

ls l myfile xfile anotherfile 2> errorcontent 1> correctcontent

shell-scripting-redirect-error-and-data

The shell redirects the normal output of the ls command that should go to STDOUT to the correctcontent file using the 1> symbol. And error messages that should go to STDERR were redirected to the errorcontent file using the 2> symbol.

You can redirect both STDERR and STDOUT to the same file, use &> symbol like this:

ls l myfile xfile anotherfile &> content

shell-scripting-redirect-all-to-file

All error and standard output are redirected to file named content.

Redirecting Output in Scripts

There are two ways for output redirection:

  • Temporarily redirection.
  • Permanently redirection.

Temporary redirections

For temporary redirections, you can use the output redirection symbol to redirect the output and don’t forget to precede the file descriptor number with an ampersand (&) like this:

shell-scripting-temp-redirection

So if we run it, we will see both lines printed normally because as we know STDERR output to STDOUT.

You can redirect STDERR like this:

./myscript 2> myfile

shell-scripting-redirect-error-to-file

Shell scripting is Awesome! The text that was sent to STDOUT appears on the screen, while the echo statement which sends output to STDERR is redirected to the output file.

Permanent redirections

If you have much data that need to be redirected, it would be difficult to redirect every echo statement. Instead, you can redirect to a specific file descriptor for the duration of the script execution by using the exec command like this:

shell-scripting-redirect-all-to-file

If we look at the file called outfile, we will see the output of echo lines.

We redirect the STDOUT at the beginning, what about in the middle of a script like this:

shell-scripting-permenant-redirection

The exec command redirects any output going to STDERR to the file myerror, then the script uses the echo statement to display a few lines to STDOUT which is the screen.

The exec command is used in the second time to redirect STDOUT to the myfile file and finally, we redirect the error from within the echo statement to go to STDERR which in this case is myerror file.

Redirecting Input in Scripts

You can use the same technique you’ve learned to redirect the output to redirect input. The exec command enables you to redirect STDIN from a file like this:

exec 0< myfile

This command tells the shell to take the input from a file called myfile instead of STDIN and here is an example:

shell-scripting-redirect-input

Shell scripting is easy.

I showed you in the previous post how to use the read command to read data entered from the keyboard by a user. If you redirect the STDIN to a file, the read command will try to read from STDIN, and it will return data from the file instead of the keyboard.

Some Linux system administrators use this technique to read the log files for processing and we will discuss more ways to read the log on the upcoming posts in a professional way.

Creating Your Own Redirection

When you redirect input and output in your shell script, you’re not limited to the three default file descriptors. You can open up to 9 file descriptors.

The other six file descriptors from 3 through 8 are available for you to use as either input or output redirection. You can assign any of them to a file and then use them in your shell scripts.

You can assign a file descriptor for output by using the exec command and here’s an example how to do that:

shell-scripting-create-redirection

Creating Input File Descriptors

To redirect input file descriptors do the following:

1-  Save the STDIN to another file descriptor.

2- Redirecting it to a file.

3- Revert STDIN to its original location.

Look at the following code to understand these steps:

shell-scripting-create-input-file-descriptor

The STDIN is saved to file descriptor 7 and the STDIN is redirected to a file.

The STDIN reverted back to its original location after iterating over file lines.

The last read command just to make sure that STDIN is reverted back to and you can use the keyboard normally.

Close File Descriptors

The file descriptors are closed automatically when the script exits. If you want to close the file descriptor yourself, redirect the file descriptor to this symbol &- it will be closed.

shell-scripting-closing-file-desciptor

As you can see, it gives error bad file descriptor because it is no longer exist.

Note: be careful when closing file descriptors. If you open the same output file later in your shell script, the shell replaces the existing file with a new file. That means if you output any data, it will overwrite the existing file.

lsof Command

The lsof command is used to list all the opened files on the system.

On many Linux systems like Fedora, the lsof command is located under /usr/sbin.

This command is very useful, it displays information about the currently opened files on the Linux system. This includes all the running processes on background, as well as any user accounts logged into the system.

This command has a lot of options so I think I will make a special post about it later, but let’s take the important parameters that we need.

-p, allows you to specify a process ID.

-d, allows you to specify the file descriptor numbers to display.

To get the current PID of the process, you can use the special environment variable $$, which the shell sets to the current PID.

The -a option performs a Boolean AND of the results of the other two options to produce the following:

lsof -a -p $$ -d 0,1,2

shell-scripting-list-opened-desciptors

The output file name is the device name of the terminal.

Now, check the results of the lsof command from inside a script.

shell-scripting-list-custom-descriptors

The shell script creates three file descriptors, two for output (3 and 6) and one for input (7).

And you can see the pathname for the files used in the file descriptors.

Suppressing Command Output

Sometimes you don’t want to see any output. We redirect the output to the black hole which is /dev/null.

For example, we can suppress errors like this:

ls -al badfile anotherfile 2> /dev/null

And this idea is also used when you want to truncate a file without deleting it completely.

cat /dev/null > myfile

Now you understand the input, output, how to redirect them, how to create your own file descriptor, and redirect to it. This is very important in shell scripting.

I hope you enjoy it. keep coming back.

Thank you.