In the previous post, we talked about parameters and options in detail, today we will talk about something very important in shell scripting which are Input, Output, and Redirection.
You can display the output from your shell scripts in two ways:
- Display the output on the screen.
- Redirect the output to a file.
So how Linux handles input and output?
Table of Contents
- 1 Standard File Descriptors
- 2 STDIN
- 3 STDOUT
- 4 STDERR
- 5 Redirecting Errors
- 6 Redirecting Errors and Normal Output
- 7 Redirecting Output in Scripts
- 8 Redirecting Input in Scripts
- 9 Creating Your Own Redirection
- 10 Creating Input File Descriptors
- 11 Close File Descriptors
- 12 lsof Command
- 13 Suppressing Command Output
Standard File Descriptors
Everything is a file in Linux and that includes input and output.
Each process can have up to nine open file descriptors at a time. The first three file descriptors 0, 1, 2 are reserved for the bash shell.
You can handle input and output from your shell script using the above file descriptors.
You need to fully understand these three because they are like the backbones of your shell scripting. So we are going to describe every one of them in detail.
This stands for the standard input to the shell. For a terminal, the standard input is the keyboard.
By using the input redirect symbol (<), Linux substitutes the standard input file descriptor with the file referenced. It reads the file and sends the data just as if it were typed on the keyboard. No magic.
If no file specified, some bash commands will accept input from STDIN like the cat command.
When you type the cat command in the command line without anything, it accepts input from STDIN. Any line you type, the cat command prints that line to the screen.
This stands for the standard output for the shell. The standard output is the screen.
Most Linux commands deliver their output to the STDOUT file descriptor which is the screen by default.
You can also append data to a file. You can do this using the >> symbol.
So if we have a file contains data, we can append other data to it using this symbol like this:
pwd >> myfile
The output generated by pwd is appended to myfile without deleting the existed content.
The following command tries to redirect the output to a file using > symbol.
ls –l xfile > myfile
I have no file called xfile on my PC, and that generates an error, this error message appeared on the screen and this is the third type of file descriptors.
This file descriptor is the standard error output of the shell.
When an error occurs, you see the error on the screen, this is because the STDERR file descriptor and STDOUT file descriptor referring to the same place.
So you need to redirect the errors to a log file instead of viewing it on the screen.
As we mentioned, the STDERR file descriptor is set to the value 2. We can redirect the errors by placing the file descriptor before the redirection symbol like this:
ls -l xfile 2>myfile
As you can see, the error now is in the file and nothing on the screen.
Redirecting Errors and Normal Output
To redirect errors and the normal output, you have to precede each with the proper file descriptor for the data you want to redirect like this:
ls –l myfile xfile anotherfile 2> errorcontent 1> correctcontent
The shell redirects the normal output of the ls command that should go to STDOUT to the correctcontent file using the 1> symbol. And error messages that should go to STDERR were redirected to the errorcontent file using the 2> symbol.
You can redirect both STDERR and STDOUT to the same file, use &> symbol like this:
ls –l myfile xfile anotherfile &> content
All error and standard output are redirected to file named content.
Redirecting Output in Scripts
There are two ways for output redirection:
- Temporarily redirection.
- Permanently redirection.
For temporary redirections, you can use the output redirection symbol to redirect the output and don’t forget to precede the file descriptor number with an ampersand (&) like this:
echo "This is an error" >&2
echo "This is normal output"
So if we run it, we will see both lines printed normally because as we know STDERR output to STDOUT.
You can redirect STDERR like this:
./myscript 2> myfile
Shell scripting is Awesome! The text that was sent to STDOUT appears on the screen, while the echo statement which sends output to STDERR is redirected to the output file.
If you have much data that need to be redirected, it would be difficult to redirect every echo statement. Instead, you can redirect to a specific file descriptor for the duration of the script execution by using the exec command like this:
echo "Redirecting all output"
echo "from a shell to a file."
echo "without redirecting every line"
If we look at the file called outfile, we will see the output of echo lines.
We redirect the STDOUT at the beginning, what about in the middle of a script like this:
echo "This is the start of the script"
echo "now redirecting all output to another location"
echo "This should go to the myfile file"
echo "and this should go to the myerror file" >&2
The exec command redirects any output going to STDERR to the file myerror, then the script uses the echo statement to display a few lines to STDOUT which is the screen.
The exec command is used in the second time to redirect STDOUT to the myfile file and finally, we redirect the error from within the echo statement to go to STDERR which in this case is myerror file.
Redirecting Input in Scripts
You can use the same technique you’ve learned to redirect the output to redirect input. The exec command enables you to redirect STDIN from a file like this:
exec 0< myfile
This command tells the shell to take the input from a file called myfile instead of STDIN and here is an example:
exec 0< testfile
while read line
echo "Line #$total: $line"
total=$(( $total + 1 ))
Shell scripting is easy.
I showed you in the previous post how to use the read command to read data entered from the keyboard by a user. If you redirect the STDIN to a file, the read command will try to read from STDIN, and it will return data from the file instead of the keyboard.
Some Linux system administrators use this technique to read the log files for processing and we will discuss more ways to read the log on the upcoming posts in a professional way.
Creating Your Own Redirection
When you redirect input and output in your shell script, you’re not limited to the three default file descriptors. You can open up to 9 file descriptors.
The other six file descriptors from 3 through 8 are available for you to use as either input or output redirection. You can assign any of them to a file and then use them in your shell scripts.
You can assign a file descriptor for output by using the exec command and here’s an example how to do that:
echo "This should display on the screen"
echo "and this should be stored in the file" >&3
echo "And this should be back on the screen"
Creating Input File Descriptors
To redirect input file descriptors do the following:
1- Save the STDIN to another file descriptor.
2- Redirecting it to a file.
3- Revert STDIN to its original location.
Look at the following code to understand these steps:
exec 0< myfile
while read line
echo "Line #$count: $line"
count=$(( $count + 1 ))
read -p "Are you done now? " answer
case $answer in
y) echo "Goodbye";;
n) echo "Sorry, this is the end.";;
The STDIN is saved to file descriptor 7 and the STDIN is redirected to a file.
The STDIN reverted back to its original location after iterating over file lines.
The last read command just to make sure that STDIN is reverted back to and you can use the keyboard normally.
Close File Descriptors
The file descriptors are closed automatically when the script exits. If you want to close the file descriptor yourself, redirect the file descriptor to this symbol &- it will be closed.
exec 3> myfile
echo "Testing ..." >&3
echo "This won't work" >&3
As you can see, it gives error bad file descriptor because it is no longer exist.
Note: be careful when closing file descriptors. If you open the same output file later in your shell script, the shell replaces the existing file with a new file. That means if you output any data, it will overwrite the existing file.
The lsof command is used to list all the opened files on the system.
On many Linux systems like Fedora, the lsof command is located under /usr/sbin.
This command is very useful, it displays information about the currently opened files on the Linux system. This includes all the running processes on background, as well as any user accounts logged into the system.
This command has a lot of options so I think I will make a special post about it later, but let’s take the important parameters that we need.
-p, allows you to specify a process ID.
-d, allows you to specify the file descriptor numbers to display.
To get the current PID of the process, you can use the special environment variable $$, which the shell sets to the current PID.
The -a option performs a Boolean AND of the results of the other two options to produce the following:
lsof -a -p $$ -d 0,1,2
The output file name is the device name of the terminal.
Now, check the results of the lsof command from inside a script.
exec 3> myfile1
exec 6> myfile2
exec 7< myfile3
lsof -a -p $$ -d 0,1,2,3,6,7
The shell script creates three file descriptors, two for output (3 and 6) and one for input (7).
And you can see the pathname for the files used in the file descriptors.
Suppressing Command Output
Sometimes you don’t want to see any output. We redirect the output to the black hole which is /dev/null.
For example, we can suppress errors like this:
ls -al badfile anotherfile 2> /dev/null
And this idea is also used when you want to truncate a file without deleting it completely.
cat /dev/null > myfile
Now you understand the input, output, how to redirect them, how to create your own file descriptor, and redirect to it. This is very important in shell scripting.
I hope you enjoy it. keep coming back.