Linux I/O redirection

Introduction

One of the powers of the Unix command line is the use of input/output redirection and pipes. In this session, we have covered the redirection of input, output and error streams.

stdin, stdout, and stderr

The bash shell has three basic streams; it takes input from stdin (stream 0), it sends output to stdout (stream 1) and it sends error messages to stderr (stream 2) .

The keyboard often serves as stdin, whereas stdout and stderr both go to the display. This can be confusing to new Linux users because there is no obvious way to recognize stdout from stderr. Experienced users know that separating output from errors can be very useful.

The next sections will explain how to redirect these streams.

output redirection

stdout ( > )

stdout can be redirected with a greater than sign. While scanning the line, the shell will see the > sign and will clear the file.

The > notation is, in fact, the abbreviation of 1> (stdout being referred to as stream 1).

datasoft @ datasoft-linux ~/test10$ echo It is summer today!
It is summer today!
 datasoft @ datasoft-linux ~/test10$ echo It is summer today! > summer.txt
 datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
 datasoft @ datasoft-linux ~/test10$ 
 

Note that the bash shell effectively removes the redirection from the command line before argument 0 is executed. This means that in the case of this command:

datasoft @ datasoft-linux ~/test10$ echo Hei > greeting.txt

the shell only counts two arguments (echo = argument 0, hello = argument 1). The redirection is removed before the argument counting takes place.

output file is erased

While scanning the line, the shell will see the > sign and will clear the file! Since this happens before resolving argument 0, this means that even when the command fails, the file will have been cleared!

datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
 datasoft @ datasoft-linux ~/test10$ zcho It is summer today! > summer.txt
No command 'zcho' found, did you mean:
 Command 'echo' from package 'coreutils' (main)
zcho: command not found
 datasoft @ datasoft-linux ~/test10$ cat summer.txt
 datasoft @ datasoft-linux ~/test10$

noclobber

Erasing a file while using > can be prevented by setting the noclobber option.

 datasoft @ datasoft-linux ~/test10$ cat summer.txt
 datasoft @ datasoft-linux ~/test10$ set -o noclobber
 datasoft @ datasoft-linux ~/test10$ echo It is cold today! > summer.txt
bash: summer.txt: cannot overwrite existing file
 datasoft @ datasoft-linux ~/test10$ set +o noclobber
 datasoft @ datasoft-linux ~/test10$ 

overruling noclobber

The noclobber can be overruled with >|.

datasoft @ datasoft-linux ~/test10$ set -o noclobber
 datasoft @ datasoft-linux ~/test10$ echo It is summer today! > summer.txt
bash: summer.txt: cannot overwrite existing file
 datasoft @ datasoft-linux ~/test10$ echo It is summer today! >|summer.txt
 datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
 datasoft @ datasoft-linux ~/test10$

append (>>)

Use >> to append output to a file.

datasoft @ datasoft-linux ~/test10$ echo It is summer today! > summer.txt
bash: summer.txt: cannot overwrite existing file
 datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
 datasoft @ datasoft-linux ~/test10$ echo Where is the hot summer ? >> summer.txt
 datasoft @ datasoft-linux ~/test10$ cat summer.txt
It is summer today!
Where is the hot summer ?
 datasoft @ datasoft-linux ~/test10$

error redirection

2> stderr

Redirecting stderr is done with 2>. This can be very useful to prevent error messages from cluttering your screen.

The screenshot below shows redirection of stdout to a file, and stderr to /dev/null. Writing 1> is the same as >.

 datasoft @ datasoft-linux ~/test10$ find / > allfiles.txt 2> /dev/null
 datasoft @ datasoft-linux ~/test10$
 

2>&1

To redirect both stdout and stderr to the same file, use 2>&1.

datasoft @ datasoft-linux ~/test10$ find / > allfiles_and_error.txt 2>&1

Note that the order of redirections is significant. For example, the command

ls > dirlist 2>&1

directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command

directs only the standard output to file dirlist, because the standard error made a copy of the standard output before the standard output was redirected to dirlist.

output redirection and pipes

By default, you cannot grep inside stderr when using pipes on the command line, because the only stdout is passed.

datasoft @ datasoft-linux ~/test10$ rm file35 file=10 | grep test
rm: cannot remove 'file35': No such file or directory
rm: cannot remove 'file=10': No such file or directory
 datasoft @ datasoft-linux ~/test10$ rm file10 | grep.txt

With 2>&1 you can force stderr to go to stdout. This enables the next command in the pipe to act on both streams.

datasoft @ datasoft-linux ~/test10$ rm file35 file10 file101 2>&1 1>&2 | grep file35
rm: cannot remove 'file35': No such file or directory
 datasoft @ datasoft-linux ~/test10$ 

You cannot use both 1>&2 and 2>&1 to switch stdout and stderr.

datasoft @ datasoft-linux /$ rm file35 file10 file101 2>&1 1>&2 | grep file35
rm: cannot remove 'file35': No such file or directory
 datasoft @ datasoft-linux /$ echo file35 2>&1 1>&2 | sed 's/file35/FILE35/'
FILE35

You need a third stream to switch stdout and stderr after a pipe symbol.

datasoft @ datasoft-linux /$ echo file35 2>&1 1>&2 | sed 's/file35/FILE35/'
FILE35
 datasoft @ datasoft-linux /$ echo file35 3>&1 1>&2 2>&3 | sed 's/file35/FILE35/' file35
file35
sed: can't read file35: No such file or directory
 datasoft @ datasoft-linux /$

joining stdout and stderr

The &> construction will put both stdout and stderr in one stream (to a file).

datasoft @ datasoft-linux ~$ cd test10
 datasoft @ datasoft-linux ~/test10$ rm file35 &> out_and_err
 datasoft @ datasoft-linux ~/test10$ cat out_and_err
rm: cannot remove 'file35': No such file or directory
 datasoft @ datasoft-linux ~/test10$ echo file35 &> out_and_err
bash: out_and_err: cannot overwrite existing file
 datasoft @ datasoft-linux ~/test10$ 

input redirection

stdin(<)

Redirecting stdin is done with < (short for 0<).

datasoft @ datasoft-linux ~$ cat < mno.txt
What is your name?
My name is prasanta.
 datasoft @ datasoft-linux ~$ tr 'onetw' 'onezz' < mno.txt
Whaz is your name?
My name is prasanza.
 datasoft @ datasoft-linux ~$

<< here document

The here document (sometimes called here-is-document) is a way to append input until a certain sequence (usually EOF) is encountered. The EOF marker can be typed literally or can be called with Ctrl-D.

check the below code

datasoft @ datasoft-linux ~$ cat < mno.txt
> What is your name?
> My name is prasanta.
> EOF
bash: mno.txt: cannot overwrite existing file
 datasoft @ datasoft-linux ~$ cat mno.txt
What is your name?
My name is prasanta.
 datasoft @ datasoft-linux ~$ cat < mno.txt
> What's your name?
> prasanta
> ^C
 datasoft @ datasoft-linux ~$ cat mno.txt
What is your name?
My name is prasanta.
 datasoft @ datasoft-linux ~$

<<< here string

The here string can be used to directly pass strings to a command. The result is the same as using echo string | command (but you have one less process running).

See rfc 3548 for more information about base64.

 datasoft @ datasoft-linux ~$ base64 <<< linux-training.be
bGludXgtdHJhaW5pbmcuYmUK
 datasoft @ datasoft-linux ~$ base64 -d <<< bGludXgtdHJhaW5pbmcuYmUK
linux-training.be
 datasoft @ datasoft-linux ~$

confusing redirection

The shell will scan the whole line before applying redirection. The following command line is very readable and is correct.

datasoft @ datasoft-linux ~$ cat mno.txt > summer.txt 2> wrong.txt

But this one is also correct, but less readable.

 datasoft @ datasoft-linux ~$ 2> wrong.txt cat mno.txt > summer.txt

Even this will be understood perfectly by the shell.

datasoft @ datasoft-linux ~$ < mno.txt > summer.txt 2> wrong.txt

quick file clear

So what is the quickest way to clear a file ?

 datasoft @ datasoft-linux ~$ >foo

And what is the quickest way to clear a file when the noclobber option is set ?

datasoft @ datasoft-linux ~$ > |bar
bash: syntax error near unexpected token `|'

Exercise, Practice and Solution:

1. Activate the noclobber shell option.

Code:

set -o noclobber
set -C

2. Verify that noclobber is active by repeating a ls on /etc/ with redirected output to a file.

Code:

ls /etc > etc.txt
ls /etc > etc.txt (should not work)

3. When listing all shell options, which character represents the noclobber option?

Code:

echo $- (noclobber is visible as C)

4. Deactivate the noclobber option.

Code:

set +o noclobber

5. Make sure you have two shells open on the same computer. Create an empty tailing.txt file. Then type tail -f tailing.txt. Use the second shell to append a line of text to that file. Verify that the first shell displays this line.

Code:

datasoft @ datasoft-linux ~$ > tailing.txt
datasoft @ datasoft-linux ~$ tail -f tailing.txt
hello
world
in the other shell:
datasoft @ datasoft-linux ~$ echo hello >> tailing.txt
datasoft @ datasoft-linux ~$ echo world >> tailing.txt

6. Create a file that contains the names of five people. Use cat and output redirection to create the file and use a here document to end the input.

Code:

datasoft @ datasoft-linux ~$ cat > tennis.txt << ace
> Justine Henin
> Venus Williams
> Serena Williams
> Martina Hingis
> Kim Clijsters
> ace
datasoft @ datasoft-linux ~$ cat tennis.txt
Justine Henin
Venus Williams
Serena Williams
Martina Hingis
Kim Clijsters
datasoft @ datasoft-linux ~$

Previous: Linux file globbing
Next: Linux - Filters