Guide to Sed Command: Text Editing in Linux

sed stands for Stream EDitor. This command allows you to parse and transform text data.

If the input originates from a file, a pipeline, or standard input, sed processes it line by line.

With sed, you can perform various text manipulations like insertion, deletion, search, and replacement.

 

 

Basic Syntax and Operation

The basic structure for the sed command is:

sed 'COMMAND' file_name

Where COMMAND is the operation you wish to perform, and file_name is the name of the file you’re working with.

If you omit the file name, sed will expect input from standard input.

echo "Linux is fun" | sed 's/fun/great/'

Output:

Linux is great

This pipes the output of echo into sed, which then replaces “fun” with “great”.

Specifying files as input

If you want sed to process a file directly, specify the file name.

sed 's/Linux/UNIX/' sample.txt

Assuming sample.txt contains the line “Linux is Awesome”, the output would be:

UNIX is Awesome

To edit the file in-place and save the changes, use the -i option.

sed -i 's/Linux/UNIX/' sample.txt

After this command, the content of sample.txt will permanently change to “UNIX is Awesome”.

 

Specifying Line numbers

You can specify a particular line number to restrict a command’s effect.

echo -e "Apple\nBanana\nCherry" | sed '2s/Banana/Peach/'

Output:

Apple
Peach
Cherry

Only the second line, “Banana”, gets replaced with “Peach”.

 

Address Ranges

You can specify a range of lines by indicating the start and end line numbers.

echo -e "Apple\nBanana\nCherry\nDate\nElderberry" | sed '2,4s/^/Fruit: /'

Output:

Apple
Fruit: Banana
Fruit: Cherry
Fruit: Date
Elderberry

Lines 2 through 4 have been prefixed with “Fruit: “.

 

Substituting text

The s command searches for a specified pattern and replaces it with another.

The basic syntax for substitution in sed is:

sed 's/pattern/replacement/'

Command:

echo "Cats are cute." | sed 's/cute/beautiful/'

Output:

Cats are beautiful.

Here, the word “cute” is replaced with “beautiful”.

Global Substitution

By default, sed replaces only the first occurrence in each line. To replace all occurrences, append the g flag.

echo "Cats are cute. Dogs are cute too." | sed 's/cute/beautiful/g'

Output:

Cats are beautiful. Dogs are beautiful too.

Both instances of “cute” are replaced with “beautiful”.

 

Change Slash Delimiter

While most examples use the forward slash / as the delimiter, you can pick other characters, especially when dealing with file paths or URLs.

echo "Visit http://example.com" | sed 's_http://_https://_'

Output:

Visit https://example.com

This switches an HTTP URL to HTTPS, and uses underscores as delimiters to avoid clashing with slashes in the URL.

 

Delete lines

The d command in sed allows you to delete lines based on specific criteria, be it a direct match, a regular expression, or an address range.

Delete a Specific Line

To delete a specific line, provide its line number.

echo -e "Apple\nBanana\nCherry" | sed '2d'

Output:

Apple
Cherry

The second line, “Banana”, is removed from the output.

Delete a Range of Lines

You can specify a range of lines to be deleted.

echo -e "Apple\nBanana\nCherry\nDate\nElderberry" | sed '2,4d'

Output:

Apple
Elderberry

Lines 2 through 4 (inclusive) are deleted.

Delete Lines Matching a Pattern

You can delete lines matching a specific pattern using regular expressions.

echo -e "Apple\nBig Banana\nCherry\nBig Date\nElderberry" | sed '/^Big /d'

Output:

Apple
Cherry
Elderberry

All lines beginning with “Big ” are removed from the output.

Delete All Except Lines Matching a Pattern

Sometimes, instead of deleting lines that match a pattern, you’ll want to keep only those lines and delete everything else. This can be achieved using the inverse match !.

echo -e "Apple\nBig Banana\nCherry\nBig Date\nElderberry" | sed '/^Big /!d'

Output:

Big Banana
Big Date

Only lines starting with “Big ” are retained in the output.

 

Printing lines

The p command in sed prints lines from the input, and when combined with various patterns or address ranges, it becomes a powerful tool for displaying specific content.

Printing a Specific Line

To print a certain line, just indicate its line number.

echo -e "Apple\nBanana\nCherry" | sed -n '2p'

Output:

Banana

Only the second line, “Banana”, is displayed.

Printing a Range of Lines

Defining a range of lines allows you to selectively print multiple lines.

echo -e "Apple\nBanana\nCherry\nDate\nElderberry" | sed -n '2,4p'

Output:

Banana
Cherry
Date

Lines 2 through 4 are printed.

Printing Lines Matching a Pattern

With a regular expression, you can instruct sed to print lines that match a certain pattern.

echo -e "Apple\nBig Banana\nCherry\nBig Date\nElderberry" | sed -n '/^Big /p'

Output:

Big Banana
Big Date

Only lines beginning with “Big ” are printed.

Double Printing Lines Matching a Pattern

If you use the p command without the -n option, sed will print each line of the input by default. When a line matches a pattern or address, it gets printed again.

echo -e "Apple\nBig Banana\nCherry\nBig Date\nElderberry" | sed '/^Big /p'

Output:

Apple
Big Banana
Big Banana
Cherry
Big Date
Big Date
Elderberry

Lines starting with “Big ” are printed twice.

Read more on how to print lines using sed p command.

 

Grouping & Back-references

Grouping and back-references are powerful features in sed that allow you to capture portions of text and reference them in your operations.

Use the escape-prefixed parentheses \( and \) to group parts of your pattern.

echo "The apple is red" | sed 's/\(apple\)/fruit: \1/'

Output:

The fruit: apple is red

The term “apple” is captured and referred to as \1 in the replacement string.

Using Multiple Back-references

You can have multiple groups and reference them using \1, \2, etc.

echo "Swap this around" | sed 's/\(Swap\) \(this\) \(around\)/\3 \2 \1/'

Output:

around this Swap

The words “Swap”, “this”, and “around” are captured and rearranged in the replacement.

Example: Renaming Files using Back-references

Back-references can be particularly useful in scripts for tasks like renaming files.

Consider a situation where you have filenames in the format “file-001.txt” and you wish to rename them to “001-file.txt”.

Command (hypothetical usage in a script):

echo "file-001.txt" | sed 's/\(file\)-\(.*\).txt/\2-\1.txt/'

Output:

001-file.txt

Regex inside Parentheses

You can use regular expressions inside grouping parentheses to capture dynamic content and utilize it in transformations.

echo "Price: $5.99" | sed 's/Price: $\(.*\)/The cost is \1/'

Output:

The cost is 5.99

Here, we captured the actual price and used it in a different sentence structure.

Read more about sed grouping and backreferences.

 

Changing Case with y Command

The y command in sed enables you to transform characters from one set to another, which is particularly useful for changing the case of text.

Unlike the s command, y operates on individual characters rather than patterns.

Changing to Uppercase

To convert a string to uppercase, specify the range of characters to be transformed.

echo "make this uppercase" | sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/'

Output:

MAKE THIS UPPERCASE

Every lowercase letter is transformed to its uppercase counterpart.

Changing to Lowercase

Similarly, you can change text to lowercase.

echo "MAKE THIS LOWERCASE" | sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/'

Output:

make this lowercase

Each uppercase letter is swapped for its lowercase equivalent.

Mixing Case Transformations

In some situations, you may want to swap the case of each letter.

echo "Mix\ntHiS\nCaSe" | sed 'y/abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ/ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz/'

Output:

mIX ThiS cAsE

Every lowercase character is turned to uppercase, and vice versa.

Limiting the Transformation Scope

By combining the y command with addresses or patterns, you can restrict the scope of the transformation.

echo -e "Change\nTHIS\nLine" | sed '/THIS/y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/'

Output:

Change
this
Line

Only the line containing “THIS” is converted to lowercase.

 

Multi-line Operations

The commands N, P, and D are used for multi-line processing, they allow you to read, modify, and print data across multiple lines.

Appending the Next Line with N

The N command appends the next line of input into the pattern.

echo -e "apple\nbanana\ncherry\ndate" | sed 'N;s/\n/\&/'

Output:

apple&banana
cherry&date

Here’s how it works:

N: This reads in the next line and appends it to the pattern space, separated by a newline character.

s/\n/\&/: This replaces the newline character with the & character.

Printing up to the First Newline with P

The P command prints the content of the pattern up to the first new line.

echo -e "apple\nbanana" | sed 'N;P'

Output:

apple
apple
banana

First, the N command appends “banana” to the pattern. Then P prints only “apple”, and the entire pattern is printed afterward.

Deleting up to the First Newline with D

The D command deletes the content of the pattern up to the first new line.

echo -e "apple\nbanana\ncherry" | sed 'N;D'

Output:

banana
cherry

After appending “banana” with N, D deletes “apple”, then “banana” and “cherry” get printed.

Patterns Across Multiple Lines

Multi-line operations shine in situations where patterns span multiple lines.

echo -e "apple split\nbanana" | sed 'N;/split\nbanana/s//split &/'

Output:

apple split & banana

This identifies a pattern across two lines and replaces the new line with ” & “.

Example: Removing Consecutive Blank Lines

As an example, you can remove consecutive empty lines, leaving only one empty line.

echo -e "apple\n\n\nbanana" | sed '/^$/N;/\n$/D'

Output:

apple

banana

The consecutive blank lines between “apple” and “banana” are reduced to a single blank line.

 

Basic vs. Extended Regular Expressions

In sed, there are two types of regular expressions: basic (BRE) and extended (ERE).

• Basic Regular Expressions (BRE): This is the default mode in sed. Some characters like \+, \?, and \| have literal meanings, and you need to escape them (+, ?, and |) to use their special functions.
• Extended Regular Expressions (ERE): Activated with the -E option in sed, it supports a richer set of operations. In this mode, you don’t need to escape the special characters.

Commonly Used Regex Patterns

• .: Matches any single character.
• ^: Matches the beginning of a line.
• *: Matches zero or more of the preceding character or group.
• \+: Matches one or more of the preceding character or group (BRE).
• +: Matches one or more of the preceding character or group (ERE).
• [...]: Matches any one of the characters inside the square brackets.

Practical regex use cases in sed

Example1: Matching Lines Starting with a Specific Word

echo -e "apple\nbanana\napple pie" | sed -n '/^apple/p'

Output:

apple
apple pie

Example2: Removing Trailing Whitespaces

echo -e "apple   \nbanana" | sed 's/[ \t]*$//'

Output:

apple
banana

Example3: Matching Email Addresses (simplified)

Command (Using ERE for simplicity):

echo "Contact me at contact@example.com." | sed -E 's/([a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4})/[\1]/'

Output:

Contact me at [contact@example.com].

Example4: Swapping Words Using Capturing Groups

echo "first to second" | sed 's/\(first\) to \(second\)/\2 and \1/'

Output:

second and first

The positions of “first” and “second” are switched using capturing groups.

 

Multiple sed Commands

Multiple commands can be chained together using a semicolon (;).

echo "apple banana" | sed 's/apple/fruit/; s/banana/berry/'

Output:

fruit berry

Both substitutions are performed sequentially on the same input.

Using Multiple -e Options

You can specify multiple commands using the -e option before each command.

echo "apple banana" | sed -e 's/apple/fruit/' -e 's/banana/berry/'

Output:

fruit berry

Each -e introduces a new sed command to apply to the input.

The sed Script File

When dealing with numerous commands, it’s cleaner to use a sed script file.

Save the following content to a file named script.sed.

s/apple/fruit/
s/banana/berry/

Then use -f option to use the script with sed:

echo "apple banana" | sed -f script.sed

Output:

fruit berry

 

Using Labels

For longer sed scripts with multiple branching points, labels help to navigate the control flow. You define a label using :labelname and reference it in b or t commands.

Example: Repeated Substitution Until a Condition

Let’s replace all occurrences of “aa” with “a” iteratively until no more “aa” sequences remain:

Script:

:loop
s/aa/a/
t loop

Command:

echo "baaaanaaa" | sed -f script.sed

Output:

banana

Initially, “baaaanaaa” has three occurrences of “aa”. The script continues to replace “aa” with “a” until no more matches are found.

 

The b (Branch) Command

The b command branches to a label. If no label is provided, it branches to the end of the script, skipping any commands that follow.

Example: Skipping Lines Containing Specific Text

echo -e "apple\nbanana\ncherry" | sed '/apple/b; s/.*/& is a fruit/'

Output:

apple
banana is a fruit
cherry is a fruit

For lines containing “apple”, the b command skips the substitution that follows, thus not appending “is a fruit”.

Read more about Linux sed branching.

 

The t (Test) Command

The t command tests if any substitution has taken place since the last input line was read or conditional branch was taken.

If a substitution occurred, it branches to the label. If no label is provided, it branches to the end of the script.

Example: Capitalizing a Word Once Per Line

echo -e "apple apple\nbanana" | sed 's/apple/APPLE/; t; s/apple/APPLE/g'

Output:

APPLE apple
banana

If the first substitution succeeds, the t command skips the second substitution. This ensures that only the first occurrence of “apple” gets capitalized.

 

Testing and Negation

In sed, the ! command serves as a negation, meaning it applies the following command or commands to all lines except those matching a particular address or pattern.

This is useful when you want to transform lines that don’t match a certain pattern, or when you want to skip modifications on matching lines.

Negating a Single Pattern

If you want to apply a command to lines that don’t match a specific pattern, use ! before the command.

Example: Capitalize All Lines Not Containing a Specific Text

echo -e "apple\nbanana\napple pie" | sed '/apple/!s/.*/\U&/'

Output:

apple
BANANA
apple pie

Here, only the “banana” line, which doesn’t contain “apple”, gets capitalized.

Negating Multiple Patterns with a Range

You can combine ! with line ranges to apply commands outside of those ranges.

Example: Delete All Lines Except Lines 2 to 3

echo -e "apple\nbanana\ncherry\ngrape" | sed '2,3!d'

Output:

banana
cherry

Lines 2 to 3 (i.e., “banana” and “cherry”) are the only ones retained.

Combining Negation with Substitution

Negation can be paired with other sed commands.

Example: Change Text, but not on lines containing a specific word

echo -e "apple\napple pie\napple tart" | sed '/pie/!s/apple/fruit/'

Output:

fruit
apple pie
fruit tart

The substitution takes place everywhere except on the line with “pie”.

Applying Multiple Commands to Negated Patterns

Using curly braces {}, you can group multiple commands together to execute them on negated patterns.

Example: For lines not starting with “#”, replace words and append a specific text

echo -e "#comment\napple" | sed '/^#/!{s/apple/fruit/; s/$/- done/}'

Output:

#comment
fruit- done

Commands inside {} are executed only on lines not starting with “#”.

 

Inserting Text

The i command inserts the given text immediately before the line that matches the specified pattern.

Example: Insert Text Before Any Line Containing a Specific Word

echo -e "apple pie\nbanana split" | sed '/apple/ifruit:'

Output:

fruit:
apple pie
banana split

Inserting Multiple Lines

To insert multiple lines, you can use a backslash at the end of each line you’re inserting, except the last.

Example: Insert Multiple Lines Before Specific Text

echo "banana split" | sed '/banana/i\First line to insert\nSecond line to insert'

Output:

First line to insert
Second line to insert
banana split

Insertion with Line Numbers

Instead of patterns, you can use line numbers to determine where the text should be inserted.

Example: Insert Text Before the First Line

echo -e "apple\nbanana\ncherry" | sed '1i\Top of the list:'

Output:

Top of the list:
apple
banana
cherry

Insert Command with Other Commands

sed commands can be chained together to perform a series of operations. When paired with the i command, this allows for complex manipulations.

Example: Insert Text and Delete Lines Containing Specific Text

echo -e "apple\ncherry\nbanana" | sed '/apple/i\fruit: ; /cherry/d'

Output:

fruit:
apple
banana

 

Appending Text

The a command appends the provided text right after the line that matches the specified pattern.

Example: Append Text After Any Line Containing a Specific Text

echo -e "apple pie\nbanana split" | sed '/pie/a\Delicious!'

Output:

apple pie
Delicious!
banana split

Appending Multiple Lines

To append multiple lines, you can again use a backslash at the end of each line you’re appending, except for the last one.

Example: Append Multiple Lines After a Specific Text

echo "banana split" | sed '/banana/a\More fruits:\nAnother dessert'

Output:

banana split
More fruits:
Another dessert

Appending Using Line Numbers

You can also use line numbers to determine where the text should be appended.

Example: Append Text After the Last Line

echo -e "apple\nbanana" | sed '$a\End of the list.'

Output:

apple
banana
End of the list.

Append command with Other Commands

The a command can be used in conjunction with other sed commands.

Example: Append Text and Replace Other Text

echo -e "apple\nbanana" | sed '/apple/\afruit: ; s/banana/berry/'

Output:

apple
fruit:
berry

Read more about text appending in Linux using sed command.

 

Reading Content from a File

With the r command, you can read the content of a specified file and append it after the line matching a given pattern.

Example: Append the Content of a file After Any Line Containing a Specific Text

Assuming extra.txt contains:

Extra toppings:
- Chocolate
- Sprinkles

Command:

echo "apple pie" | sed '/pie/r extra.txt'

Output:

apple pie
Extra toppings:
- Chocolate
- Sprinkles

Reading Content Using Line Numbers

You can also use line numbers to decide where to insert the external file’s content.

Example: Insert Content of a File After the Last Line

Assuming footer.txt contains:

End of menu
Visit again!

Command:

echo -e "apple\nbanana" | sed '$r footer.txt'

Output:

apple
banana
End of menu
Visit again!

Inserting Multiple Files

You can insert content from several files into your data stream by chaining multiple r commands.

Example: Inserting a Header and a Footer

Assuming header.txt contains:

Fruit Menu
----------

And using the earlier footer.txt:

echo -e "applenbanana" | sed '1r header.txt ; $r footer.txt'

Output:

Fruit Menu
----------
apple
banana
End of menu
Visit again!

 

Writing to a File

Using the w command, you can write lines matching a specific pattern to a specific file.

Example: Write Lines Containing a Specific Text to a File

echo -e "apple pie\nbanana split\napple tart" | sed -n '/apple/w apples.txt'

The lines containing “apple” are written to apples.txt. The -n option ensures that sed doesn’t print anything to standard output.

Writing Using Line Numbers

You can also use line numbers to specify which lines should be written to the external file.

Example: Write the First Line to a File

echo -e "apple\nbanana\ncherry" | sed -n '1w firstline.txt'

The first line “apple” is written to firstline.txt.

Combining Reading and Writing

The r and w commands can be combined for powerful transformations where you read from one file, perform transformations, and write to another.

Example: Replace Text in a File and Save to Another File

Assuming source.txt contains:

apple pie
banana split

Command:

sed '/apple/s/apple/fruit/w output.txt' source.txt

The modified lines with “fruit” replace “apple” and are written to output.txt.

Writing Multiple Patterns to Different Files

You can direct different patterns to different files by chaining w commands.

echo -e "apple\npie\nbanana\nice-cream" | sed -n '/apple/banana/w fruits.txt ; /pie/ice-cream/w desserts.txt'

Lines containing “apple” and “banana” are saved to fruits.txt, while “pie” and “ice-cream” are stored in desserts.txt.

 

Working with the Hold Buffer

The hold buffer is like a temporary clipboard where you can copy, swap, and replace patterns from the pattern space (the current line sed is working on). Several commands interact with the hold buffer:

• h: Copies the pattern space to the hold buffer.
• H: Appends the pattern space to the hold buffer with a new line.
• g: Copies the hold buffer to the pattern space.
• G: Appends the hold buffer to the pattern space with a new line.
• x: Exchanges the contents of the pattern space and the hold buffer.

Example: Storing and Appending Data with Hold Buffer

echo -e "apple\nbanana" | sed -n '1h ; 2G ; p'

Output:

apple
banana
apple

Here’s the step-by-step breakdown:

• The first line “apple” is copied to the hold buffer.
• The hold buffer is then appended to the second line “banana”, resulting in “banana\napple” in the pattern space.
• This modified pattern space is then printed out.

 

Resource

https://www.gnu.org/software/sed/manual/sed.html