Text Appending in Linux Using Sed Command

Sed offers the ability to append text to lines in a file based on specific patterns or conditions.

You can use the a\ command in sedto add new lines after a pattern match.

Let’s delve into real-world examples to understand how to effectively utilize this feature.

 

 

Append a line after a matched pattern

Let’s say you have a configuration file named config.txt with the following content:

server {
    listen 80;
    server_name example.com;
}

You want to add a comment after the server_name directive to remind yourself of its purpose.

sed '/server_name/a\    # This is the domain name for our server' config.txt

Output:

server {
    listen 80;
    server_name example.com;
    # This is the domain name for our server
}

The sed command works as follows:

1. /server_name/: This searches for the pattern “server_name” in the file.
2. a\: This is the append command in sed. It tells sed to append the text that follows.
3. # This is the domain name for our server: This is the text you want to append. Note that the spaces before the comment help to maintain the formatting in the file.

If you want to save the changes to the file directly, you can use the -i flag with sed:

sed -i '/server_name/a\    # This is the domain name for our server' config.txt

After executing this command, config.txt will be modified in place with the appended text.

 

Append multiple lines after a matched pattern

Imagine you have a configuration file, config.txt, and it’s structured like this:

database {
    user db_user;
    password db_pass;
}

You wish to add a couple of comment lines after the password directive for clarity.

sed '/password/a\    # Ensure to change the password regularly.\n    # Contact the DB admin for more details.' config.txt

Output:

database {
    user db_user;
    password db_pass;
    # Ensure to change the password regularly.
    # Contact the DB admin for more details.
}

The \n represents a new line, enabling us to append another line immediately after the first.

 

Using Basic Regular Expressions for pattern matching

Basic regular expressions (BRE) help in defining search patterns that are more flexible than exact string matches.

Let’s walk through a few examples where basic regular expressions in sed can be helpful.

Appending after lines starting with a specific pattern

Suppose you have a file, data.txt, containing:

apple 5
banana 8
apple 7
cherry 3

You wish to add a comment after lines that start with “apple”.

sed '/^apple/a\# This is an apple entry' data.txt

Output:

apple 5
# This is an apple entry
banana 8
apple 7
# This is an apple entry
cherry 3

Here, ^apple is the regular expression. The caret (^) asserts that “apple” is at the start of a line.

Appending after lines ending with a specific pattern

With the same data.txt, imagine you want to append comments after lines ending with the number “8”.

sed '/8$/a\# This line ends with 8' data.txt

Output:

apple 5
banana 8
# This line ends with 8
apple 7
cherry 3

The regular expression 8$ finds lines that terminate with the number “8”. The dollar sign ($) ensures the matching character or string is at the end of a line.

Appending after lines containing digits

Again, with the data.txt file, say you wish to comment after lines that have any digit in them.

sed '/[0-9]/a\# This line has a number' data.txt

Output:

apple 5
# This line has a number
banana 8
# This line has a number
apple 7
# This line has a number
cherry 3
# This line has a number

In this case, the pattern [0-9] matches any single digit.

 

Using Extended Regular Expressions

Extended Regular Expressions (ERE) expand the capabilities of basic regular expressions by introducing additional meta-characters and constructs, allowing for more advanced pattern matching.

You can use sed with extended regular expressions by using -E flag.

Appending after lines with specific word matches

Assuming you have a file, words.txt, containing:

programming
coder
development
programmer
coding

You want to append a comment after lines containing words that end in “ing” or “ment”.

sed -E '/(ing|ment)$/a\# Relevant profession' words.txt

Output:

programming
# Relevant profession
coder
development
# Relevant profession
programmer
# Relevant profession
coding
# Relevant profession

Here, the pattern (ing|ment)$ uses the | operator which matches either “ing” or “ment” at the end of a line.

Appending after lines with repeated patterns

With the same words.txt, suppose you want to append comments after lines with repeated characters.

sed -E '/(.)\1/a\# Repeated character detected' words.txt

Output:

programming
# Repeated character detected
coder
development
programmer
# Repeated character detected
coding
# Repeated character detected

The regular expression (.)\1 identifies repeated characters.

The (.) captures any character, while \1 refers back to the captured character, ensuring a match only if it appears twice consecutively.

Appending after lines with optional matches

Suppose you want to comment on lines that contain either “code” or “coder”.

sed -E '/cod(er)?/a\# Coding related' words.txt

Output:

programming
coder
# Coding related
development
programmer
coding
# Coding related

In this pattern, cod(er)?, the ? makes the preceding “er” optional, meaning it will match both “code” and “coder”.

 

Append Based on line numbers

sed allows you to append text based on specific line numbers or a range of line numbers.

This feature is particularly useful when editing configuration files or scripts with a known structure.

Let’s dive into some examples.

Appending after a specific line number

Assume you have a file, lines.txt, that looks like this:

first line
second line
third line
fourth line

You want to append a comment after the second line.

Command:

sed '2a\# This is after the second line' lines.txt

Output:

first line
second line
# This is after the second line
third line
fourth line

In this command, the number 2 before the a\ command specifies that the append should occur after the second line.

Appending after a range of line numbers

Using the same lines.txt, let’s say you want to append comments after lines 2 through 3.

sed '2,3a\# This is a comment' lines.txt

Output:

first line
second line
# This is a comment
third line
# This is a comment
fourth line

By specifying 2,3 before the a\ command, you indicate that the append should occur after every line in the range between line 2 and line 3, inclusive.

Appending at the end of a file (Last Line)

To append text at the end of your file, you can target the last line using $.

sed '$a\# This is the end of the file' lines.txt

Output:

first line
second line
third line
fourth line
# This is the end of the file

 

Appending after lines NOT containing a pattern

Imagine you have a file, tasks.txt, which contains:

Complete the report
Attend the meeting
Send emails
Update the software

Suppose you want to append a note after lines that don’t contain the word “Update”.

sed '/Update/!a\Note: This task is regular' tasks.txt

Output:

Complete the report
Note: This task is regular
Attend the meeting
Note: This task is regular
Send emails
Note: This task is regular
Update the software

Here, the ! symbol is used to negate the match. So, the pattern /Update/! matches lines not containing the word “Update”.

 

Handling newlines (\n) in sed

By default, sed reads one line at a time from the input and strips off the terminating newline character.

Thus, within the body of a sed script, the newline character isn’t present until you introduce it.

Let’s say you have a file named sample.txt containing:

apple
banana
cherry

To insert new line after a match:

sed '/banana/a\\ndate\ngrape' sample.txt

Output:

apple
banana

date
grape
cherry

Notice the use of double backslashes \\n before “date”.

 

Append Lines with Special Characters

In sed, certain characters are deemed “special” because they carry a unique meaning within the command syntax.

Characters like &, $, ^, *, and more are all essential components of regular expressions or sed command constructs.

When you want to append lines that contain these characters, you need to handle them correctly to avoid unintentional behaviors.

Appending a line with &

Imagine a file named special.txt with the following content:

start
middle
end

Appending a line with $

If you wish to add a line containing the $ character after “start”:

sed '/start/a\$100 is the price' special.txt

Output:

start
$100 is the price
middle
end

Here, the $ in the append command is simply treated as a regular character.

Appending multiple special characters

For lines containing several special characters, simply use them as regular characters within the append command:

sed '/end/a\Special chars: & * $ ^ @' special.txt

Output:

start
$100 is the price
middle
end
Special chars: & * $ ^ @

 

Append with Shell Variables

In scripting and command-line operations, you often need to interpolate or insert values from shell variables into your sed commands.

This can be done easily by enclosing the sed command in double quotes, allowing for shell variable expansion. Here’s how to achieve this:

Basic Appending with a Variable

Consider a variable:

user="Alice"

And a file, notes.txt, that contains:

Start your day
Reply to emails

To append a custom message with the user’s name after “Reply to emails”:

sed "/Reply to emails/a\\
Hello, $user! Have a productive day!" notes.txt

Output:

Start your day
Reply to emails
Hello, Alice! Have a productive day!

Handling Special Characters in Variables

If your variable looks like this:

path="/home/alice/documents"

To append this path after “Start your day”, you’d choose a different delimiter for the sed command:

sed "/Start your day/a\\
$path" notes.txt

Output:

Start your day
/home/alice/documents
Reply to emails

While the forward slash / is commonly used as a delimiter in sed commands, you can use other delimiters, such as |, #, or : if your content might conflict with the default.

 

Append with sed captured groups

In sed, captured groups allow you to capture parts of the input pattern and reference them later.

Capturing and Appending

Let’s consider a file named products.txt with the following content:

apple 10
banana 20
cherry 30

You want to append the name of each product after its line, in uppercase.

sed '/\([a-z]*\) [0-9]*/ { h; s/\([a-z]*\) [0-9]*/\U\1/; x; G; }' products.txt

Output:

apple 10
APPLE
banana 20
BANANA
cherry 30
CHERRY

Here’s what’s happening:

• /\([a-z]*\) [0-9]*/: This matches lines that start with a lowercase word followed by a space and digits.
• h: This command copies the content of the pattern space (the current line) to the hold buffer.
• s/\([a-z]*\) [0-9]*/\U\1/: This captures the product name and transforms it to uppercase using \U.
• x: This command swaps the content of the pattern space and the hold buffer.
• G: This appends the content of the hold buffer (which now has the uppercase product name) to the pattern space, essentially appending the uppercase product name to the original line.

 

Using Multiple -e options for sequential appending

The -e option in sed allows you to specify multiple editing commands to be applied in sequence.

This feature is beneficial when you have several independent transformations or appending operations to perform on the input, and you want to execute them in a single pass over the data.

Sequential Appending of Different Lines

Consider a file named agenda.txt:

Meeting at 10am
Lunch break
Review session

You want to append reminders after both the “Meeting at 10am” and “Review session” entries:

sed -e '/Meeting at 10am/a\Bring presentation slides' -e '/Review session/a\Prepare feedback notes' agenda.txt

Output:

Meeting at 10am
Bring presentation slides
Lunch break
Review session
Prepare feedback notes

Combining Appends with Substitution

Using the same agenda.txt, suppose you want to both append a reminder after “Lunch break” and change the time of the meeting:

sed -e '/Lunch break/a\Visit the new cafe' -e 's/10am/11am/' agenda.txt

Output:

Meeting at 11am
Lunch break
Visit the new cafe
Review session

Here, the -e options allow for both appending and substitution operations in a single sed command.

Organizing Complex Operations

When dealing with complex files or multiple modifications, breaking your sed commands into multiple -e sections can make your commands more readable.

For example:

sed -e '/Meeting/a\Ensure the projector is working' \
    -e '/Lunch break/a\Confirm reservation' \
    -e '/Review session/a\Gather all team feedback' \
    agenda.txt

Output:

Meeting at 10am
Ensure the projector is working
Lunch break
Confirm reservation
Review session
Gather all team feedback

 

Efficient Ways to Append to Large Files

When dealing with very large files, appending content using tools like sed can be resource-intensive. However, there are methods to efficiently handle such operations without exhausting your system’s memory or CPU.

Use Direct File Redirection

For simple tasks like appending content to the end of a large file, direct file redirection is the most efficient:

echo "Your appended content here." >> largefile.txt

This command will append the text to the end of largefile.txt without needing to load the file into memory.

Split, Process, and Concatenate (Chunking)

If you need to append based on certain patterns, consider splitting the file into manageable chunks, processing each chunk separately, and then concatenating them:

Use split to divide the large file:

split -l 1000000 largefile.txt chunk_

This command splits largefile.txt into chunks of 1 million lines each, named chunk_aa, chunk_ab, and so on.

Process each chunk with sed or another tool:

for f in chunk_*; do
    sed 'your_sed_command' $f > ${f}_processed
done

Concatenate the processed chunks back together:

cat chunk_*_processed > largefile_processed.txt

Cleanup the chunks:

rm chunk_*

Avoid Using -i In-Place Editing for Very Large Files

The -i option for sed makes in-place edits, but it can be less efficient for very large files since it effectively creates a copy of the original file.

Instead, redirect the output to a new file and then rename it if necessary.

 

Real-world Examples

Let’s focus on examples where the sed a\ (append) command is used.

Appending Descriptive Headers to Configuration Files

Example: You have a configuration file, and for clarity, you want to append a header after a comment indicating the beginning of a new section.

File: config.conf

# Begin User Configuration
username = guest
password = guest123

Command:

sed '/# Begin User Configuration/a\# Details for the guest user' config.conf

Output:

# Begin User Configuration
# Details for the guest user
username = guest
password = guest123

Adding Comments in Scripts

Example: In a Python script, you want to append a comment after lines that contain the word “FIXME” to give a hint about fixing the issue.

File: script.py

def some_function():
    x = y + 2  # FIXME

Command:

sed '/FIXME/a\# A fix is pending for the above line' script.py

Output:

def some_function():
    x = y + 2  # FIXME
    # A fix is pending for the above line

Providing Section Separators in Configuration Files

Example: In a server configuration file, after each section, you want to append a separator for readability.

File: server.conf

[database]
user = admin

[logging]
level = INFO

Command:

sed '/\[\w\+\]/a\##########' server.conf

Output:

[database]
##########
user = admin

[logging]
##########
level = INFO

this command scans through file.txt, and every time it finds a word enclosed in square brackets (like a section header in a config file), it appends the text “##########” immediately after that line.