Iterate over files in Linux using for loop

Looping over files in Linux is a handy skill. It helps you manage and organize your files more easily. In this tutorial, we’ll learn how to use Bash to loop through files.

We’ll look at how to use for loop to iterate over regular files, files with spaces in their names, directories, ignore case sensitivity, and much more.

By the end of this tutorial, you’ll learn how to loop over different types of files and patterns.

 

 

The for file in /path/to/files/* Pattern

Command:

for file in /path/to/files/*; do
    echo "$file"
done

Output:

/path/to/files/file1.txt
/path/to/files/file2.jpg
/path/to/files/file3.doc
...

This script uses the shell globbing mechanism.

The /path/to/files/* pattern matches every file in the directory /path/to/files/.

For each file in the directory, the echo command prints the full path to the console.

The loop continues until all files have been processed. The result is a list of every file within the specified directory.

 

Looping Over Regular Files

There are various wildcard characters in Bash that can help you target specific file types or patterns. Let’s explore some of the most commonly used ones.

Using (*)

Command:

for file in /path/to/files/*.txt; do
    echo "$file"
done

Output:

/path/to/files/file1.txt
/path/to/files/file2.txt
...

The * character matches any string, allowing you to filter files based on a specific extension or naming convention.

Using (?)

Command:

for file in /path/to/files/file?.txt; do
    echo "$file"
done

Output:

/path/to/files/file1.txt
/path/to/files/file2.txt
...

The ? character matches any single character, providing a way to target files with a specific naming structure.

Using ([])

Command:

for file in /path/to/files/file[1-3].txt; do
    echo "$file"
done

Output:

/path/to/files/file1.txt
/path/to/files/file2.txt
/path/to/files/file3.txt

By using [], you can match specific characters or a range of characters, granting even more precision in file targeting.

 

The find Command with for (Recursive looping)

Linux find command allows you to search for files or directories based on various criteria.

By combining find with the for loop, you can effectively iterate over files in nested directories.
Command:

for file in $(find /path/to/files/ -type f); do
    echo "$file"
done

Output:

/path/to/files/subdir/file4.txt
/path/to/files/file1.txt
/path/to/files/file3.doc
...

The find command is used to retrieve all regular files (-type f) within /path/to/files/.

The results from find are then processed by the for loop. The echo command prints each file’s full path to the console.

The loop continues until all files found by find are processed.

 

 

Handling File Names with Special Characters

Special characters, especially spaces, can pose problems when looping through files in Bash. Let’s address this challenge.
Command:

IFS=$'\n'
for file in $(find /path/to/files/ -type f); do
    echo "$file"
done

Output:

/path/to/files/My Document.txt
/path/to/files/Program #123.log
...

By setting the Internal Field Separator (IFS) to a newline character.

We instruct the shell to treat only newlines as delimiters, ignoring spaces.

This ensures filenames with spaces are not split into multiple entries.

As a result, the for loop can iterate over filenames with spaces or other special characters without any issues.

 

Looping Over Directories

Sometimes, instead of files, you’ll need to loop over directories. Here’s how to do it:
Command:

for dir in /path/to/files/*/; do
    echo "$dir"
done

Output:

/path/to/files/directory1/
/path/to/files/directory2/
...

The trailing / after the * wildcard ensures you’re matching only directories.

Each matched directory’s path is printed to the console. The loop continues until all directories in the specified location are processed.

 

Looping Over Symbolic Links

Symbolic links (or symlinks) are essentially references to other files or directories.
Command:

for link in $(find /path/to/files/ -type l); do
    echo "$link"
done

Output:

/path/to/files/link_to_file1.txt
/path/to/files/link_to_directory2/
...

We use the find command with the -type l option to target only symbolic links.

The results from find are processed by the for loop. Each symlink’s path is printed to the console.

 

Mixed File Types

You can loop over a mixture of regular files, directories, and symbolic links. Let’s see how this can be done.
Command:

for item in /path/to/files/*; do
    if [[ -f "$item" ]]; then
        echo "File: $item"
    elif [[ -d "$item" ]]; then
        echo "Directory: $item"
    elif [[ -L "$item" ]]; then
        echo "Symlink: $item"
    fi
done

Output:

File: /path/to/files/file1.txt
Directory: /path/to/files/directory1/
Symlink: /path/to/files/link_to_file1.txt
...

The loop iterates over all items in the directory. Conditionals (if, elif) are used to determine the type of each item.

The type and path of the item are printed based on their classification.

 

Looping Over Specific File Extensions

At times, you’ll need to loop over files with specific extensions. This is often the case when working with certain types of data or file formats.
Command:

for file in /path/to/files/*.{txt,jpg,png}; do
    echo "$file"
done

Output:

/path/to/files/file1.txt
/path/to/files/image1.jpg
/path/to/files/picture1.png
...

The curly braces {} allow you to match multiple file extensions at once.

Each file matching the extensions .txt, .jpg, and .png is printed to the console.

The loop proceeds until all matching files are processed.

 

Case Sensitivity and Matching

By default, Bash is case-sensitive. However, there are scenarios where you want to loop over files regardless of their case.
Command:

shopt -s nocaseglob
for file in /path/to/files/*.[Tt][Xx][Tt]; do
    echo "$file"
done
shopt -u nocaseglob

Output:

/path/to/files/file1.TXT
/path/to/files/file2.txt
...

The shopt -s nocaseglob command is used to turn off case sensitivity for globbing.

The pattern *.[Tt][Xx][Tt] matches both lowercase and uppercase variations of .txt files.

Each matching file’s path is printed to the console. Finally, return to the default case sensitivity using shopt -u nocaseglob.

 

Ignoring Hidden Files

If you want to process files while excluding hidden ones. Here’s how to achieve this:
Command:

for file in /path/to/files/*; do
    if [[ ! $file =~ ^..* ]]; then
        echo "$file"
    fi
done

Output:

/path/to/files/file1.txt
/path/to/files/image1.jpg
...

The loop iterates over all items in the directory. The if statement checks if the file name does not start with a dot (.).

If the file isn’t hidden, its path is printed to the console.

 

Using Regular Expressions

Bash supports regular expressions, providing a powerful means for advanced pattern matching.
Command:

for file in /path/to/files/*; do
    if [[ $file =~ ^/path/to/files/file[0-9]+.txt$ ]]; then
        echo "$file"
    fi
done

Output:

/path/to/files/file1.txt
/path/to/files/file23.txt
...

The loop iterates over every item in /path/to/files/.

The if statement checks if the file matches the regex pattern ^/path/to/files/file[0-9]+\.txt$.

Files named file followed by one or more digits and ending in .txt match this pattern.