Adding Rows to Pandas DataFrame

In this tutorial, you’ll learn how to add rows to a Pandas DataFrame. This capability becomes useful when working with datasets that evolve over time.

By the end of this tutorial, you will learn multiple methods to add rows to your DataFrame.

 

 

Add Rows to Pandas DataFrame Using loc

You can use loc property to add rows to Pandas DataFrame:

import pandas as pd
data = {
    "CustomerID": [101, 102, 103],
    "Name": ["Alice", "Bob", "Charlie"],
    "Plan": ["Basic", "Premium", "Standard"]
}
df = pd.DataFrame(data)

# Adding a new row using the loc method
df.loc[3] = [104, "David", "Premium"]

print(df)

Output:

   CustomerID      Name      Plan
0         101     Alice     Basic
1         102       Bob   Premium
2         103   Charlie  Standard
3         104     David   Premium

By specifying a new index (in this case, 3), you can directly assign data to a new row.

Handling and avoiding index overlap

It’s crucial to ensure that the index you’re trying to assign doesn’t already exist, or you’ll overwrite the existing data.

# Attempting to add a row using an existing index
df.loc[2] = [105, "Eve", "Basic"]
print(df)

Output:

   CustomerID      Name      Plan
0         101     Alice     Basic
1         102       Bob   Premium
2         105       Eve     Basic
3         104     David   Premium

Notice how “Charlie” is replaced by “Eve” because we used an existing index (2).

 

Using Concat Function

The concat function allows you to add rows by merging multiple DataFrames or series of records to add at once.

Basic row-wise Concatenation

data = {
    "CustomerID": [106, 107],
    "Name": ["Frank", "Grace"],
    "Plan": ["Standard", "Basic"]
}
new_data = pd.DataFrame(data)

# Concatenate the original df with the new_data DataFrame
result = pd.concat([df, new_data])
print(result)

Output:

   CustomerID      Name      Plan
0         101     Alice     Basic
1         102       Bob   Premium
2         105       Eve     Basic
3         104     David   Premium
0         106     Frank  Standard
1         107     Grace     Basic

Managing Index During Concatenation

As you can see, the indices in the output above. They restart from zero for the concatenated rows.

If you want to maintain a continuous index, you can do so using the reset_index method.

result_reset = result.reset_index(drop=True)
print(result_reset)

Output:

   CustomerID      Name      Plan
0         101     Alice     Basic
1         102       Bob   Premium
2         105       Eve     Basic
3         104     David   Premium
4         106     Frank  Standard
5         107     Grace     Basic

Now, the index is continuous, making your DataFrame appear more uniform.

Using the ignore_index Parameter

Alternatively, Pandas provides an ignore_index parameter within the concat function.

This results in a continuous index without needing the additional reset_index step.

result_ignore_index = pd.concat([df, new_data], ignore_index=True)
print(result_ignore_index)

Output:

   CustomerID      Name      Plan
0         101     Alice     Basic
1         102       Bob   Premium
2         105       Eve     Basic
3         104     David   Premium
4         106     Frank  Standard
5         107     Grace     Basic

 

Add DataFrame Row at a Specific Position

While Pandas doesn’t provide a direct function for this operation, you can achieve this by splitting the DataFrame and then concatenating it around the new row.

new_entry = pd.DataFrame({
    "CustomerID": [108],
    "Name": ["Hannah"],
    "Plan": ["Premium"]
})

# Split the original DataFrame to insert the new entry at the second position
first_part = result_ignore_index.iloc[:2]
second_part = result_ignore_index.iloc[2:]

# Concatenate the parts together with the new entry in between
final_df = pd.concat([first_part, new_entry, second_part], ignore_index=True)
print(final_df)

Output:

   CustomerID      Name      Plan
0         101     Alice     Basic
1         102       Bob   Premium
2         108    Hannah   Premium
3         105       Eve     Basic
4         104     David   Premium
5         106     Frank  Standard
6         107     Grace     Basic

You can insert multiple rows at any position by simply adjusting the indices of the split and providing the appropriate new data.

 

Add rows to Multi-index DataFrame

You can add a new row to a multi-index DataFrame by specifying the name of the Series as a tuple that matches the structure of the multi-index

# Sample data with multi-index
arrays = [
    ["A", "A", "B", "B"],
    [1, 2, 1, 2]
]
index = pd.MultiIndex.from_arrays(arrays, names=('Letter', 'Number'))
data = {
    "CustomerID": [121, 122, 123, 124],
    "Name": ["Tina", "Uma", "Victor", "Wendy"]
}
multi_df = pd.DataFrame(data, index=index)

# New row as a DataFrame with matching multi-index structure
new_row_df = pd.DataFrame({
    "CustomerID": [125],
    "Name": ["Xavier"]
}, index=pd.MultiIndex.from_tuples([("B", 3)], names=('Letter', 'Number')))

# Using concat to add the new row
multi_df = pd.concat([multi_df, new_row_df])
print(multi_df)

Output:

           CustomerID    Name
Letter Number                
A      1          121    Tina
       2          122     Uma
B      1          123  Victor
       2          124   Wendy
       3          125  Xavier

 

Troubleshooting Common Issues When Adding Rows

In this section, we’ll tackle some commonly encountered issues when adding rows to DataFrames.

Resolving data type mismatches

When adding rows or merging DataFrames, you might encounter issues where the data types of columns don’t match. This can lead to unexpected results or errors.

Solution:

Always ensure consistent data types across operations:

# Main DataFrame
data_warn = {
    "CustomerID": [130, 131],
    "Balance": [50, 60]
}
warn_df = pd.DataFrame(data_warn)

# Data with mismatched data types
data_mismatch = {
    "CustomerID": ["132", "133"],
    "Balance": ["70", "80"]
}
mismatch_df = pd.DataFrame(data_mismatch)

# Convert data types to align with the main DataFrame
mismatch_df = mismatch_df.astype({"CustomerID": int, "Balance": int})

# Combining DataFrames using concat
final_df = pd.concat([warn_df, mismatch_df], ignore_index=True)
print(final_df)

Output:

   CustomerID  Balance
0         130       50
1         131       60
2         132       70
3         133       80

By first converting the data types in mismatch_df to match those in warn_df and then using concat, you ensure a seamless and consistent merged DataFrame.

Handling out-of-order Column Names

When appending or merging DataFrames, the order of columns might not always match. This can lead to misalignment of data.

Solution:

Ensure that the column order matches or explicitly specify the desired column order:

data_warn = {
    "CustomerID": [130, 131],
    "Balance": [50, 60]
}
warn_df = pd.DataFrame(data_warn)

# Sample data with columns in a different order
data_order = {
    "Balance": [70, 80],
    "CustomerID": [132, 133]
}
order_df = pd.DataFrame(data_order)

# Using concat to combine the DataFrames ensuring column order
final_df = pd.concat([warn_df, order_df], axis=0, join="outer", ignore_index=True, sort=False)
print(final_df)

Output:

   CustomerID  Balance
0         130       50
1         131       60
2         132       70
3         133       80

The data is combined in the right order even if the source DataFrames have columns in different sequences.