How to create a Python terminal progress bar using tqdm?

Progress bars are valuable tools for estimating and displaying the amount of time that the task will take.
These can also be added to your Python scripts or code blocks to indicate the amount of time that the code execution will require.
There are various libraries available in Python such as progressbar and tqdm that can be used for this purpose.

 

 

What is tqdm?

tqdm is a Python library that provides functions that wrap around the specified iterable to give a smart progress bar as an output.
Python is a widely-used language to perform computationally intensive tasks that run over longer periods.
A tqdm progress bar gives an indicator of the progress of these tasks.

The name “tqdm” is derived from the Arabic word ‘taqadum’ meaning progress in Arabic.
The library does allow for customizable progress bars, however at its base,
the code tqdm(iterable) is enough to get you started with a smart progress meter displaying the progress of the iterable.

 

Install tqdm progress

Before you use tqdm, you need to install it using pip. You can run:

pip install tqdm

For Python3:

pip3 install tqdm

For conda environments, you can run:

conda install tqdm

Once the installation is done, you can wrap any iterable (for instance range, list, tuple, etc.) inside the function tqdm.tqdm()
to see the progress of the iteration of the iterable when using, for example, a for loop to traverse it.

from tqdm import tqdm
for i in tqdm(range(0,100)): 
    pass # do nothing

Output:

In this code, we first import the tqdm library.
Then we write a normal for loop to iterate over a range object, except that we pass the range object inside the method tqdm of the tqdm module.
As the for loop cannot be empty, and since we don’t want to do anything in the loop, we use the pass keyword.

NOTE: Ensure that you do not name the file in which you are using the tqdm library as tqdm.py.
This may result in an undesired output as your file may end up overriding the tqdm library for all the Python code that runs from the same directory.

 

Print a message with the progress bar

A good developer practice involves not only making your code self-explanatory with descriptive comments,
but also making the output of your code as descriptive and self-explanatory as possible.
As a step in that direction, we can print text along with the progress bar in the terminal to indicate the purpose of the code being executed, or to add any other desired comments.

For this, you need to set the desc parameter in the tqdm function call with your desired comment.
This text will then be printed beside the progress bar in the terminal.

from tqdm import tqdm
for i in tqdm(range(0,100),desc="This loop does nothing"): 
    pass

Output:

 

Nested Progress bars

Tqdm bars can also be used to indicate the progress of nested loops.
Multiple bars indicating the progress of each of the loops will be displayed.
In the code below, we will be using the labels ‘Outer loop’ and ‘inner loop’ to display the respective progress bars.

from tqdm import tqdm
from tqdm import trange
for i in tqdm(range(5), desc='Progress of Outer loop'):
    for j in trange((100), desc='Progress of inner loop'):
        pass

Output:

After some time, we see multiple bars, each corresponding to one inner loop,
while there is only one progress bar showing the progress of the outer loop

Note that in the code above, we have used the trange function from the tqdm library.
This function acts as a replacement for using the range object parameter with the tqdm function.
The trange function call indicates that a tqdm bar is to be displayed for the given range and all the additional features of tqdm such as desc can be used the same way.

 

tqdm with async task

To run concurrent tasks in Python, we use the asyncio package.
It has a single-thread, single process design and uses ‘cooperative multi-tasking’ to give a feel of concurrency.
While executing tasks with the asyncio library, we can use the tqdm progress bar to track its progress.
The following code example runs multiple sleep threads simultaneously, and the tqdm bar is wrapped around the tasks to display a progress bar.

import asyncio
import time
import tqdm
import tqdm.asyncio
async def sleep_duration(value = 1):
    start_time = time.time()
    await asyncio.sleep(delay=value)
    ending_time = time.time()
    intermittent_time = ending_time - starting_time
    return intermittent_time
async def run_async(values):
    taskslist = []
    for value in range(len(values)):
        i_task = asyncio.create_task(sleep_duration(value=value))
        taskslist.append(i_task)
    actual_async = [
    await x
    for x in tqdm.tqdm(asyncio.as_completed(taskslist), total=len(taskslist))
    ]
    return actual_async
n = 10
values = [3.12,3.44,4.66,2.33,1.87,4.77,5.68]
actual_async = asyncio.run(
        run_async(values=values))

Output:

The above code example may be a bit convoluted to grasp, which is normal for Python codes using async-await patterns!
run_async is used to run multiple sleep threads using the coroutine obtained with the help of the sleep_duration function which uses time module to measure the intermittent time.
tqdm.tqdm is wrapped around the asynchronous process to generate the progress bar.
The total parameter is optional and can be removed while asyncio.as_completed wrapper is used.

 

Additional tqdm parameters

The tqdm progress bar has some additional features that can be invoked with the help of a few additional parameters.
Here’s a list of some of them:

1. total – If it is not already implicitly specified (for eg. by the length of the iterable), the total number of iterations can be explicitly specified by setting the total parameter.

from tqdm import tqdm
from time import sleep
for i in tqdm(range(0, 50), total = 10,
              desc ="total demo"):
    sleep(1)

 

Output:

Here, since we specify the total number of iterations to be 10, the progress bar will be displayed only for the first 10 iterations and will disappear after that.
This is how it looks after 10 iterations are over:

Output:

2. ncols – The width of the description text (specified by the desc parameter) along with the progress bar, is determined dynamically based on the size of the window.
However, we can fix this width with the help of the ncols parameter.
In the following example, the text along with the progress bar will be displayed using just 40 columns of the output screen, even though more are available.

from tqdm import tqdm
from time import sleep
for i in tqdm(range(0, 50), ncols = 40,
               desc ="ncols demo"):
    sleep(1)

 

Output:

3. initial – The initial parameter can be set if you desire to start the progress from an arbitrary value such as 50 instead of the standard 0.

from tqdm import tqdm
from time import sleep
for i in tqdm(range(0, 20), initial = 10,
              desc ="initial demo"):
    sleep(1)

 

Output:

The above output is a snapshot of the first iteration.
The progress bar appears and shows iterations ’11-20′ for the first 10 iterations and then it disappears for the remaining iterations

4. disable – The progress bar can be disabled by setting the disable parameter to True.

from tqdm import tqdm
from time import sleep
for i in tqdm(range(0, 10), disable = True,
               desc ="disable demo"):
    sleep(1)

 

Output:

As can be seen from the output, nothing appears for the entire duration of the iteration.
Using this parameter is as good as not using the tqdm library at all!

5. mininterval – The progress of the bar is updated every 0.1 seconds by default. This behavior can be changed with the use of the mininterval parameter.

from tqdm import tqdm
from time import sleep
for i in tqdm(range(0, 50), mininterval = 5,
              desc ="min interval demo"):
    sleep(1)

 

Output for the first 5 seconds:

Output for the next 5 seconds(5th to 10th sec):

 

Integration with pandas

Pandas is a popular library that is mainly used to manipulate numerical data.
It is one of the most commonly used libraries today for handling tabular data.
Pandas uses two different data types for data representation, ‘Series’ and ‘DataFrame‘.
While Series is a one-dimensional data structure analogous to a column in an excel sheet,
DataFrame is a two-dimensional data structure with rows and columns and can also represent heterogeneous data.
As pandas is built on top of the NumPy library, it is extensively used in the field of Data Science.

In pandas, you can use the apply method to apply a function to every value of a series or every row/column of a dataframe.
We can use the tqdm progress bar with this method.
To use pandas, first install it using pip as:

pip install pandas

(For Python3, replace pip with pip3, and for conda environment, replace it with conda)

import pandas as pd
import numpy as np
from tqdm import tqdm
df = pd.DataFrame(np.random.randint(0, 100, (100, 100)))
print(df.head(10).iloc[:,:5]) #print first 10 rows and first 5 columns
tqdm.pandas(desc='pandas integration demo')
df=df.progress_apply(lambda number:number +5) #add 5 to each number
print(df.head(10).iloc[:,:5])

Output:

After importing the pandas and tqdm library, we initialize a dataframe of size 100×100 with random integers between 0 and 100.
Now we use the tqdm.pandas method to register the pandas.progress_apply function with tqdm.
Now, instead of the usual apply function, we use the progress_apply function to display a tqdm bar to denote the percentage of progress.
Both the apply and progress_apply methods accept a function as a parameter.
In our case, we pass a lambda function that takes a number and adds 5 to it, and returns it.

 

tqdm notebook

Jupyter notebooks are open-source notebooks that can run multiple programming languages such as Python, R although it is most commonly used by Python users.
It provides a live and interactive Python runtime environment.
It is a popular choice, especially among ML practitioners and data engineers, to inspect data interactively or to test step-by-step the components of an ML pipeline.

We can display tqdm loaders inside a jupyter notebook.
After setting up your jupyter notebook, while the usage of tqdm remains somewhat similar, the tqdm.notebook module is used to display the progress bar in notebooks.
Runpip3 install ipywidgets to ensure that the progress bar is displayed and dynamically updated in the notebook.
The tqdm notebook bar can contain one of the following three colors:
Green for successfully completed process.
Blue for an ongoing process.
Red for a process that has been terminated midway.

from tqdm.notebook import tqdm_notebook
import time
for i in tqdm_notebook(range(15), desc = 'Tqdm notebook’):
    time.sleep(1.5)

Output for an ongoing process:

Output for a terminated process:

Output for a successful process:

Similar to tqdm, tqdm_notebook should be wrapped around an iterable.
The notebook function can also be used with all the features of the tqdm library by using keywords such as using desc keyword to print a description of the progress bar, total to specify the total iterations, etc.

 

Alternatives to tqdm

tqdm is a powerful tool to track the progress of an operation in Python.
Along with tqdm, several other libraries can be also used for achieving the same or similar goal.
Some of these libraries are progressbar, progressbar2, and alive-progress.

progressbar is a simple library and is fairly intuitive.
The default progress bar is printed with the # symbol. We can also print the progress bar as a spinner like the loading circle shown in web browsers.
Similar to tqdm, it can be wrapped around an iterable or can be used along with a context manager.
If you want to use the print function along with the progress bar, an extremely messy output will be displayed as a new progress bar is displayed on the output screen after every print function.
To avoid this, the progressbar2 library can be used. It allows standard redirect to output, allowing for a clean progress bar along with the print function.

The last alternative mentioned here is the alive-progress library which offers some of the coolest looking progress bar options.
If animated progress bars or an interactive progress bar is a requirement for your project then alive-progress will be best suited for you.

 

Advantages of tqdm

One of the major advantages that the tqdm progress bar has over its alternatives is its speed.
The tqdm bar has an overhead of 60ns per iteration as compared to progressbar which has an overhead of 800ns per iteration.
The tqdm bar also has other advantages such as it can not only be used with jupyter notebook but also with popular Python frameworks such as Keras, Dask, etc.
Options for custom integrations with Discord, Slack, and Telegram based on project requirements are also available.

 

Conclusion

Progress bars offer multiple advantages such as :

1. Visual estimation of the progress of your task
2. An estimate of the time required to complete your task
3. An indication of an issue or error in your program.

The tqdm bar is a low overhead, smart progress bar that you can leverage in your project with the multiple customizations it offers.
Some of them were mentioned in this article and you can refer to the documentation at https://github.com/tqdm/tqdm for more detailed documentation.