Python

Downloading Files using Python (Simple Examples)

In this tutorial, you will learn how to download files from the web using different Python modules. You will download regular files, web pages, YouTube videos, Google drive files, Amazon S3, and other sources.

Also, you will learn how to overcome many challenges that you may counter such as downloading files that redirects,  downloading large files, multithreaded download, and other tactics.


 

 

Using requests

You can download files from a URL using the requests module.

Consider the code below:

Simply, get the URL using the get method of requests module and store the result into a variable named “myfile”. Then you write the contents of the variable into a file.

 

Using wget

You can also download a file from a URL by using the wget module of Python. The wget module can be installed using pip as follows:

pip install wget

Consider the following code in which we will download the logo image of Python:

In this code, the URL along with the path (where the image will be stored) is passed to the download method of the wget module.



Download file that redirects

In this section, you will learn to download from a URL which redirects to another URL with a .pdf file using requests. The URL is like the following:

https://readthedocs.org/projects/python-guide/downloads/pdf/latest/

To download this pdf file, use the following code:

In this code, the first step we specify the URL. Then we use the get method of the requests module to fetch the URL. In the get method, we set the allow_redirects to True which will allow redirection in the URL and the content after redirection will be assigned to the variable myfile.

Finally, we open a file to write the fetched content.

 

Download large file in chunks

Consider the code blew:

First, we use the get method of the requests module as we did before but this time we will set the stream attribute to True.

Then we create a file named PythonBook.pdf in the current working directory and open it for writing.

Then we specify the chunk size that we want to download at a time. We have set to 1024 bytes. Iterate through each chunk and write the chunks in the file until the chunks finished.

The Python shell will look like the following when the chunks are downloading:

Download large file in chunks

Not pretty? Don’t worry, we will show a progress bar for the downloading process later.

 

Download multiple files (Parallel/bulk download)

To download multiple files at a time, import the following modules:

We imported the os and time modules to check how much time it takes to download files. The module ThreadPool lets you run multiple threads or processes using the pool.

Let’s create a simple function which sends the response to a file in chunks:

The URLs is a two-dimensional array which specifies the path and the URL of a page you want to download.

Pass the URL to requests.get as we did in the previous section. Finally, open the file (path specified in the URL) and write the content of the page.

Now we can call this function for each URL separately and we can also call this function for all the URLs at the same time. Let’s do it for each URL separately in for loop and notice the timer:

The result will be like this:

Download multiple files

Now replace the for loop with the following line of code:

ThreadPool(9).imap_unordered(url_response, urls)

Run the script:

Download multiple files using threads

 


Download with a progress bar

The Progress bar is a UI widget of the clint module. To install the clint module, type the following command:

pip install clint

Install clint module

Consider the following code:

In this code, we imported the requests module and then from clint.textui we imported the progress widget. The only difference is in the for loop. We used the bar method of progress module while writing the content into the file. The output will be like the following:

Download a file with a progress bar

Download a webpage using urllib

In this section, we will be downloading a webpage using the urllib.

The urllib library is a standard library of Python so you do not need to install it.

The following line of code can easily download a webpage:

urllib.request.urlretrieve('url', 'path')

Specify the URL here that you want to save as and where you want to store it:

urllib.request.urlretrieve('https://www.python.org/', 'c:/users/LikeGeeks/documents/PythonOrganization.html')

In this code, we used the urlretrieve method and passed the URL of a file along with the path where we will save the file. The file extension will be .html.

 


Download via proxy

If you need to use a proxy to download your files, you can use the ProxyHandler of the urllib module. Check the following code:

In this code, we created the proxy object and opened the proxy by invoking the build_opener method of urllib and passed the proxy object. Then we made the request to retrieve the page.

Download a file via a proxy

Also, you can use the requests module as documented in the official documentation:

Simply import the requests module and create your proxy object. Then you can retrieve the file.

Download via a proxy using requests

Using urllib3

The urllib3 is an improved version of the urllib module. You can download and install it using pip:

pip install urllib3

We will fetch a web page and store it in a text file by using urllib3.

Import the following modules:

import urllib3, shutil

The shutil module is used when working with files.

Now initialize the URL string variable like this:

url = 'https://www.python.org/'

Then we use the PoolManager of urllib3 that keeps track of necessary connection pools.

c = urllib3.PoolManager()

Create a file:

filename = "test.txt"

Finally, we send a GET request to fetch the URL and open a file and write the response into that file:

 

Download from Google drive

You can use googledrivedownloader to download any file from google drive. To install it, use the following command:

pip install googledrivedownloader

Check out the code below:

Here we passed the file id of the Google drive file along with the destination path where we will save the file. Then we have the unzip parameter. If it is True, the downloaded file will be unzipped in the same destination folder.

Download files from google drive

In this example, we download the zip folder then the folder is unzipped.

 


Download file from S3 using boto3

To download files from Amazon S3, you can use the Python boto3 module.

Before getting started, you need to install the awscli module using pip:

pip install awscli

Install awscli

For AWS configuration, run the following command:

aws configure

Now enter your details as:

Download files from Amazon S3

To download a file from Amazon S3, import boto3 and botocore. Boto3 is an Amazon SDK for Python to access Amazon web services such as S3. Botocore provides the command line services to interact with Amazon web services.

Botocore comes with awscli. To install boto3 run the following:

pip install boto3

Install boto3

Now import these two modules:

import boto3, botocore

When downloading files from Amazon, we need three parameters:

  1. The name of Bucket
  2. The name of the file you need to download
  3. The name of the file after it has downloaded

Initialize the variables:

Now initialize a variable to use the resource of a session. For this, we will call the resource() method of boto3 and pass the service which is s3:

service = boto3.resource(s3)

Finally, download the file by using the download_file method and pass in the variables:

service.Bucket(bucket).download_file(file_name, downloaded_file)

Download videos from Youtube

To download videos from YouTube, you can use the pytube module.

First, install the pytube module using pip as:

pip install pytube

Install pytube

Now, import YouTube from pytube as:

from pytube import YouTube

Then pass the video URL you need to download:

YouTube('https://www.youtube.com/watch?v=himEMfYQJ1w').streams.first().download()

In this line of code, we passed the URL. Then there are streams (list of formats) that the video has. For example:

This will print all formats along with the video resolution as follows:

The first() method grabs the first format and the download() method downloads the video.

If you want to fetch information about a video, for example, the title, then use .title as follows:

 

Using asyncio

The asyncio module is focused on handling system events. It works around an event loop that waits for an event to occur and then reacts to that event. The reaction can be calling another function. This process is called even handling. The asyncio module uses coroutines for event handling.

To use the asyncio event handling and coroutine functionality, we will import the asyncio module:

import asyncio

Also, we need to install aiohttp module.

pip install aiohttp

The keyword async tells that this is a native asyncio coroutine. Inside the body of the coroutine, we have the await keyword which returns a certain value. The return keyword can also be used.

Now let’s create a code using a coroutine to download files from the web:

In this code, we created an async coroutine function that downloads our files in chunks and saves them with a random file name and returns a message.

Then we have another async coroutine calls the get_url and waits for the URLs and make a queue of all URLs.

Now to start the coroutine, we have to put the coroutine inside the event loop by using the get_event_loop() method of asyncio and finally, the event loop is executed using the run_until_complete() method of asyncio.

 

Downloading files using Python is fun. I hope you find the tutorial useful.

Thank you.

Mokhtar Ebrahim
I'm working as a Linux system administrator since 2010. I'm responsible for maintaining, securing, and troubleshooting Linux servers for multiple clients around the world. I love writing shell and Python scripts to automate my work.

6 thoughts on “Downloading Files using Python (Simple Examples)

    1. Thanks for commenting.
      Youtube-dl is awesome too! I think I’ll add it after finishing my current work.

  1. Mokhtar, appreciate your effort in taking time to compile these tutorials. Thank you for sharing your knowledge to the world. More blessings to you bro!.

    1. Thank you very much for the kind words! Appreciate it so much.
      That drives me to do my best.
      Have a great day.

  2. Dunno if my previous comment went through. Might be due to the link? Please feel free to delete this comment if the previous one is just waiting for moderation.

    Would you be willing to change your asyncio example? The one you currently have isn’t super useful. I made a comment on the /r/Python post with an example that would work better. Your current example doesn’t do anything concurrently, for example. Here’s the last part of the URL so you can see for yourself, please just prepend reddit.com to the front of it:

    /r/Python/comments/aq86xc/slug/ege6u55

    1. Thank you very much Evan! Appreciate it.
      I modified the code. Check it and tell me if there anything needs to be modified.
      Regards,

Leave a Reply

Your email address will not be published. Required fields are marked *