Linux curl command: Web Interaction from the Shell
curl
is a powerful command that lets you transfer data to or from a server.
It supports a wide range of protocols, and it’s equipped to deal with various network-related tasks without requiring user interaction.
- 1 Install curl
- 2 Supported protocols
- 3 URL Globbing
- 4 Dynamic URLs using Variables
- 5 Downloading files
- 6 Uploading files
- 7 Form submissions with POST
- 8 Using other HTTP methods
- 9 Authentication with cURL
- 10 Crafting custom headers and user agents
- 11 Manipulating the Expect: header
- 12 Handling compressed responses
- 13 Following redirects
- 14 Using cookies and sessions
- 15 cURL and APIs
- 16 SSL/TLS and certificate verification
- 17 Using curl behind a proxy
- 18 Setting custom DNS servers
- 19 Reusing connections for multiple requests
- 20 Limiting download/upload speeds
- 21 Using curl for FTP transfers
- 22 Displaying response headers
- 23 IPv6 and curl
- 24 Streaming data with curl
- 25 Using .curlrc for default settings
- 26 The Magic of Curl in a Telecom Crisis
- 27 Resources
Install curl
curl
is a widely-used tool, and most Linux distributions include it in their repositories. Depending on your operating system, the installation command can vary.
Debian/Ubuntu-based Distributions
To install curl
on Debian, Ubuntu, or any other Debian-based system:
$ sudo apt update $ sudo apt install curl
Red Hat/Fedora/CentOS
For Red Hat and its derivatives:
$ sudo yum install curl
If you’re on a system using dnf
:
$ sudo dnf install curl
Supported protocols
With curl
, you can work with numerous protocols including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, LDAP, LDAPS, and many others.
$ curl --version
Output:
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
You’ll notice the protocols supported by your installed version of curl listed at the beginning of the output.
It’s crucial to know which protocols your curl
installation supports, especially if you’re dealing with many servers and services.
URL Globbing
URL globbing in curl
allows you to fetch multiple URLs with a single command by specifying parts of the URL in a brace {}
sequence.
This can save time and automate processes when dealing with sites or servers that have a predictable URL structure.
Fetching Sequential URLs
If you know that URLs are numerically sequenced, you can use URL globbing to fetch them all:
$ curl http://example.com/file[1-5].zip
This command fetches file1.zip
, file2.zip
, … up to file5.zip
.
Fetching Based on Specific Patterns
You can specify multiple patterns within the braces:
$ curl http://example.com/file_{A,B,C}.zip
This fetches file_A.zip
, file_B.zip
, and file_C.zip
.
Combining Multiple Patterns
URL globbing allows multiple pattern sequences in a single URL:
$ curl http://example.com/{year2022,year2023}/event_{A,B,C}.html
This command would fetch six URLs: from http://example.com/year2022/event_A.html
to http://example.com/year2023/event_C.html
.
Use globbing responsibly to avoid unintentional DoS attacks or being rate-limited.
Dynamic URLs using Variables
Starting from version 8.3.0, curl
introduced support for command-line variables. These variables can be set using the --variable name=content
or --variable name@file
syntax.
The latter allows you to set the variable’s content from a file, and if the file is set to a single dash (-), it reads from stdin.
To expand the content of these variables in option parameters, you can use the {{name}}
syntax, provided the option name is prefixed with --expand-
.
For instance, if you have a variable named “username”, you can use it in a URL like this: --expand-url "https://example.com/{{username}}/details"
.
Basic Variable Expansion:
Set a variable named “username” and use it in a URL.
$ curl --variable username=JohnDoe --expand-url "https://example.com/profile/{{username}}"
Using Environment Variables:
Use the USER
environment variable in a URL.
$ curl --variable %USER --expand-url "https://example.com/api/{{USER}}/details"
Using Variables from a File:
If you have a file named token.txt
containing an authentication token, you can use its content in a URL.
$ curl --variable token@token.txt --expand-url "https://api.example.com/data?auth={{token}}"
Using Multiple Variables:
Combine multiple variables in a single URL.
$ curl --variable user=JohnDoe --variable action=edit --expand-url "https://example.com/{{user}}/{{action}}"
By leveraging variables, you can make your curl
commands more dynamic and adaptable to different scenarios.
Downloading files
To download a file using curl
, you simply provide the URL of the resource:
$ curl http://example.com/file.txt -o localfile.txt
With this command, you download “file.txt” from “example.com” and save it as “localfile.txt” on your local machine. The -o
flag lets you specify the name of the saved file.
Always make sure to use the -o
flag to define the local filename, ensuring you don’t overwrite other files.
Uploading files
You can upload files to a server using the HTTP PUT or POST methods.
For PUT method:
$ curl -T localfile.txt http://example.com/upload/
Here, you upload “localfile.txt” to the specified URL using the HTTP PUT method.
For POST method:
$ curl -F "data=@localfile.txt" http://example.com/upload/
This command uploads “localfile.txt” using the HTTP POST method. -F
signals curl to POST data as a multipart/form-data.
The string “data=@localfile.txt” means to post the content of the file as the value for the “data” parameter.
When uploading, ensure you have appropriate permissions and know the method the server expects (either PUT or POST).
Form submissions with POST
Submitting forms is a common task. With curl
, you can easily simulate a form submission.
$ curl -d "username=john&password=12345" http://example.com/login
This command submits a POST request to “example.com/login” with the provided data as “application/x-www-form-urlencoded”
The -d
flag tells curl
to send data in the POST request body.
Ensure that you correctly encode data for transmission, especially if the data contains special characters.
Using other HTTP methods
curl
can utilize various HTTP methods, not just GET and POST.
$ curl -X HEAD http://example.com
This command fetches the headers from “example.com” using the HEAD method. The -X
flag specifies the HTTP method.
For DELETE method:
$ curl -X DELETE http://example.com/resource/1
This command sends a DELETE request to remove a resource on “example.com”.
Authentication with cURL
Sometimes, resources are protected and require authentication.
$ curl -u username:password http://example.com/protected
This command sends a request to “example.com/protected” with Basic Authentication using the provided username and password. The -u
flag followed by the credentials is what allows this authentication.
Always be cautious with this method. Transmitting credentials in plaintext can be a security risk, especially over unencrypted connections.
Crafting custom headers and user agents
You may need to send custom headers or change the User-Agent.
$ curl -H "X-Custom-Header: customvalue" -A "MyUserAgent/1.0" http://example.com
This sends a request to “example.com” with a custom header and a custom User-Agent.
The -H
flag sets custom headers, and the -A
flag sets the User-Agent.
Custom headers can be useful for APIs or services expecting specific information, while changing the User-Agent can help emulate different devices or browsers.
Manipulating the Expect: header
Sometimes, when sending POST requests with a large payload, curl
sends an “Expect: 100-continue” header. To explicitly control this behavior:
$ curl -H "Expect:" -d "large_data_payload" http://example.com/upload
By setting the “Expect:” header with no value, you effectively disable the default behavior, allowing immediate transmission of the data without waiting for server confirmation.
When dealing with servers that don’t handle “Expect: 100-continue” gracefully, this manipulation can be crucial.
Handling compressed responses
Many servers compress their responses to save bandwidth. curl
can automatically decompress these responses for you.
$ curl --compressed http://example.com/resource
By using the --compressed
option, if the server response includes content encoded with gzip or deflate, curl
will decompress it automatically.
This option ensures that the data you receive is immediately readable and usable.
Following redirects
Servers can redirect requests to other URLs. To make curl
follow these redirects:
$ curl -L http://example.com/redirect
Using the -L
or --location
flag instructs curl
to follow any redirects sent by the server.
It’s important to be aware when using this option, as following redirects blindly might lead you to unexpected destinations.
Cookies are essential for maintaining sessions or retaining certain preferences across requests.
$ curl -c cookies.txt -b cookies.txt http://example.com/login
Here, -c cookies.txt
tells curl
to save cookies to the file “cookies.txt”. The -b cookies.txt
instructs curl
to read and send cookies from the same file for subsequent requests.
Managing cookies properly ensures seamless interaction with sites that require session persistence.
Read more on managing cookies with Linux curl command.
cURL and APIs
APIs often require specific headers, methods, or data formats. curl
is a valuable tool for interacting with them.
$ curl -H "Authorization: Bearer YOUR_API_TOKEN" -H "Content-Type: application/json" -d '{"key":"value"}' http://api.example.com/data
This sends a POST request to the API endpoint with an authorization header, and a JSON payload.
The multiple -H
flags allow setting required headers.
When working with APIs, ensure you’re using the correct headers, methods, and data formats expected by the API.
SSL/TLS and certificate verification
By default, curl
verifies the SSL certificate of the server.
$ curl https://secured-example.com
If there’s an issue with the server’s certificate, curl
will produce an error.
However, in some scenarios, like testing or internal networks, you might encounter self-signed certificates. To bypass the verification:
$ curl -k https://internal-server
The -k
or --insecure
option disables the SSL certificate verification. Use this option with caution and only in trusted environments.
Read more about SSL/TLS with curl command.
Using curl behind a proxy
If you’re operating behind a proxy, curl
can be configured to use it:
$ curl -x http://proxyserver:port https://example.com
The -x
flag followed by the proxy details informs curl
to route its request through the specified proxy.
Setting custom DNS servers
In cases where you want curl
to resolve hostnames using a specific DNS server:
$ curl --resolve example.com:80:1.2.3.4 http://example.com
Here, curl
will use the IP address “1.2.3.4” for “example.com” on port 80. This is useful for testing or bypassing DNS-related issues.
Ensure you trust the DNS server you’re specifying to prevent potential redirect attacks.
Reusing connections for multiple requests
When hitting the same server multiple times, you can reuse connections:
$ curl -H "Connection: keep-alive" http://example.com/page1 -o page1.html $ curl -H "Connection: keep-alive" http://example.com/page2 -o page2.html
The “Connection: keep-alive” header signals to keep the TCP connection open for future requests, reducing overhead.
Keep-alive is particularly useful for scripts or automated tasks that interact repeatedly with the same server.
Limiting download/upload speeds
To limit the bandwidth used by curl
:
$ curl --limit-rate 100K -O http://example.com/largefile.zip
This limits the download rate to 100 KBytes per second. The --limit-rate
option followed by a speed (like 100K or 500K) sets the restriction.
Limiting speeds can be useful when network bandwidth is necessary.
Using curl for FTP transfers
curl
supports FTP and allows you to upload and download files.
$ curl -u ftpuser:ftppass -O ftp://ftp.example.com/file.zip
This command downloads “file.zip” from the FTP server using the provided credentials.
For uploading:
$ curl -u ftpuser:ftppass -T localfile.zip ftp://ftp.example.com/
This uploads “localfile.zip” to the FTP server.
Displaying response headers
To view the headers returned by a server:
$ curl -I http://example.com
The -I
option fetches only the headers. It’s beneficial to check metadata, cookies, or other header-related information.
This method helps you understand server responses without fetching the entire content.
IPv6 and curl
curl
fully supports IPv6. To make a request over IPv6:
$ curl http://[2001:0db8:85a3:0000:0000:8a2e:0370:7334]/
Brackets are used to encapsulate IPv6 addresses in URLs.
Ensure your network infrastructure supports IPv6 when making such requests.
Streaming data with curl
curl
can stream data, useful for watching logs or ongoing data feeds:
$ curl -N http://example.com/streaming-data
The -N
flag turns off buffering, allowing data to stream directly to your console.
When streaming, always monitor the amount of data being received to ensure it doesn’t overwhelm your system or network.
Using .curlrc for default settings
For frequently used options, you can set defaults in a .curlrc
file in your home directory:
Sample .curlrc content:
--compressed --user-agent "MyUserAgent/1.0"
With this file in place, curl
will always use the --compressed
option and set the specified User-Agent by default.
Customizing .curlrc
can save time and ensure consistency in repeated operations.
The Magic of Curl in a Telecom Crisis
I received a distress call from one of the top telecom operators.
They were experiencing a major hiccup in their system: a large chunk of their prepaid customers were unable to top up their accounts, leading to widespread dissatisfaction.
The issue? The application they used to communicate with their payment gateway was malfunctioning and they needed a quick way to validate if the issue was from their end or the gateway’s end.
The current toolchain the company had required a complex process of logging into the system, running multiple scripts, waiting for batch processes, and then obtaining logs.
To extract data for just one user, it took roughly 20 minutes. Given that they wanted to verify data for 500 users as a sample, we were looking at about 166 hours of continuous work. An unrealistic task under the circumstances.
Familiar with the company’s APIs, I proposed using the curl command to bypass the faulty application and directly communicate with the payment gateway.
By scripting a loop in bash and using curl, I could programmatically send HTTP requests and get immediate feedback on the responses.
Here’s the simplified version of what I did:
#!/bin/bash for user in $(cat user_list.txt); do curl -X POST https://gateway.example.com/topup -d "userid=$user&amount=50" >> results.txt done
After implementing the curl solution:
The entire process took roughly 30 seconds per user, a significant improvement from the initial 20 minutes. Within 4 hours, we had results for all 500 users. That’s a 97.6% reduction in time!
Analysis of the results.txt
showed that the gateway responded correctly to our requests, indicating that the issue was indeed within the company’s application.
Resources
https://curl.se/docs/manpage.html
Mokhtar is the founder of LikeGeeks.com. He is a seasoned technologist and accomplished author, with expertise in Linux system administration and Python development. Since 2010, Mokhtar has built an impressive career, transitioning from system administration to Python development in 2015. His work spans large corporations to freelance clients around the globe. Alongside his technical work, Mokhtar has authored some insightful books in his field. Known for his innovative solutions, meticulous attention to detail, and high-quality work, Mokhtar continually seeks new challenges within the dynamic field of technology.