RESTful API Interactions Using Curl Command

One of the primary uses of curl is interacting with RESTful APIs, which are the backbone of many modern web applications and services.

In this tutorial, you’ll learn how to interact with RESTful APIs using curlcommand, fetching data, posting information, and parsing various responses.

 

 

GET Request

When you need to fetch data from an API endpoint, the GET request is your go-to method.

It’s the simplest type of request, querying an API for information.

curl https://api.example.com/users/123

Output:

{
  "id": 123,
  "name": "John Doe",
  "email": "john@example.com"
}

The response received is a JSON formatted data, presenting details of the user with ID 123.

This data can be processed by other tools or scripts, or simply reviewed manually.

Most modern RESTful APIs will provide responses in JSON format.

When you want to fetch data with specific parameters, you can append them directly to the URL.

curl https://api.example.com/users?role=admin

Output:

[
  {
    "id": 101,
    "name": "Alice",
    "role": "admin"
  },
  {
    "id": 102,
    "name": "Bob",
    "role": "admin"
  }
]

In this example, we’re querying the API for users with the role of “admin”. The response lists all users fitting this criteria.

 

POST Request

When it comes to creating new resources or submitting data to an API, the POST request is your primary method.

POST requests allow you to send data to the server, often resulting in the creation of a new record or resource.

curl -X POST -H "Content-Type: application/json" -d '{"name":"Jane Doe", "email":"jane@example.com"}' https://api.example.com/users

Output:

{
  "id": 124,
  "name": "Jane Doe",
  "email": "jane@example.com",
  "status": "created"
}

Breaking down the command:

• -X POST indicates the type of request you’re making, which is POST.
• -H "Content-Type: application/json" sets the header for the request, informing the server that you’re sending JSON formatted data.
• -d is followed by the data payload you’re sending, enclosed in single quotes.

The server’s response confirms that the user “Jane Doe” was successfully created, assigning her an ID of 124 and providing a status of “created”.

curl -X POST -F "username=jane_doe" -F "password=secret123" https://api.example.com/login

Output:

{
  "token": "abcd1234",
  "status": "authenticated"
}

This command demonstrates another way of sending data using POST, especially when dealing with forms.

The -F flag denotes form data. Here, you’re trying to log in, sending a username and password.

After successful authentication, the server returns an authentication token which will be used for subsequent requests that require authentication.

 

PUT and PATCH Requests

Updating existing resources in an API often involves the use of PUT or PATCH requests. While both are used for updates, they serve slightly different purposes:

• PUT: Typically used when you’re providing a completely updated version of a resource.
• PATCH: Used when you’re updating only specific parts of a resource.

Let’s dive into both with practical examples.

Command for PUT

curl -X PUT -H "Content-Type: application/json" -d '{"name":"Jane Smith", "email":"jane.smith@example.com"}' https://api.example.com/users/124

Output:

{
  "id": 124,
  "name": "Jane Smith",
  "email": "jane.smith@example.com",
  "status": "updated"
}

In this PUT example:

• -X PUT specifies the type of request.
• You’re again setting the content type to JSON.
• The data you’re sending is intended to completely replace the user with ID 124’s existing data.
• The response confirms the user details have been updated.

Command for PATCH

curl -X PATCH -H "Content-Type: application/json" -d '{"email":"jane.doe@example.com"}' https://api.example.com/users/124

Output:

{
  "id": 124,
  "name": "Jane Smith",
  "email": "jane.doe@example.com",
  "status": "partially updated"
}

In this PATCH example:

• You’re only sending a new email address for user with ID 124, leaving other fields unchanged.
• The server acknowledges the partial update by returning “partially updated” as the status.

 

DELETE Request

You can use the DELETE request when you need to remove a resource or record from an API.

curl -X DELETE https://api.example.com/users/124

Output:

{
  "status": "deleted",
  "message": "User with ID 124 has been removed."
}

In this example:

• -X DELETE indicates the type of request you’re initiating.
• The URL points to the specific user resource you want to delete, identified by its ID, which in this instance is 124.

The server’s response confirms the successful deletion of the user. Often, APIs will return a status message to provide clarity about the result of the operation.

Note that not all APIs will return detailed data when performing a DELETE operation.

 

Fetching and Parsing JSON

The jq tool allows you to parse JSON from the command line, it works seamlessly with curl to fetch and parse JSON data.

curl https://api.example.com/users/123

Output:

{
  "id": 123,
  "name": "John Doe",
  "email": "john@example.com"
}

Now, imagine you only want to extract the user’s name from this JSON response.

curl -s https://api.example.com/users/123 | jq '.name'

Output:

"John Doe"

Here’s a breakdown:

• The -s flag ensures curl operates in “silent” mode, reducing unnecessary output.
• | pipes the output of the curl command into the jq command.
• jq '.name' instructs jq to parse the JSON and extract the value associated with the “name” key.

jq is extremely powerful and can perform complex queries, filtering, and manipulations on JSON data.

Read more on JSON Manipulation Using Linux jq Command.

 

Fetching and Parsing XML

Just like with JSON, you can fetch XML data using curl and then parse it with suitable tools, such as xmlstarlet.

To use xmlstarlet with curl, ensure both tools are installed on your system. xmlstarlet can usually be installed from popular package managers like aptor yum.

curl https://api.example.com/users/123.xml

Output:

<user>
  <id>123</id>
  <name>John Doe</name>
  <email>john@example.com</email>
</user>

If you want to extract the user’s name from this XML response.

curl -s https://api.example.com/users/123.xml | xmlstarlet sel -t -v "/user/name"

Output:

John Doe

Here’s a breakdown of the command:

• As before, the -s flag keeps curl in “silent” mode.
• | pipes the XML output of the curl command into the xmlstarlet command.
• The sel command in xmlstarlet stands for “select”, and it’s used for querying the XML data.
• -t specifies a template pattern.
• -v "/user/name" extracts the value associated with the XML path /user/name.

 

Using Pagination Headers and Query Parameters

Most APIs use query parameters like page, limit, or offset to control pagination.

curl https://api.example.com/users?page=2&limit=50

Output:

[
    // Array of 50 users starting from the 51st user
]

By adding ?page=2&limit=50 to the URL, you’re requesting the second page of users with a limit of 50 users per page.

Reading pagination info from headers

Some APIs provide pagination details in the response headers rather than the body.

curl -I https://api.example.com/users?page=2

Output:

HTTP/1.1 200 OK
Date: Mon, 23 Aug 2023 09:00:00 GMT
Link: <https://api.example.com/users?page=1>; rel="prev", <https://api.example.com/users?page=3>; rel="next"

• The -I flag tells curl to fetch only headers of the response.
• The Link header provides URLs for both previous and next pages. By examining these links, you can navigate through the paginated results.

Using both headers and query parameters

You can combine both techniques:

curl -I -G -d "page=2" -d "limit=50" https://api.example.com/users

Output:

HTTP/1.1 200 OK
Date: Mon, 23 Aug 2023 09:00:00 GMT
Link: <https://api.example.com/users?page=1&limit=50>; rel="prev", <https://api.example.com/users?page=3&limit=50>; rel="next"

• The -G flag tells curl to send data specified with -d as a GET request query string.
• Multiple -d flags help build the complete query string.

 

Benchmarking API endpoints

One of the simplest metrics is the total time taken for a request. You can use curl‘s -w (or --write-out) option to print this information.

curl -s -o /dev/null -w "Total time: %{time_total}\n" https://api.example.com/users/1

Output:

Total time: 0.215

Here’s what’s happening in the command:

• -s runs curl in silent mode.
• -o /dev/null discards the actual response body, ensuring only our custom output is visible.
• -w "Total time: %{time_total}\n" prints the total time taken for the request.

Comparing Endpoint Speeds

To compare the responsiveness of different endpoints, you can use a simple loop in shell scripting:

endpoints=("users/1" "users/2" "posts/1")
for endpoint in "${endpoints[@]}"; do
    curl -s -o /dev/null -w "${endpoint} took %{time_total} seconds\n" https://api.example.com/$endpoint
done

Output:

users/1 took 0.203 seconds
users/2 took 0.187 seconds
posts/1 took 0.220 seconds

This script sends requests to various endpoints and prints out how long each one took, giving a direct comparison of their speeds.

For detailed performance analytics, you can use a tool like Apache Benchmark (ab) which offers more extensive benchmarking capabilities than curl.

 

Optimization Using Compression

By using compressed data, you can reduce the amount of data transferred between the client and server, leading to faster response times.

curl -H "Accept-Encoding: gzip" -s -o /dev/null -w "Downloaded size: %{size_download}\n" https://api.example.com/large-data

By sending the Accept-Encoding: gzip header, you’re indicating to the server that your client can handle gzip-compressed responses.

If the server supports compression, it will send the data in a compressed format.

 

Real-world Example

Let’s take GitHub’s RESTful API as an example. This walkthrough will demonstrate how to:

• Authenticate with the GitHub API.
• Fetch the details of the authenticated user.
• List repositories of the authenticated user.
• Create a new repository.

Authenticate with the GitHub API

Before making calls to the GitHub API, you need a personal access token, which you can generate from the GitHub settings.

TOKEN="YOUR_GITHUB_TOKEN"

Replace YOUR_GITHUB_TOKEN with your generated personal access token.

Fetch the Details of the Authenticated User

With your token in place, fetch your own user details.

curl -s -H "Authorization: token $TOKEN" https://api.github.com/user

List Repositories of the Authenticated User

Retrieve a list of repositories associated with your account.

curl -s -H "Authorization: token $TOKEN" https://api.github.com/user/repos

Create a New Repository

To create a new repository, use a POST request with a JSON payload specifying the repository name and other attributes.

curl -s -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" -d '{"name":"my-new-repo"}' https://api.github.com/user/repos

This command creates a new repository named “my-new-repo”.