Curl Cheatsheet

Selection of helpful curl commands
Dec 1, 2019 Jul 12, 2021 / Posts / 7 min
Table of Contents

Introduction

Curl is an invaluable tool for anyone who needs to debug webapps fequently. Given it’s vast amount of options, it is easy to miss some of it’s simplest yet powerful uses.

Listing here some options that I use everyday. Use the topics below to navigate direct to what you are looking for.

I’ll keep updating it as and when I discover something new! 🤘

Inspect

Response headers

The simplest way to do that is by using the -I option -

1$ curl -I https://httpbin.org/ip
2
3HTTP/2 200 
4date: Thu, 12 Nov 2019 19:08:13 GMT
5content-type: application/json
6content-length: 32
7server: gunicorn/19.9.0
8access-control-allow-origin: *
9access-control-allow-credentials: true

The above works in most cases, but if your webapp/endpoint doesn’t support the HEAD method or behaves differently to it compared to the usual GET method, this will dissappoint you.

To inspect the headers with a GETmethod, following options can be helpful.

1$ curl -D - https://httpbin.org/ip -s -o /dev/null
2
3HTTP/2 200 
4date: Thu, 12 Nov 2019 19:10:39 GMT
5content-type: application/json
6content-length: 32
7server: gunicorn/19.9.0
8access-control-allow-origin: *
9access-control-allow-credentials: true

OK, this works 👍 but there are a lot of options. What do they tell curl?

  • -D dump the headers to a file.
  • - dump it to stdin instead of a file.
  • -s suppress the progress update (like the following), helps to keep the output clean.
    1% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2                               Dload  Upload   Total   Spent    Left  Speed
30     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--   0HTTP/2 200
  • -o send any output to file
  • -o /dev/nulldiscard the response completely.

Request headers

I find -v option to be the most useful to inspect the request headers. Check the request headers on the lines starting with >

 1$ curl -v https://httpbin.org/ip -s -o /dev/null | grep ">"
 2
 3*   Trying 3.211.1.78:443...
 4* Connected to httpbin.org (3.211.1.78) port 443 (#0)
 5* Using HTTP2, server supports multi-use
 6* Connection state changed (HTTP/2 confirmed)
 7* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
 8} [5 bytes data]
 9* Using Stream ID: 1 (easy handle 0x1da3b7de150)
10} [5 bytes data]
11> GET /ip HTTP/2
12> Host: httpbin.org
13> user-agent: curl/7.73.0
14> accept: */*

Redirects

Redirects are implemented for a variety of reasons. Redirect responses return a 30X response code and a location header which tells the requestor where to go next. The following request gives us these filtering out the rest.

1$ curl https://httpbin.org/status/302  -w "%{response_code} %{redirect_url}" -s -o /dev/null
2302 https://httpbin.org/redirect/1

OK, this is nice 👌 but what does w tell curl?

  • -w is write-out option and takes a number of variables. These variables need to be enclosed in a %{ } block. I have used 2 here. response_code and redirect_url which are sort of self explanatory. Full list of variables it supports is here

What if you want to inspect the redirects for multiple pages? Add the urls in a list and run this one liner loop to check all the redirects in one shot, and also format the output a bit while we are at it.

https://httpbin.org/status/301
https://httpbin.org/status/302
https://httpbin.org/status/307

1$ while read url; do curl $url -w " %{response_code} | $url --> %{redirect_url} \n" -s -o /dev/null; done < urls.txt
2
3301 | https://httpbin.org/status/301 --> https://httpbin.org/redirect/1 
4302 | https://httpbin.org/status/302 --> https://httpbin.org/redirect/1 
5307 | https://httpbin.org/status/307 --> https://httpbin.org/redirect/1

Customize Requests

Change the User-Agent

By default, curl uses a user-agent request header like, user-agent: curl/7.73.0

If you need to send a custom user-agent, simply use the -A option. Check the User-Agent key value in the returned json response.

 1$ curl https://httpbin.org/headers -A "bob-the-builder"
 2
 3{
 4  "headers": {
 5    "Accept": "*/*",
 6    "Host": "httpbin.org",
 7    "User-Agent": "bob-the-builder",
 8    "X-Amzn-Trace-Id": "Root=1-5fadb05b-490211b54febef30592912b5"
 9  }
10}

Custom headers

To send custom headers with a request, you can use the -H option. Check the headers section in the returned json response.

 1$ curl https://httpbin.org/headers -H "CustomHeader1: value1" -H "CustomHeader2: value2"
 2
 3{
 4  "headers": {
 5    "Accept": "*/*",
 6    "Customheader1": "value1",
 7    "Customheader2": "value2",
 8    "Host": "httpbin.org",
 9    "User-Agent": "curl/7.73.0",
10    "X-Amzn-Trace-Id": "Root=1-5fadb190-26af70e1691093c76d180401"
11  }
12}

Cookies

To send cookies with a request, you can use the -b option. The cookies can either be provided inline or read from a file. Check the cookies in the json response.

Send the cookie values inline

1$ curl https://httpbin.org/cookies -b "country=GB; language=EN; debug=1"
2
3{
4  "cookies": {
5    "country": "GB",
6    "debug": "1",
7    "language": "EN"
8  }
9}

Send the cookie values from a file

 1$ curl https://httpbin.org/cookies -b cookies.txt
 2
 3{
 4  "cookies": {
 5    "country": "GB",
 6    "debug": "2",
 7    "language": "EN",
 8    "source": "file"
 9  }
10}

Note The cookie.txt file needs to be in the following format for curl to read and parse the cookies

httpbin.org	FALSE	/	FALSE	0	country GB
httpbin.org	FALSE	/	FALSE	0	language    EN
httpbin.org	FALSE	/	FALSE	0	source  file
httpbin.org	FALSE	/	FALSE	0	debug   2

Reuse cookies

To save cookies received in a response, you can use the -c option. The cookies can then be easily used in subsequent requests using the -b option as we saw above.

1$ curl https://httpbin.org/cookies/set/locale/en_GB -c cookies.txt
2
3<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
4<title>Redirecting...</title>
5<h1>Redirecting...</h1>
6<p>You should be redirected automatically to target URL: <a href="/cookies">/cookies</a>.  If not click the link.

Let’s inspect the file, if cookie is saved. The contents of the file would be as follows -

# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

httpbin.org     FALSE   /       FALSE   0       locale  en_GB

Chaining the above requests we can make a request like this. -L causes curl to continue following the redirect.

1$ curl https://httpbin.org/cookies/set/locale/en_GB -c cookies.txt -b cookies.txt -L
2
3{
4  "cookies": {
5    "locale": "en_GB"
6  }
7}

POST data

The option to use in this case is -D flag. When you use this option, the Content-Type: application/x-www-form-urlencoded request header is automatically set, so you don’t need to specify it explicitly. You can specify the data to be posted immediately after this flag. A sample request and response is added below. Check the response json’s form field.

 1$ curl "https://httpbin.org/post" -d "firstname=John&lastname=Doe"
 2
 3{
 4  "args": {}, 
 5  "data": "", 
 6  "files": {},
 7  "form": {
 8    "firstname": "John",
 9    "lastname": "Doe"
10  },
11  "headers": {
12    "Accept": "*/*",
13    "Content-Length": "27",
14    "Content-Type": "application/x-www-form-urlencoded",
15    "Host": "httpbin.org",
16    "User-Agent": "curl",
17    "X-Amzn-Trace-Id": "Root=1-60ec27f5-4d461f2d3bbf20ed16291ed3"
18  },
19  "json": null,
20  "url": "https://httpbin.org/post"
21}

You could also hide the data you are submitting from the command line or history by reading it off a file like so -

1$ curl  "https://httpbin.org/post" -d "@sensitive-data.txt"

POST multifield form

It’s quite straightforward to submit a multifield form using curl. The option to use is -F. You can add as many fields as required by adding multiple -F flags. When you use this option, the Content-Type: multipart/form-data request header is automatically set, so you don’t need to specify it explicitly.

Here is one such request and response -

 1$ curl -F "firstname=John" -F "lastname=Doe" "https://httpbin.org/post"
 2
 3{
 4  "args": {},
 5  "data": "",
 6  "files": {},
 7  "form": {
 8    "firstname": "John",
 9    "lastname": "Doe"
10  },
11  "headers": {
12    "Accept": "*/*",
13    "Content-Length": "248",
14    "Content-Type": "multipart/form-data; boundary=------------------------e41227aceee312a6",
15    "Host": "httpbin.org",
16    "User-Agent": "curl",
17    "X-Amzn-Trace-Id": "Root=1-60ec23d7-3a82be0162af35456010c778"
18  },
19  "json": null,
20  "url": "https://httpbin.org/post"
21}

File upload

Similar to using the multifield form above, you can also upload a file using the -F flag. The only thing to be aware of is the filepath should be prefixed with a @ char. Check the field files in the response json.

 1$ curl -F "firstname=John" -F "lastname=Doe" "https://httpbin.org/post" -F "file=@file.txt"
 2
 3{
 4  "args": {},
 5  "data": "",
 6  "files": {
 7    "file": "This is a test file\n"
 8  },
 9  "form": {
10    "firstname": "John",
11    "lastname": "Doe"
12  },
13  "headers": {
14    "Accept": "*/*",
15    "Content-Length": "408",
16    "Content-Type": "multipart/form-data; boundary=-----afd9220cdc8dae3d",
17    "Host": "httpbin.org",
18    "User-Agent": "curl",
19    "X-Amzn-Trace-Id": "Root=1-60ec24b5"
20  },
21  "json": null,
22  "url": "https://httpbin.org/post"
23}

Conclusion

While this list is just a start, I hope it helps you use curl more effectively. As promised, I will try to keep this updated, so that over time, it may eventually become a lot more exhaustive. 👍

References (2)

  1. Manpage 
  2. Ec.haxx.se 

Related