Curl Cheatsheet

Curl is an invaluable tool for anyone who needs to debug webapps fequently. Given it’s vast amount of options, it is easy to miss some of it’s simplest yet powerful uses.

Listing here some options that I use everyday. Use the topics below to navigate direct to what you are looking for.

I’ll keep updating it as and when I discover something new! 🤘

Topics

Inspect response headers

The simplest way to do that is by using the -I option -

curl -I https://httpbin.org/ip
HTTP/2 200 
date: Thu, 12 Nov 2019 19:08:13 GMT
content-type: application/json
content-length: 32
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

The above works in most cases, but if your webapp/endpoint doesn’t support the HEAD method or behaves differently to it compared to the usual GET method, this will dissappoint you.

To inspect the headers with a GETmethod, following options can be helpful.

curl -D - https://httpbin.org/ip -s -o /dev/null
HTTP/2 200 
date: Thu, 12 Nov 2019 19:10:39 GMT
content-type: application/json
content-length: 32
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

OK, this works 👍 but there are a lot of options. What do they tell curl?

-D dump the headers to a file.
- dump it to stdin instead of a file.

-s suppress the progress update (like the following), helps to keep the output clean.

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                    Dload  Upload   Total   Spent    Left  Speed
                            0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200

-o send any output to file
-o /dev/nulldiscard the response completely.

Inspect request headers

I find -v option to be the most useful to inspect the request headers. Check the request headers on the lines starting with >

curl -v https://httpbin.org/ip -s -o /dev/null | grep ">"

*   Trying 3.211.1.78:443...
* Connected to httpbin.org (3.211.1.78) port 443 (#0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x1da3b7de150)
} [5 bytes data]
> GET /ip HTTP/2
> Host: httpbin.org
> user-agent: curl/7.73.0
> accept: */*

Inspect redirect responses

Redirects are implemented for a variety of reasons. Redirect responses return a 30X response code and a location header which tells the requestor where to go next. The following request gives us these filtering out the rest.

curl https://httpbin.org/status/302  -w "%{response_code} %{redirect_url}" -s -o /dev/null
302 https://httpbin.org/redirect/1

OK, this is nice 👌 but what does w tell curl?

-w is write-out option and takes a number of variables. These variables need to be enclosed in a %{ } block. I have used 2 here. response_code and redirect_url which are sort of self explanatory. Full list of variables it supports is here

What if you want to inspect the redirects for multiple pages? Add the urls in a list and run this one liner loop to check all the redirects in one shot, and also format the output a bit while we are at it.

https://httpbin.org/status/301
https://httpbin.org/status/302
https://httpbin.org/status/307

while read url; do curl $url -w " %{response_code} | $url --> %{redirect_url} \n" -s -o /dev/null; done < urls.txt
301 | https://httpbin.org/status/301 --> https://httpbin.org/redirect/1 
302 | https://httpbin.org/status/302 --> https://httpbin.org/redirect/1 
307 | https://httpbin.org/status/307 --> https://httpbin.org/redirect/1

Change the User-Agent

By default, curl uses a user-agent request header like, user-agent: curl/7.73.0

If you need to send a custom user-agent, simply use the -A option. Check the User-Agent key value in the returned json response.

curl https://httpbin.org/headers -A "bob-the-builder"

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "bob-the-builder",
    "X-Amzn-Trace-Id": "Root=1-5fadb05b-490211b54febef30592912b5"
  }
}

Send custom headers with request

To send custom headers with a request, you can use the -H option. Check the headers section in the returned json response.

curl https://httpbin.org/headers -H "CustomHeader1: value1" -H "CustomHeader2: value2"

{
  "headers": {
    "Accept": "*/*",
    "Customheader1": "value1",
    "Customheader2": "value2",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.73.0",
    "X-Amzn-Trace-Id": "Root=1-5fadb190-26af70e1691093c76d180401"
  }
}

Send cookies with request

To send cookies with a request, you can use the -b option. The cookies can either be provided inline or read from a file. Check the cookies in the json response.

Send the cookie values inline

$ curl https://httpbin.org/cookies -b "country=GB; language=EN; debug=1"

{
  "cookies": {
    "country": "GB",
    "debug": "1",
    "language": "EN"
  }
}

Send the cookie values from a file

$ curl https://httpbin.org/cookies -b cookies.txt

{
  "cookies": {
    "country": "GB",
    "debug": "2",
    "language": "EN",
    "source": "file"
  }
}

Note The cookie.txt file needs to be in the following format for curl to read and parse the cookies

httpbin.org	FALSE	/	FALSE	0	country GB
httpbin.org	FALSE	/	FALSE	0	language    EN
httpbin.org	FALSE	/	FALSE	0	source  file
httpbin.org	FALSE	/	FALSE	0	debug   2

Save response cookies, and reuse

To save cookies received in a response, you can use the -c option. The cookies can then be easily used in subsequent requests using the -b option as we saw above.

curl https://httpbin.org/cookies/set/locale/en_GB -c cookies.txt

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href="/cookies">/cookies</a>.  If not click the link.

Let’s inspect the file, if cookie is saved. The contents of the file would be as follows -

# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

httpbin.org     FALSE   /       FALSE   0       locale  en_GB

Chaining the above requests we can make a request like this. -L causes curl to continue following the redirect.

curl https://httpbin.org/cookies/set/locale/en_GB -c cookies.txt -b cookies.txt -L

{
  "cookies": {
    "locale": "en_GB"
  }
}

POST multifield form

It’s quite straightforward to submit a multifield form using curl. The option to use is -F. You can add as many fields as required by adding multiple -F flags. When you use this option, the Content-Type: multipart/form-data request header is automatically set, so you don’t need to specify it explicitly.

Here is one such request and response -

curl -F "firstname=John" -F "lastname=Doe" "https://httpbin.org/post"

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "firstname": "John",
    "lastname": "Doe"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "248",
    "Content-Type": "multipart/form-data; boundary=------------------------e41227aceee312a6",
    "Host": "httpbin.org",
    "User-Agent": "curl",
    "X-Amzn-Trace-Id": "Root=1-60ec23d7-3a82be0162af35456010c778"
  },
  "json": null,
  "url": "https://httpbin.org/post"
}

File upload

Similar to using the multifield form above, you can also upload a file using the -F flag. The only thing to be aware of is the filepath should be prefixed with a @ char. Check the field files in the response json.

curl -F "firstname=John" -F"lastname=Doe" "https://httpbin.org/post" -F"file=@file.txt"

{
  "args": {},
  "data": "",
  "files": {
    "file": "This is a test file\n"
  },
  "form": {
    "firstname": "John",
    "lastname": "Doe"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "408",
    "Content-Type": "multipart/form-data; boundary=------------------------afd9220cdc8dae3d",
    "Host": "httpbin.org",
    "User-Agent": "curl",
    "X-Amzn-Trace-Id": "Root=1-60ec24b5-1cdf63422773e29d6c0b653f"
  },
  "json": null,
  "url": "https://httpbin.org/post"
}

POST data

The option to use in this case is -D flag. When you use this option, the Content-Type: application/x-www-form-urlencoded request header is automatically set, so you don’t need to specify it explicitly. You can specify the data to be posted immediately after this flag. A sample request and response is added below. Check the response json’s form field.

curl  "https://httpbin.org/post" -d "firstname=John&lastname=Doe"

{
  "args": {}, 
  "data": "", 
  "files": {},
  "form": {
    "firstname": "John",
    "lastname": "Doe"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "27",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "curl",
    "X-Amzn-Trace-Id": "Root=1-60ec27f5-4d461f2d3bbf20ed16291ed3"
  },
  "json": null,
  "url": "https://httpbin.org/post"
}

You could also hide the data you are submitting from the command line or history by reading it off a file like so -

curl  "https://httpbin.org/post" -d "@sensitive-data.txt"

Conclusion

While this list is just a start, I hope it helps you use curl more effectively. As promised, I will try to keep this updated, so that over time, it may eventually become a lot more exhaustive. 👍

Curl Cheatsheet

Topics

Inspect response headers

Inspect request headers

Inspect redirect responses

Change the User-Agent

Send custom headers with request

Send cookies with request

Save response cookies, and reuse

POST multifield form

File upload

POST data

Conclusion

References (2)

Related

S3 Presigned URLs

Topics

Inspect response headers#

Inspect request headers#

Inspect redirect responses#

Change the User-Agent#

Send custom headers with request#

Send cookies with request#

Save response cookies, and reuse#

POST multifield form#

File upload#

POST data#

Conclusion#

References (2)

Related

S3 Presigned URLs

Inspect response headers

Inspect request headers

Inspect redirect responses

Change the User-Agent

Send custom headers with request

Send cookies with request

Save response cookies, and reuse

POST multifield form

File upload

POST data

Conclusion