Introduction
S3 is one of the core services offered by AWS, and more often than not, fulfills a critical storage need in most architectures. Over time, AWS has made it incredibly easy to hook it up with other services, further increasing its adoption. As the Google trends below indicate, S3 is possibly THE most widely used AWS service, even edging out the likes of Lambda, EC2, IAM etc. It may not be a 100% accurate inference, but I am sure, directionally, it is not too far off.
With such huge adoption, security becomes a critical issue. AWS provides a number of tools to help reduce bucket misconfigurations. But, they are still rampant, and cause data breaches quite often. You can read about some recent examples here and here. For more, take a look at this neat list, going back several years.
If anything, these breaches show us, how crucial it is to safeguard the S3 bucket. There are a lot of strategies to do this, like bucket policies, IAM policies, encryption, config rules etc. Refer to this best practice guide published by AWS, to get more details.
Let’s now look at one approach, which can help keep S3 buckets secure. It still allows specific actions (read/write) on specific data, but for a limited time.
Problem
How to provide temporary access to your bucket?
Maybe, you need to temporarily share a file in your private S3 bucket with a third party, which doesn’t have any AWS access.
Or, you need to temporarily allow access to the third part to upload a file to your private S3 bucket.
These cases can easily be extended to cover similar use cases like apps trying to download from or upload to S3.
Solution
For providing temporary access to S3 buckets, you can use what AWS calls Presigned URLs. These are essentially temporary urls which you can share with anyone who needs access to your bucket/object. Using these temporary urls, the recepient can then complete the desired action (read/write) in the stipulated time. Remember, since these urls are tied to the particular object and are temporary, (expiring after a set duration) there is low risk of exploits using them.
How to create a presigned url?
A presigned url for an object in a bucket can only be created by an IAM entity which has access to it. So, somebody with only a read access on your bucket will not be able to create a presigned url to allow an upload. This is important to mention, since, if you are creating these urls programatically in an app, the role assigned to this app must allow the same action.
AWS SDK (Go/Python/JS etc.) provides methods to generate these presigned urls. You can even use AWS CLI to generate a presigned url, if you want to give read-only (download) access.
Using Python Boto3
Note: this is a basic example with the bare minimum steps
Let’s create a presigned upload url using Python boto3 generate_presigned_post method, which will be valid for the 10 mins.
1import boto3
2
3s3_client = boto3.client('s3', region_name='us-east-1')
4resp = s3_client.generate_presigned_post("mybucket", "path/to/file.txt", ExpiresIn=600)
5print(resp)
Similarly, to create a presigned url to download a file, the appropriate boto3 method to use would be generate_presigned_url.
Once you execute the above code, you will receive a json response as follows
1{
2 "url": "https://mybucket.s3.amazonaws.com/",
3 "fields": {
4 "key": "path/to/file.txt",
5 "AWSAccessKeyId": "access_key",
6 "x-amz-security-token": "security_token",
7 "policy": "base64_encoded_policy",
8 "signature": "signature"
9 }
10}
Now, you can share this json with the third party to allow them to upload this file to your bucket. They will not be able to do anything on any other object in the bucket or even any other action on the same object.
Validating the presigned URL
The simplest way to validate this url is to upload a file by the same name as specified under “key” above. Let’s do that using curl.
For uploading a file, use the form fields with the -F flag. The -v flag will allow you to inspect the response headers Ref. Include all the fields as received in the json above. Remember to include the file only at the end, otherwise you may receive an error.
The final curl command will look like this (assuming the file to upload file.txt exists in the local working directory) -
1$ curl -v "https://mybucket.s3.amazonaws.com/" \
2-F "key=path/to/file.txt" \
3-F "AWSAccessKeyId=access_key" \
4-F "x-amz-security-token=security_token" \
5-F "policy=base64_encoded_policy" \
6-F "signature=signature" \
7-F "file=@file.txt"
The response to this call will NOT be 200 OK. You will instead get a 204 Content not modified response. Nothing to worry though, as this means the transfer has been successful. Check your bucket to verify the file is now available.
Conclusion
Presigned URLs are the recommended way to grant temporary access to your bucket and objects. It is a great way to offload the overheads associated with file upload/download to Amazon. This is an incredibly useful design pattern for serverless applications as they can simply pass on the presigned url to the client (whether it’s a mobile app or web app), and then clients deal with S3 directly to upload/download the file.
It can also come in handy while working with people outside your organisation (maybe vendors), to share information/data which you cannot send/share by other means.
Hopefully, this quick walkthrough was helpful in explaining the usefulness of presigned urls. I have also linked some useful pages on the topic below.