When your application handles a large number of files, you need to store the files in a certain storage system outside your application server. One of the best storage services is Amazon S3. This article will show you how to create an Amazon S3 Bucket, create access key id and secret, and upload files to Amazon S3 with Go.
What is Amazon S3
Amazon S3 or Simple Storage Service is a storage service provided by Amazon AWS. We can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.
To store data to S3 with a Go application, first, we need to create the bucket and access credentials.
If you already have your bucket and credentials, you can skip to this section.
Create an S3 bucket
To create your S3 bucket, first, go to the AWS console page. Then search S3 in the search bar. You will be directed to the S3 dashboard page.
Create Bucket button.
Bucket nameand select your prefered
AWS Region. Leave other setting as is for now.
Create Bucketbutton at the bottom of the page. The bucket will be created.
Create IAM to access the bucket
After the bucket is created, we need to create an access key id and secret so our Go application can access the bucket. To create the credential, go to the AWS console page, then search IAM in the search bar. Then click IAM to go to the IAM dashboard page.
On the IAM dashboard page, click
Users on the left sidebar. Then click
Input the User name and check the
Access key - Programmatic accesscheckbox. Then click the
We need to attach S3 access policy to the user.
Attach existing policies directly.
- Search S3 in the filter.
- Then tick
You can add tags if you want. It is optional.
Review the user creation to make sure the parameters are correct. Then click
The user is successfully created.
Access Key IDand
Secret access key. It is needed to access the bucket from Go application.
How to upload file to S3 with Go
Once we have the bucket and the access key, our Go app can upload files to the S3. AWS provided official Go SDK to manage S3 (https://github.com/aws/aws-sdk-go/).
To upload a file to S3 we need to create an S3 uploader and call the Upload method of the uploader.
Use the following code sample to create the uploader.
On line 11, we create AWS config with our bucket’s region and access key ID and secret key of the IAM. This config is needed to create a session for the uploader. Make sure to import all the necessary libraries.
Once the uploader is created, we can call the
UploadWithContextmethod of the uploader to upload a file to the S3. Let’s see the sample code below.
UploadInputwhich contains the bucket’s name and attributes of the file. The context can be used to limit upload time and cancelation.
Now try to call the function to upload a file and see if it is uploaded in the bucket. Note that by default, you can’t access the file url because public access is not enabled. To enable it, you need to change the bucket policy.
To upload files to Amazon S3, we can use the official Go SDK provided by AWS. We need the bucket and access key ID and secret key to upload from the Go app. Once we have those, the Go code to upload the file is simple.
The complete code
The complete code is here (click to expand)