How to Set Up and Serve Private Content using Amazon S3 and CloudFront

Here are my notes regarding an excellent post by Mark S. Rasmussen about Serving Private Content using Amazon S3 and CloudFront.  He uses Dot Net and C# for the steps.  By now, the browser-based AWS console has sufficient features to let us do this without compiling anything in C#.  These notes follow his sequence and tell you how to do each step without C#.

"Why you don’t want to rely on third party GUIs"

I agree with Rasmussen that none of this will make much sense unless you have already learned a great deal about AWS, S3, CloudFront and SHA1 signing.  He is right; using the AWS command line tools or the AWS API is the ultimate way to mastery.  

Beginners, do not cross this line, sorry.

Creating the private bucket

Do NOT use a period ('.') in your bucket name if you (ever) want to use the AWS feature for accelerated upload.  And you really should enable the acceleration feature.  This can be done inside the aws browser-based console, after the bucket exists.  Transfer acceleration one of the S3 bucket properties. 

I used MSP360 Explorer (formerly CloudBerry Explorer) to make a new AWS S3 bucket.   If you are not using USA-East-1 aka US-Standard aka the data center in Virginia USA, be especially careful to control your bucket region.

Creating a CloudFront Origin Access Identity

Login to aws.

Services > CloudFront

On the left-side menu, Private Content > Origin Access Identity

Click [Create OAI]

Enter comment, e.g. OAI used for private distribution access to ___ bucket in ___ region

Be sure to jot down the OAI ID and OAI S3 canonical user ID, or just open the next step in a separate browser window so you can clipboard those details when needed.

Creating the private CloudFront distribution

Services > CloudFront

On the left-side menu, Distributions

Click [Create Distribution]

You will probably be making a "Web" distribution, not Adobe RTMP.  Click the appropriate [Get Started] button.

Answer the questions on the form:

Origin Domain Name: select the private bucket you just made.  Note that the ending of the bucket name SHOULD be .s3.amazonaws.com

Origin Path: leave blank

Origin ID: accept default provided

Restrict Bucket Access: Yes !!  If you leave the default, No, you will not end up with a secure solution.  Click the (i) info-help button for details.

Origin Access Identity:  Use an Existing Identity

Your Identities: select the one you just created. Look for the comment you specified.

Grant Read Permissions on Bucket: Yes

Origin Custom Headers: leave blank

Default Cache Behavior Settings: leave all defaults except

Restrict Viewer Access: Yes

  Trusted Signers: Self

Compress Objects Automatically: Yes  (optional - your decision)

Distribution Settings

Price Class: leave default

AWS WAF Web ACL: None

Alternate Domain Names: optional; I entered a CNAME here

SSL: optional, I used the default CloudFront cert for now

Default Root Object: index.html

leave defaults for remaining questions

Click [Create Distribution]

Setting up a bucket policy to grant our OAI access to the S3 bucket

This will have happened automatically if you selected Grant Read Permissions on Bucket = Yes.  To verify, go to Services > S3, look at Properties of your bucket, open Permissions, click into Edit bucket policy.  Look, do not touch.  Close that without saving any changes.

CORS policy on the bucket

For added security, customize the CORS policy to restrict uploads to a limited set of domain names. If you have the WebHubDemos handy, you can see a sample file: webhubdemos\Source\WHApps\Lite Examples\AWS\showcase-005-cors-on-bucket.xml

Uploading a test object

I used CloudBerry Explorer to upload a jpg file to the top of my protected bucket.

Creating a time limited signed URL for a given object

( wait, make the key pair first )

Creating CloudFront key pairs

For use with MSP360 (formerly CloudBerry) Explorer with CloudFront, you will need a keypair where the public key is the short plain string and the private key is a PEM file.  You make this keypair inside the AWS console.

In AWS management console, look top-right for your account name which has a menu under it. Drop that down and go into Security Credentials.  From there, under the heading Your Security Credentials, click into + CloudFront Key Pairs.  Click [Create New Key Pair].

Note that for .Net and Java, the private key must be changed from the default .pem format to one suitable to the platform.

Reference: docs.aws.amazon.com Creating CloudFront Key Pairs 

Creating a time limited signed URL for a given object

CloudBerry Explorer helps with this as well.  Right-click the test jpg file, Generate Web URL. 

You should see an alert as part of the "Generate Web URL" dialog, "...You need to specify a polity to generate a valid signed url. To manage policies click [this link].  Click [this link] ! and then [Add Policy] and fill out the form. 

Once you have a policy, you can select it and then click [Generate] to have CloudBerry Explorer create the signed URL. 

Then [Open Link] to test it.

If you used a policy on a CloudFront domain, the generated URL will look like this:

http://sub.cloudfront.net/path/filename.jpg?Policy=...&Signature=...&Key-Pair-Id=...

The Key-Pair-Id will be the Access Key ID picked up from the earlier step where you created the CloudFront Key Pair. 

If you Base64 Decode the Policy parameter, you will get the policy json as plain-text.

Making Signed URLs under Program Control

Now you JUST need to use your favorite programming language and call a function to create those signed URLs yourself.  Make sure you read Amazon Docs: Serving Private Content  especially  Sample Code and Third-Party Tools and AWS Docs: Creating a Signed URL Using a Custom Policy.  

OpenSSL syntax to sign the policy JSON

type input.policy.utf8.json | bin\openssl.exe sha1   -sign private.key.pem | bin\openssl.exe  base64 > policy.signed.viaopenssl.base64.txt


Links for Delphi Coders

December 2009: Reading, writing and converting RSA keys in PEM, DER, PUBLICKEYBLOB and PRIVATEKEYBLOB formats
Jan 2011: Use the Windows Crypto API to get a Hash of a file

Mar 2003: Wrapper for Crypto API 2.0 functions

TurboPower Lockbox v3.6.3.0

OpenSSL notes in the Delphi docwiki

Delphi System.Hash.THashSHA1.GetHMAC

Wikipedia: Hash based message authentication code "HMAC"

Not FREE but WORKS with Delphi 10.1 Berlin: Chilkat RSA (dll + pas)