Jamstack Cloudformation

Problem and Audience

A simple clouformation template makes it easy to stamp out AWS infrastructure for a jamstack web site. A jamstack is a web site composed of static presentation (non-API) resources (html, css, javascript) assembled at build time - as opposed to a site that requires some kind of dynamic server side rendering of resources at request time. A nice feature of this architecture is that it allows a site to be served inexpensively from a serverless object storage system like S3. We manage https://apps.frickjack.com by copying presentation resources to an S3 bucket that acts as an origin for a cloudfront distribution.

We originally setup the infrastructure for https://apps.frickjack.com by clicking around the AWS web console (like this), but we want to tear down that infrastructure, and move to cloudformation managed infrastructure to realize the benefits of infrastructure as code including:

• it is less work to manipulate infrastructure by editing json files than clicking through the web console
• a cloudformation template allows us to deploy multiple copies of our architecture (for different products, test environments, etc) in a consistent way
• cloudformation templates capture best practices and institutional conventions, and allow infrastructure to evolve over time
• tracking the cloudformation template and parameters in git gives an audit trail
• cloudformation makes automation easy

Jamstack Requirements

We have a handful of requirements for our jamstack infrastructure. First we want the S3 bucket to remain private and encrypted. Even though the content of the bucket is publicly accessible via the cloudfront CDN, making the origin bucket conform to standard S3 best practices simplifies compliance, since we do not need to note exceptions to organization policies that expect a bucket to be private and encrypted. The cloudfront CDN is granted access to the private bucket via a bucket policy that gives read permission to an origin access identity associated with the cloudfront distribution.

Most of our other requirements are addressed by adjusting knobs on our cloudfront configuration. For example, we want all http traffic to be redirected to https. We want to use an input parameter to our cloudformation stack to associate an alias domain with the distribution - we manage the DNS setup for the alias in another Route53 stack. We use another input parameter to feed the ARN of ourACM-managed TLS certificate. We configure cloudfront to require clients to use TLS 1.2 or better.

Finally, our little stack tools make it easy to follow the tagging conventions that we want to enforce across our infrastructure.

the little stack

We setup the following template (also in github) to start managing our jamstack infrastructure with cloudformation. The template takes advantage of the nunjucks extensions to cloudformation templates supported by our little stack automation.

One "gotcha" that we ran into was that we originally intended to setup a new stack with the apps.frickjack.com domain alias already in use by our live CDN, copy our web content to the new bucket (modify our codebuild CI pipeline configuration to do that for us), then update DNS to point the apps.frickjack.com domain at the new CDN. However, it turns out that cloudfront does not allow two distributions to have the same alias, so we setup our new CDN with a temporary alias, then took a few minutes of downtime while we removed the apps.frickjack.com alias from the old CDN, then updated our new stack.

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Metadata": {
    "License": "Apache-2.0"
  },
  "Description":
    "Generalize AWS s3-cdn sample template from - https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/S3/S3_Website_With_CloudFront_Distribution.yaml",
  "Parameters": {
    "CertificateArn": {
      "Type": "String",
      "Description": "ACM Certificate ARN"
    },
    "DomainName": {
      "Type": "String",
      "Description": "The DNS name of the new cloudfront distro",
      "AllowedPattern": "(?!-)[a-zA-Z0-9-.]{1,63}(?<!-)",
      "ConstraintDescription": "must be a valid DNS zone name."
    },
    "BucketSuffix": {
      "Type": "String",
      "Description": "The suffix of the bucket name - prefix is account number",
      "AllowedPattern": "[a-zA-Z0-9-]{1,63}",
      "ConstraintDescription": "must be a valid S3-DNS name"
    }
  },
  "Resources": {
    "S3Bucket": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "AccessControl": "Private",
        "BucketName": { "Fn::Join": [ "-", [ { "Ref" : "AWS::AccountId" }, { "Ref": "BucketSuffix" } ] ] },
        "BucketEncryption": {
          "ServerSideEncryptionConfiguration" : [ 
            {
              "BucketKeyEnabled" : "true",
              "ServerSideEncryptionByDefault" : {
                "SSEAlgorithm": "AES256"
              }
            }
          ]
        },        
        "WebsiteConfiguration": {
          "IndexDocument": "index.html",
          "ErrorDocument": "error.html"
        },
        "Tags": [
          {{ stackTagsStr }}
        ]
      }
    },
    "CloudFrontOriginIdentity": {
      "Type": "AWS::CloudFront::CloudFrontOriginAccessIdentity",
      "Properties": {
        "CloudFrontOriginAccessIdentityConfig": {
          "Comment": "origin identity"
        }
      }
    },
    "BucketPolicy": {
      "Type": "AWS::S3::BucketPolicy",
      "Properties": {
        "Bucket": { "Ref": "S3Bucket" },
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            { 
              "Effect": "Allow",
              "Principal": {
                "AWS": { "Fn::Sub": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${CloudFrontOriginIdentity}" }
              },
              "Action": "s3:GetObject",
              "Resource": { "Fn::Sub": "arn:aws:s3:::${S3Bucket}/*" }
            }
          ]
        }
      }
    },
    "CdnDistribution": {
      "Type": "AWS::CloudFront::Distribution",
      "Properties": {
        "DistributionConfig": {
          "Aliases": [
            { "Ref": "DomainName" }
          ],
          "Origins": [
            { 
              "DomainName": { "Fn::Sub": "${S3Bucket}.s3.${AWS::Region}.amazonaws.com" },
              "Id": "S3-private-bucket",
              "S3OriginConfig": {
                "OriginAccessIdentity": { "Fn::Sub": "origin-access-identity/cloudfront/${CloudFrontOriginIdentity}" }
              }
            }
          ],
          "DefaultRootObject": "index.html",
          "Enabled": "true",
          "Comment": { "Ref": "DomainName" },
          "DefaultCacheBehavior": {
            "AllowedMethods": [ "GET", "HEAD", "OPTIONS" ],
            "CachedMethods": [ "GET", "HEAD", "OPTIONS" ],
            "TargetOriginId": "S3-private-bucket",
            "ForwardedValues": {
              "QueryString": "false",
              "Cookies": {
                "Forward": "none"
              }
            },
            "ViewerProtocolPolicy": "redirect-to-https"
          },
          "ViewerCertificate": {
            "AcmCertificateArn": { "Ref": "CertificateArn" },
            "SslSupportMethod": "sni-only",
            "MinimumProtocolVersion": "TLSv1.2_2019"
          }
        },
        "Tags": [
          { "Key": "Name", "Value": { "Ref": "DomainName" } },
          {{ stackTagsStr }}
        ]
      }
    }
  },
  "Outputs": {
    "CdnAliasDomain": {
      "Value": { "Fn::GetAtt": [ "CdnDistribution", "DomainName" ] },
      "Description": "The URL of the newly created website"
    },
    "BucketName": {
      "Value": { "Ref": "S3Bucket" },
      "Description": "Name of S3 bucket to hold website content"
    }
  }
}

Summary

A simple clouformation template makes it easy to stamp out AWS infrastructure for a jamstack web site.

Publish Web Content to S3

I host a little website at https://apps.frickjack.com by publishing static web files to an S3 bucket behind a cloudfront CDN. One way to improve the performance of an S3 web site is to configure each file's (s3 object) content encoding (gzip'ing the file if required), content type, and cache control at upload time (S3 natively specifies an Etag).

Here's a little script that makes that easy (also at https://github.com/frickjack/misc-stuff/blob/master/AWS/bin/s3web.sh), but rather than copy the script - you can install it like this:

git clone https://github.com/frickjack/misc-stuff.git
alias little="bash $(pwd)/Code/misc-stuff/AWS/little.sh"

little help s3web
little s3web publish localFolder s3Path [--dryrun]

Given a local source folder with html and other web content, then little s3web publish localFolder s3://destiation does the following:

• html files are gzipped, and annotated with headers content-encoding gzip, mime-type, cache for one hour
• css, js, svg, map, json, md files are gzipped, and annotated with content-encoding, mime-type, and cache for 1000 hours
• png, jpg, wepb files are not gzipped, and annotated with mime-type and cache for 1000 hours

This caching policy assumes that html files may periodically change to reference new assets (javascript, css, etc), but the referenced assets are immutable. For example, https://apps.frickjack.com loads javascript and CSS assets from a versioned subpath, and every update to https://apps.frickjack.com publishes a new folder of assets.

#
# helpers for publishing web content to S3
#

source "${LITTLE_HOME}/lib/bash/utils.sh"

# lib ------------------------

#
# Determine mime type for a particular file name
#
# @param path
# @return echo mime type
#
s3webPath2ContentType() {
    local path="$1"
    shift
    case "$path" in
        *.html)
            echo "text/html; charset=utf-8"
            ;;
        *.json)
            echo "application/json; charset=utf-8"
            ;;
        *.js)
            echo "application/javascript; charset=utf-8"
            ;;
        *.mjs)
            echo "application/javascript; charset=utf-8"
            ;;
        *.css)
            echo "text/css; charset=utf-8"
            ;;
        *.svg)
            echo "image/svg+xml; charset=utf-8"
            ;;
        *.png)
            echo "image/png"
            ;;
        *.jpg)
            echo "image/jpeg"
            ;;
        *.webp)
            echo "image/webp"
            ;;
        *.txt)
            echo "text/plain; charset=utf-8"
            ;;
        *.md)
            echo "text/markdown; charset=utf-8"
            ;;
        *)
            echo "text/plain; charset=utf-8"
            ;;
    esac
}

#
# Get the npm package name from the `package.json`
# in the current directory
#
s3webPublish() {
    local srcFolder
    local destPrefix
    local dryRun=off

    if [[ $# -lt 2 ]]; then
        gen3_log_err "s3web publish takes 2 arguments"
        little help s3web
        return 1
    fi
    srcFolder="$1"
    shift
    destPrefix="$1"
    shift
    if [[ $# -gt 0 && "$1" =~ ^-*dryrun ]]; then
        shift
        dryRun=on
    fi
    if [[ ! -d "$srcFolder" ]]; then
        gen3_log_err "invalid source folder: $srcFolder"
        return 1
    fi
    if ! [[ "$destPrefix" =~ ^s3://.+ ]]; then
        gen3_log_err "destination prefix should start with s3:// : $destPrefix"
        return 1
    fi
    local filePath
    local ctype
    local encoding
    local cacheControl
    local errCount=0
    local gzTemp
    local commandList
    (
        commandList=()
        cd "$srcFolder"
        gzTemp="$(mktemp "$XDG_RUNTIME_DIR/gzTemp_XXXXXX")"
        find . -type f -print | while read -r filePath; do
            ctype="$(s3webPath2ContentType "$filePath")"
            encoding="gzip"
            cacheControl="max-age=3600000, public"
            destPath="${destPrefix%/}/${filePath#./}"
            if [[ "$ctype" =~ ^image/ && ! "$ctype" =~ ^image/svg ]]; then
                encoding=""
            fi
            if [[ "$ctype" =~ ^text/html || "$ctype" =~ ^application/json ]]; then
                cacheControl="max-age=3600, public"
            fi

            if [[ "$encoding" == "gzip" ]]; then
                gzip -c "$filePath" > "$gzTemp"
                commandList=(aws s3 cp "$gzTemp" "$destPath" --content-type "$ctype" --content-encoding "$encoding" --cache-control "$cacheControl")
            else
                commandList=(aws s3 cp "$filePath" "$destPath" --content-type "$ctype" --cache-control "$cacheControl")
            fi
            gen3_log_info "dryrun=$dryRun - ${commandList[@]}"
            if [[ "$dryRun" == "off" ]]; then
                "${commandList[@]}" 1>&2
            fi
            errCount=$((errCount + $?))
            if [[ -f "$gzTemp" ]]; then
                /bin/rm "$gzTemp"
            fi
        done
        [[ $errCount == 0 ]]
    )
}

# main ---------------------

# GEN3_SOURCE_ONLY indicates this file is being "source"'d
if [[ -z "${GEN3_SOURCE_ONLY}" ]]; then
    command="$1"
    shift

    case "$command" in
        "content-type")
            s3webPath2ContentType "$@"
            ;;
        "publish")
            s3webPublish "$@"
            ;;
        *)
            little help s3web
            ;;
    esac
fi

paging s3 listObjects as a scala stream

One of Scala's fun features is its lazy list Stream class. I guess for a Haskell programmer a lazy list is not a big deal, but it's cool for the rest of us java, ruby, python, javascript, C#, ... programmers.

Anyway - I plan to host some web content on S3, so I've been writing some scala code using S3's java API to automate some tasks. One of the patterns S3 employs is to "page" large result sets, and scala streams provide a natural way to load the pages on demand - something like this:

      lazy val s3Listing:Stream[s3.model.ObjectListing] = 
          s3Client.listObjects( 
            new s3.model.ListObjectsRequest( folder.getHost, queryFolderPath, null, "/", null ) 
          ) #:: s3Listing.takeWhile( _.isTruncated ).map( (part) => s3Client.listNextBatchOfObjects( part ) )

Pretty cool, but, unfortunately, Stream has some quirk where its flatMap method can easily overflow the stack. For example - the following code for assembling the directories in a file system explodes:

          lazy val folderStream:Stream[jio.File] = new java.io.File( folder.path.getPath ).getCanonicalFile #:: 
            folderStream.flatMap( (d) => { d.listFiles().filter( (sd) => sd.isDirectory ) } )

We can construct a stream, and avoid calling flatMap with a recursive method that manages its own stack.

          def depthFirst( stack:List[jio.File] ):Stream[jio.File] = {
            stack match {
              case head :: tail => {
                  head #:: depthFirst( tail ++ head.listFiles.filter( _.isDirectory ) )
              }
              case Nil => Stream.empty
            }
          }

          depthFirst( List( root ) )

Extra S3 log parameters with AWS Java SDK

Amazon web services' S3 REST API supports custom access log information by adding custom query string parameters beginning with "x-" that S3 ignores but logs. Unfortunately custom query parameters are not directly supported in the AWS Java SDK, so we were planning to drop down to writing HTTP REST code for a recent project that required logging of some custom client-session parameters.

I hate working with HTTP, so I was glad when we figured out a workaround. The trick we came up with was to register a custom RequestHandler with our instance of the SDK's AmazonS3Client. The RequestHandler is an internal class in the API, so it's not included in the API documentation, but you can download the source code to see its simple interface:

/**
 * Interface for addition request handling in clients. A request handler is
 * executed on a request object before it is sent to the client runtime
 * to be executed.
 */
public interface RequestHandler {

    /**
     * Runs any additional processing logic on the specified request (before it
     * is executed by the client runtime).
     *
     * @param request
     *            The low level request being processed.
     */
    public void beforeRequest(Request request);

 /**
  * Runs any additional processing logic on the specified request (after is
  * has been executed by the client runtime).
  *
  * @param request
  *            The low level request being processed.
  * @param response
  *            The response generated from the specified request.
  * @param timingInfo
  *            Timing information on the request's processing.
  */
    public void afterResponse(Request request, Object response, TimingInfo timingInfo);

 /**
  * Runs any additional processing logic on a request after it has failed.
  *
  * @param request
  *            The request that generated an error.
  * @param e
  *            The error that resulted from executing the request.
  */
    public void afterError(Request request, Exception e);

}

So the RequestHandler's beforeRequest method can add parameters to a Request before it is processed by the underlying HTTP engine. The only trick is to determine which parameters to add to a particular Request, but it's easy to extend the SDK's API-level request objects (like S3's GetObjectRequest) with subtypes that add application-specific properties, then check for for those subtypes in the RequestHandler via the getOriginalRequest accessor.