Friday, April 02, 2021

Sign and Verify JWT with ES256

Problem and Audience

A developer of a system that uses json web tokens (JWT) to authenticate HTTP API requests needs to generate asymmetric cryptographic keys, load the keys into code, then use the keys to sign and validate tokens.

We are building a multi-tenant system that implements a hierarchy where each tenant (project) may enable one or more api's. An end user authenticates with the global system (OIDC authentication client) via a handshake with a Cognito identity provider, then acquires a short lived session token to interact with a particular api under a particular project (OIDC resource server).

It would be nice to simply implement the session token as a Cognito OIDC access token, but our system has a few requirements that push us to manage our own session tokens for now. First, each (project, api) pair is effectively an OIDC resource server in our model, and projects and api's are created dynamically, so managing the custom JWT claims with Cognito resource servers would be messy.

Second, we want to be able to support robot accounts at the project level, and a Cognito mechanism to easily provision robot accounts and tokens is not obvious to us. So we decided to manage our "session" tokens in the application, and rely on Cognito to federate identity providers for user authentication.

JWTS with ES256

I know very little about cryptography, authentication, and authorization; but fortunately people that know more than me share their knowledge online. Scott Brady's bLog gives a nice overview of JWT signing. We want to sign and verify JWTs in scala using the elliptic curve ES256 algorithm - which improves on RSA256 in a few ways, and is widely supported.

There are different ways to generate an elliptic curve sha-256 key pair, but EC keys saved to pem files are supported by multiple tools, and are easy to save to configuration stores like AWS SSM parameter store or secrets manager.

This bash function uses openssl to generate keys in pem files.

#
# Bash function to generate new ES256 key pair
#
newkey() {
    local kid=${1:-$(date +%Y%m)}
    local secretsFolder=$HOME/Secrets/littleAudit

    (
        mkdir -p "$secretsFolder"
        cd "$secretsFolder" || return 1
        if [[ ! -f ec256-key-${kid}.pem ]]; then
          openssl ecparam -genkey -name prime256v1 -noout -out ec256-key-${kid}.pem
        fi
        # convert the key to pkcs8 format
        openssl pkcs8 -topk8 -nocrypt -in ec256-key-${kid}.pem -out ec256-pkcs8-key-${kid}.pem
        # extract the public key
        openssl ec -in ec256-pkcs8-key-${kid}.pem -pubout -out ec256-pubkey-${kid}.pem
    )
}

Load the keys into code

Now that we have our keys - we need to load them into our scala application.

class KeyHelper @inject.Inject() (
  gs: gson.Gson, 
  ecKeyFactory:KeyHelper.EcKeyFactory, 
  rsaKeyFactory:KeyHelper.RsaKeyFactory
  ) {    
    /**
     * @return pem input with pem file prefix/suffix and empty space removed
     */
    def decodePem(pem:String): String = {
      pem.replaceAll(raw"-----[\w ]+-----", "").replaceAll("\\s+", "")
    }


    def loadPublicKey(kid:String, pemStr:String):SessionMgr.PublicKeyInfo = {
      val key = ecKeyFactory.generatePublic(decodePem(pemStr))
      SessionMgr.PublicKeyInfo(kid, "ES256", key)
    }


    def loadPrivateKey(kid:String, pemStr:String):SessionMgr.PrivateKeyInfo = {
      val key = ecKeyFactory.generatePrivate(decodePem(pemStr))
      SessionMgr.PrivateKeyInfo(kid, "ES256", key)
    }

    /**
     * Load keys from a jwks url like 
     *    https://www.googleapis.com/oauth2/v3/certs
     */
    def loadJwksKeys(jwksUrl:java.net.URL): Set[SessionMgr.PublicKeyInfo] = {
      val jwksStr = {
        val connection = jwksUrl.openConnection()
        connection.setRequestProperty("Accept-Charset", KeyHelper.utf8)
        connection.setRequestProperty("Accept", "application/json")
        val response = new java.io.BufferedReader(new java.io.InputStreamReader(connection.getInputStream(), KeyHelper.utf8))
        try {
            littleware.base.Whatever.get().readAll(response)
        } finally {
            response.close()
        }
      }

      gs.fromJson(jwksStr, classOf[gson.JsonObject]).getAsJsonArray("keys").asScala.map(
          { 
            json:gson.JsonElement =>
            val jsKeyInfo = json.getAsJsonObject()
            val kid = jsKeyInfo.getAsJsonPrimitive("kid").getAsString()
            val n = jsKeyInfo.getAsJsonPrimitive("n").getAsString()
            val e = jsKeyInfo.getAsJsonPrimitive("e").getAsString()
            val pubKey = rsaKeyFactory.generatePublic(n, e)
            SessionMgr.PublicKeyInfo(kid, "RSA256", pubKey)
          }
      ).toSet 
    }
}

object KeyHelper {
    val utf8 = "UTF-8"

    /**
     * Little injectable key factory hard wired to use X509 key spec for public key
     */
    class EcKeyFactory {
        val keyFactory = java.security.KeyFactory.getInstance("EC")
        val b64Decoder = java.util.Base64.getDecoder()

        def generatePublic(base64:String):ECPublicKey = {
            val bytes = b64Decoder.decode(base64.getBytes(utf8))
            val spec = new X509EncodedKeySpec(bytes)

            keyFactory.generatePublic(spec).asInstanceOf[ECPublicKey]
        }

        def generatePrivate(base64:String):ECPrivateKey = {
            val bytes = b64Decoder.decode(base64.getBytes(utf8))
            val spec = new PKCS8EncodedKeySpec(bytes)

            keyFactory.generatePrivate(spec).asInstanceOf[ECPrivateKey]
       }
    }

    /**
     * Little injectable key factory hard wired for RSA jwks decoding
     * See: https://github.com/auth0/jwks-rsa-java/blob/master/src/main/java/com/auth0/jwk/Jwk.java
     */
    class RsaKeyFactory {
        private val keyFactory = java.security.KeyFactory.getInstance("RSA")
        private val b64Decoder = java.util.Base64.getUrlDecoder()

        def generatePublic(n:String, e:String):RSAPublicKey = {
            val modulus = new java.math.BigInteger(1, b64Decoder.decode(n))
            val exponent = new java.math.BigInteger(1, b64Decoder.decode(e))
            keyFactory.generatePublic(new RSAPublicKeySpec(modulus, exponent)).asInstanceOf[RSAPublicKey]
        }
    }
}

Sign and verify JWTs

Now that we have loaded our keys, we can use them to sign and verify JWTs. Okta has published open source code for working with JWTs , Auth0 has published open source code for working with JWK, and AWS KMS supports elliptic curve digital signing algorithms with asymmetric keys.

import com.google.inject
// see https://github.com/jwtk/jjwt#java-jwt-json-web-token-for-java-and-android
import io.{jsonwebtoken => jwt}
import java.security.{ Key, PublicKey }
import java.util.UUID
import scala.util.Try

import littleware.cloudmgr.service.SessionMgr
import littleware.cloudmgr.service.SessionMgr.InvalidTokenException
import littleware.cloudmgr.service.littleModule
import littleware.cloudutil.{ LRN, Session }

/**
 * @param signingKey for signing new session tokens
 * @param verifyKeys for verifying the signature of session tokens
 */
@inject.ProvidedBy(classOf[LocalKeySessionMgr.Provider])
@inject.Singleton()
class LocalKeySessionMgr (
    signingKey: Option[SessionMgr.PrivateKeyInfo],
    sessionKeys: Set[SessionMgr.PublicKeyInfo],
    oidcKeys: Set[SessionMgr.PublicKeyInfo],
    issuer:String,
    sessionFactory:inject.Provider[Session.Builder]
    ) extends SessionMgr {

    val resolver = new jwt.SigningKeyResolverAdapter() {
        override def resolveSigningKey(jwsHeader:jwt.JwsHeader[T] forSome { type T <: jwt.JwsHeader[T] }, claims:jwt.Claims):java.security.Key = {
            val kid = jwsHeader.getKeyId()
            (
                {
                    if (claims.getIssuer() == issuer) {
                        sessionKeys
                    } else {
                        oidcKeys
                    }
                }
            ).find(
                { it => it.kid == kid }
            ).map(
                { _.pubKey }
            ) getOrElse {
                throw new SessionMgr.InvalidTokenException(s"invalid auth kid ${kid}")
            }
        }
    }

    ...

    def jwsToClaims(jwsIdToken:String):Try[jwt.Claims] = Try(
        { 
            jwt.Jwts.parserBuilder(
            ).setSigningKeyResolver(resolver
            ).build(
            ).parseClaimsJws(jwsIdToken
            ).getBody()
        }
    ).flatMap( claims => Try( {
                    Seq("email", jwt.Claims.EXPIRATION, jwt.Claims.ISSUER, jwt.Claims.ISSUED_AT, jwt.Claims.AUDIENCE).foreach({
                        key =>
                        if(claims.get(key) == null) {
                            throw new InvalidTokenException(s"missing ${key} claim")
                        }
                    })
                    claims
                }
            )
    ).flatMap(
        claims => Try(
            {
                if (claims.getExpiration().before(new java.util.Date())) {
                    throw new InvalidTokenException(s"auth token expired: ${claims.getExpiration()}")
                }
                claims
            }
        )
    )


    def sessionToJws(session:Session):String = {
        val signingInfo = signingKey getOrElse { throw new UnsupportedOperationException("signing key not available") }
        jwt.Jwts.builder(
        ).setHeaderParam(jwt.JwsHeader.KEY_ID, signingInfo.kid
        ).setClaims(SessionMgr.sessionToClaims(session)
        ).signWith(signingInfo.privKey
        ).compact()
    }

    def jwsToSession(jws:String):Try[Session] = jwsToClaims(jws
        ) map { claims => SessionMgr.claimsToSession(claims) }

    ...
}

This code is all online under this github repo, but is in a state of flux.

Summary

To sign and verify JWTs we need to generate keys, load the keys into code, and use the keys to sign and verify tokens. We plan to add support for token signing with AWS KMS soon.

Saturday, March 20, 2021

A little framework for scala builders

Many developers use immutable data structures in application code, and leverage the builder pattern to construct the application's initial state, and progress to new versions of the state in response to user inputs and other side effects. The scala programming language supports immutable data structures well, but does not provide a native implementation of the builder pattern. The following describes a simple but useful scala builder framework that leverages re-usable data-validation lambda functions.

A Scala Framework for Builders

The following are typical examples of an immutable case class in scala:

/**
 * Little resource name URI:
 * "lrn://${cloud}/${api}/${project}/${resourceType}/${drawer}/${path}"
 */
trait LRN {
    val cloud: String
    val api: String
    val projectId: UUID
    val resourceType: String
    val drawer: String
    val path: String
}

/**
 * Path based sharing - denormalized data
 */
case class LRPath(
    cloud: String,
    api: String,
    projectId: UUID,
    resourceType: String,
    drawer: String,
    path: String
) extends LRN {
}

/**
 * Id based sharing - normalized data
 */
case class LRId(
    cloud: String,
    api: String,
    projectId: UUID,
    resourceType: String,
    resourceId: UUID
) extends LRN {
    override val drawer = ":"
    val path = resourceId.toString()
}

Builders like the following simplify object construction and validation (compared to passing all the property values to the constructor).

object LRN {
    val zeroId:UUID = UUID.fromString("00000000-0000-0000-0000-000000000000")

    trait Builder[T <: LRN] extends PropertyBuilder[T] {
        val cloud = new Property("") withName "cloud" withValidator dnsValidator
        val api = new Property("") withName "api" withValidator LRN.apiValidator
        val projectId = new Property[UUID](null) withName "projectId" withValidator notNullValidator
        val resourceType = new Property("") withName "resourceType" withValidator LRN.resourceTypeValidator
        val path = new Property("") withName "path" withValidator pathValidator

        def copy(lrn:T):this.type = this.projectId(lrn.projectId).api(lrn.api
            ).cloud(lrn.cloud).resourceType(lrn.resourceType
            ).path(lrn.path)

        def fromSession(session:Session): this.type = this.cloud(session.lrp.cloud
            ).api(session.api
            ).projectId(session.projectId
            )
    }

    class LRPathBuilder extends Builder[LRPath] {        
        val drawer = new Property("") withName "drawer" withValidator drawerValidator

        override def copy(other:LRPath) = super.copy(other).drawer(other.drawer)

        def build():LRPath = {
            validate()
            LRPath(cloud(), api(), projectId(), resourceType(), drawer(), path())
        }
    }

    class LRIdBuilder extends Builder[LRId] {
        def build():LRId = {
            validate()
            LRId(cloud(), api(), projectId(), resourceType(), UUID.fromString(path()))
        }
    }

    def apiValidator = rxValidator(raw"[a-z][a-z0-9-]+".r)(_, _)

    def drawerValidator(value:String, name:String) = rxValidator(raw"([\w-_.*]+:)*[\w-_.*]+".r)(value, name) orElse {
        if (value.length > 1000) {
            Some(s"${name} is too long: ${value}")
        } else {
            None
        }
    }

    def pathValidator(value:String, name:String) = pathLikeValidator(value, name) orElse {
        if (value.length > 1000) {
            Some(s"${name} is too long: ${value}")
        } else {
            None
        }
    }

    def resourceTypeValidator = rxValidator(raw"[a-z][a-z0-9-]{1,20}".r)(_, _)

    // ...
}

This builder implementation does not leverage the type system to detect construction errors at compile time (this blog shows an approach with phantom types), but it is composable in a straight forward way. A couple fun things about this implementation are that it leverages the builder pattern to define the properties in a builder (new Property... withName ... withValidator ...), and the setters on the nested property class return the parent Builder type, so we can write code like this:

    @Test
    def testLRNBuilder() = try {
        val builder = builderProvider.get(
        ).cloud("test.cloud"
        ).api("testapi"
        ).drawer("testdrawer"
        ).projectId(LRN.zeroId
        ).resourceType("testrt"
        ).path("*")

        val lrn = builder.build()
        assertTrue(s"api equal: ${lrn.api} ?= ${builder.api()}", lrn.api == builder.api())
    } catch basicHandler

Unfortunately, the code (in https://github.com/frickjack/littleware under littleAudit/ and littleScala/) is in a state of flux, but the base PropertyBuilder can be copied into another code base - something like this:

/**
 * Extends Validator with support for some scala types
 */
trait LittleValidator extends Validator {
  @throws(classOf[ValidationException])
  override def validate():Unit = {
    val errors = checkSanity()
    if ( ! errors.isEmpty ) {
      throw new ValidationException(
        errors.foldLeft(new StringBuilder)( (sb,error) => { sb.append( error ).append( littleware.base.Whatever.NEWLINE ) } ).toString
      )
    }
  }


  /**
   * Same as checkIfValid, just scala-friendly return type
   */
  def checkSanity():Iterable[String]
}

trait PropertyBuilder[B] extends LittleValidator {
  builder =>
  import PropertyBuilder._
  type BuilderType = this.type

  import scala.collection.mutable.Buffer

  /**
   * List of properties allocated under this class - used by isReady() below -
   * use with caution.
   */
  protected val props:Buffer[Property[_]] = Buffer.empty

  /**
   * Default implementation is props.flatMap( _.checkSanity ) 
   */
  def checkSanity():Seq[String] = props.toSeq.flatMap( _.checkSanity() )

  override def toString():String = props.mkString(",")   

  def copy(value:B):BuilderType
  def build():B

  /**
   * Typical property, so build has things like
   *     val a = new Property(-1) withName "a" withValidator { x => ... }
   *
   * Note: this type is intertwined with PropertyBuilder - don't
   *    try to pull it out of a being a subclass - turns into a mess
   */
  class Property[T](
      var value:T
    ) extends LittleValidator {    
    type Validator = (T,String) => Option[String]

    def apply():T = value

    var name:String = "prop" + builder.props.size
    var validator:Validator  = (_, _) => None

    override def checkSanity() = validator(this.value, this.name)
    def withValidator(v:Validator):this.type = {
      validator = v
      this
    }

    def withName(v:String):this.type = {
      this.name = v
      this
    }

    override def toString():String = "" + name + "=" + value + " (" + checkSanity().mkString(",") + ")"

    /** Chainable assignment */
    def apply(v:T):BuilderType = { value = v; builder }

    builder.props += this 
  }

  /**
   * Property accepts multiple values
   */
  class BufferProperty[T] extends Property[Buffer[T]](Buffer.empty) {
    def add( v:T ):BuilderType = { value += v; builder }
    def addAll( v:Iterable[T] ):BuilderType = { value ++= v; builder }
    def clear():BuilderType = { value.clear(); builder; }

    def withMemberValidator(memberValidator:(T,String) => Option[String]):this.type =
      withValidator(
        (buff, propName) => buff.view.flatMap({ it => memberValidator(it, propName) }).headOption
      )  
  }  

  class OptionProperty[T] extends Property[Option[T]](None) {
    def set(v:T):BuilderType = { value = Option(v); builder }

    def withMemberValidator(memberValidator:(T,String) => Option[String]):this.type =
      withValidator(
        (option, propName) => option.flatMap({ it => memberValidator(it, propName) })
      )  

  }
}

object PropertyBuilder {  
  /** littleware.scala.Messages resource bundle */
  val rb = java.util.ResourceBundle.getBundle( "littleware.scala.Messages" )

  def rxValidator(rx:Regex)(value:String, name:String):Option[String] = {
    if (null == value || !rx.matches(value)) {
      Some(s"${name}: ${value} !~ ${rx}")
    } else {
      None
    }
  }

  def notNullValidator(value:AnyRef, name:String):Option[String] = {
    if (null == value) {
      Some(s"${name}: is null")
    } else {
      None
    }
  }

  def positiveIntValidator(value:Int, name:String):Option[String] = {
    if (value <= 0) {
      Some(s"${name}: is not positive")
    } else {
      None
    }
  }

  def positiveLongValidator(value:Long, name:String):Option[String] = {
    if (value <= 0) {
      Some(s"${name}: is not positive")
    } else {
      None
    }
  }

  def dnsValidator = rxValidator(raw"([\w-]{1,40}\.){0,10}[\w-]{1,40}".r)(_, _)
  def emailValidator = rxValidator(raw"[\w-_]{1,20}@\w[\w-.]{1,20}".r)(_, _)
  def pathLikeValidator = rxValidator(raw"([\w-:_.*]{1,255}/){0,20}[\w-:_.*]{1,255}".r)(_, _)


}

Wednesday, January 20, 2021

Asynchronous Toolbox Pattern

A javascript application designer must structure her app to asynchronously load configuration and bootstrap its modules at startup. The asynchronous toolbox pattern addresses this challenge with a simple dependency injection framework.

The Asynchronous Bootstrap Problem

Consider an application with three modules in addition to modMain: modA, modB, and modC that export classA, classB, and classC respectively where modC imports (depends on) modA and modB, and classC is a singleton, and each module asynchronously loads configuration at startup. How should a developer structure the code in each module?

If each module's startup (configuration load) were synchronous, then the application could be structured like this.

// modC

import classA from "modA";
import classB from "modB";

function loadConfig() { ... }

class C {
    a;
    b;
    config;

    constructor(a, b, config) {
        this.a = a;
        this.b = b;
        this.config = config;
    }
    ...

    static get providerName() {
        return "driver/myApp/modA/classA";
    }
}

let lazySingleton = null;

export getC() {
    if (! lazySingleton) {
        lazySingleton = new C(new A(), new B(), loadConfig());
    }
    return lazySingleton;
}
// modB

function loadConfig() {}

const sharedConfig = loadConfig();

export class B {}
// modA

function loadConfig() {}

const sharedConfig = loadConfig();

export class A {}
// modMain
import getC from "modC";

function go() {
    const c = getC();
    ...
}

go();

If the mod*.loadConfig functions are asynchronous, then bootstrapping this simple application becomes more complicated as the asynchrony propogates through the call graph to the class constructors. This simple implementation also has a few other shortcomings that can become troublesome in a larger application.

  • First, the various loadConfig functions can be factored out to a single configuration module, modConfig, that supports default and override configurations and loading configuration from external data sources.
  • Second, in some cases it is useful to decouple an interface from its implementation, so that different applications or application deployments can use an implementation that fits its environment. Dependency injections frameworks like guice and Spring are one way to decouple interface from implementation in java applications.
  • Finally, an application that composes multiple modules that load external configuration and connect to external systems may benefit from a framework that manages the application lifecycle - or at least startup and shutdown.

Asynchronous Toolbox

The little-elements package includs an appContext module that provides an "asynchronous toolbox" framework for managing application configuration and bootstrap and decoupling interfaces from implementation.

Each module participating in the asynchronous toolbox provides zero or more asynchronous tool factories, and consumes a toolbox of zero or more tool factories from other modules. For example, modA in the example above (with typescript types) would look something like this.

// modA

import { Logging } from "../../@littleware/little-elements/common/logging.js";
import AppContext, { getTools, ConfigEntry } from "../../@littleware/little-elements/common/appContext.js";
import { SharedState } from "../../@littleware/little-elements/common/sharedState.js";
import { LazyProvider } from "../../@littleware/little-elements/common/provider.js";

/**
 * The configuration that classA consumes
 */
interface Config {
   foo: string;
   endpoint: string;
}

export const configKey = "config/myApp/modA";


/**
 * The tools that classA consumes - including config
 */
interface Tools {
    config: Config;
    log: Logger;
    state: SharedState;
}

class A {
    tools: Tools;

    constructor(tools:Tools) {
        this.tools = tools;
    }
    ...
}

AppContext.get().then(
    (cx) => {
        // register a default config. 
        // other modules can provide overrides.
        // the runtime also loads overrides from
        // registered configuration sources 
        // like remote or local json files
        cx.putDefaultConfig(configKey, { foo: "foo", endpont: "endpont" });

        // register a new provider
        cx.putProvider(
            classC.providerName,
            {
                config: configKey,
                log: Logger.providerName,
                state: StateManager.providerName,
            },
            async (toolBox) => {
                // the injected toolBox is full of asynchronous 
                // factories: async get():Tool
                // the `getTools()` helper below invokes get()
                // on every factory in the toolbox
                const tools: Tools = await getTools(toolBox);
                // the configuration factory
                // returns a ConfigEntry that includes 
                // both the defaults and overrides
                tools.config = { ...tools.config.defaults, ...tools.config.overrides };
                return LazyProvider(() => new C(tools as Tools));
            }
        );
    }
);

// Also export an asynchronous provider
// that modules not participating in the
// app context can use
export async function getC(): Promise<C> {
    return AppContext.get().then(
        (cx) => cx.getProvider(C.providerName),
    ).then(
        (provider: Provider<C>) => provider.get(),
    );
}

Then the main module for the application does something like this.

// mainMod

class Tools {
    c: classC;
}

function go(tools:Tools) { ... }

appContext.get().then(
    cx => cx.onStart( 
        // register a callback to run once the app
        // main module triggers start
        { c: classC.providerName },
        (tools:Tools) => go(tools),
).then(
    cx => cx.start();  // start the app
);

Example

The little-elements package leverages the asynchronous toolbox pattern in its authentication UX (signin, signout). The authn system relies on interactions between several modules.

The lw-auth-ui custom element builds on a lw-drop-down custom element to populate a drop-down menu with items that trigger internal navigation events. The drop-down component loads its menu contents (keys and navigation targets) as configuration from its asynchronous toolbox. The toolbox also contains helpers for internationalization and shared state. The UI listens for changes to the "logged in user" key in the shared state store.

The lw-auth-controller custom element consumes a toolbox that includes configuration, historyHelper, and sharedState tools. The controller listens for the internal navigation events triggered by the UI, then redirects the user's browser to the appropriate OIDC endpoint. The controller also polls the backend user-info endpoint to maintain the user data in the shared state store that the UI consumes.

Summary

The asynchronous toolbox is a flexible approach to manage asynchronous application bootstrap and configuration. It is adaptable to different situations like injecting dependencies into HTML custom elements. Unfortunately this pattern introduces boiler plate into the codebase. We hope to streamline the framework as it evolves over time.

Friday, July 10, 2020

little tools for API gateway

Introduction

An API gateway on AWS provides a scalable and serverless platform for deploying an HTTP API backed by a lambda function, served to clients via a cloudfront cdn, and metered per client request. This architecture trades the traditional costs and complexity of maintaining a cluster of servers to host an API for a new set of challenges. The people (developers, QA, operators, managers) running an API gateway project must become familiar with the technology stack, and adopt tools for integrating the gateway platform into a development and deployment process. We have begun to adopt a simple process using our own little tools.

API Overview

An API gateway deploys several AWS resources with the goal (usually) of making a lambda function executable via an HTTP API by linking the lambda to an API definition with a gateway stage. Let's look at the infrastructure from the bottom up.

  • First, we deploy a lambda function that implements the API gateway proxy integration protocol.
  • Next we write an openapi description of our HTTP API with some AWS extensions to describe how the API integrates with our lambda.
  • We create an ApiGateway RestAPI resource to hold our API definition, and an API deployment - which is an immutable copy of the API definition that links to one or more API stages (described below).
  • We now create beta and production API stages that each link to the deployment holding the API definition that we want to publish to the stage. The stage configures a regional endpoint for accessing the API.
  • Finally, we create a custom domain for our prod and beta API's, and a domain mapping that maps an API stage to a particular URL prefix under the domain (something like https://beta-my.domain/my-api1). We can setup mappings to link stages for different API's to a common domain (https://beta-my.domain/my-api1, https://beta-my.domain/my-api2, ...).
image/svg+xml Lambda API Deployment Stage DomainMapping Domain

Development Process

The usual way of working with cloudformation is to define a template for a particular kind of deployment that takes configuration parameters, then stamp out different stacks with the template. Unfortunately API gateway does not lend itself to that pattern in a straight forward way, because common gateway tasks require adding new resources to the template rather than simply changing parameters on existing resources.

Systems like the following take different approaches to address this problem.

We implement our own little tools that extend cloudformation templates with expressions (from the nunjucks template library) that can dynamically add resources to a stack based on variable values. The little stack tools consume a json stack definition with three main parts:

  • a reference to a clouformation template
  • values for the cf parameters defined in the template
  • values for the nunjucks variables leveraged in the template

If an optional openapi.yaml file is present, then its contents are also exposed as a nunjucks variable.

OIDC Client Example

Let's look in a little more detail at how we use the little stack and little lambda helpers to deploy one stack that uses this nunjucks-extended cloudformation template. The OIDC API fits into a larger appliction infrastructure with these components:

  • a cognito identity provider - auth.domain
  • a cloudfront distribution that serves static files for webapps from an S3 bucket - apps.domain
  • API's implemented as lambda functions behind an API gateway - api.domain

The OIDC client API is one of the API's under api.domain.

Deploying Code

The authcilent/code/ folder alongside the authclient/apiGateway.json cloudformation template holds the lambda deployment package that integrates the @littleware/little-authn node module with AWS lambda and API gateway. The beta API gateway stage (described below) executes the $LATEST lambda code, and the prod stage executes the lambda version referenced by the gateway_prod lambda alias.

The LambdaBucket and LambdaKey template parameters specify the location of the code package to deploy to the lambda function. We do something like this to deploy a code change in the beta stage:

  • update code/
  • upload the new code package with little lambda upload
  • update the default values for the LambdaBucket and LambdaKey parameters in the apiGateway.json template definition to point at the new code package
  • update the cloudformation stacks that use the template to deploy the new code: little stack update ./stackParams.json
  • the little stack events ./stackParams.json helper shows the latest cloudformation events to verify that the stack updated successfully

Configuration

The authclient/ cloudformation template expects the lambda function's configuration to be saved as json in an SSM parameter. The little ssm helper simplifies saving a new parameter, and the little-authn documentation has details about the expected configuration.

The lambda receives its configuration as an environment variable. The adapter code in code/index.js customizes the configuration depending on which stage is executing.

Publish code to prod

Publishing a new lambda version is one of the operations that relies upon our cloudformation template extensions to simplify the process of adding new resources (lambda versions) to a stack.

  • add a new version to the .Littleware.Variables.lambdaVersions list in the stackParams.json file that little stack consumes - this will deploy a new lambda version with the currently deployed lambda code package
  • point .Littleware.Variables.prodLambdaVersion at the new version to update the gateway_prod lambda alias
  • little stack update ./stackParams.json

Deploy a new version of the API

Updating the API definition is the other operation that relies upon our little stack cloudformation template extensions.

  • edit the openapi.yaml to make the api changes
  • run cloudformation to update the api definition: little stack update ./stackParams.json
  • add a new deployment to the .Littleware.Variables.gatewayDeployments array in stackParams.json, and point the beta domain stage at it by setting the .Littleware.Variables.betaDeployment (the prodDeployment variable updates the prod stage)
  • run cloudformation to update the beta stage: little stack update ./stackParams.json

Tests

The authclient/smokeTest.sh script runs an interactive test from the underlying @littleware/little-authn node module. The test walks the caller through a simple OIDC flow to verify the basic functionality of an API domain.

Summary

Transitioning to an API gateway and lambda based technology stack requires the people that run a project to learn new concepts, and adopt new tools for development and deployment processes. We have begun to adopt a simple process using our own little tools.

Saturday, February 29, 2020

Install local packages with npm link and npm config set prefix

In nodejs if you have a Bin package that depends on a Lib package, and before publishing a new version of Lib you want to test that local changes to Lib do not break Bin, then you can do something like this to link your local copy of Bin to your local copy of Lib:

cd Bin/ && npm link ../Lib/

A problem with this (at least on Ubuntu linux) is that npm link creates a soft link to Lib/ in npm's global folder (/usr/lib/node_modules/ on Ubuntu). So npm link both requires sudo and has a broader affect on the local system than what you want. A solution to this problem is to change npm's global prefix for your login to $HOME/.local, and add $HOME/.local/bin/ to your PATH:

npm config set prefix "$HOME/.local" --global

Now npm install --global bla installs bla under $HOME/.local/lib/node_modules/, and npm link ... sets up links there too, and testing Bin with Lib is a little less painful.

Thursday, February 06, 2020

simple expressjs logging middleware

There is a small trick to configure an expressjs middleware function to capture a request's run time and status. Here it is:

const serverLogger = (req, res, next) => {
    const startMs = Date.now();
    res.on("finish", () => {
        log.info(
            {
                req: {
                    path: req.path,
                },
                res: {
                    statusCode: res.statusCode,
                },
                timeMs: (Date.now() - startMs),
            },
        );
    });
    next();
};

app.use(serverLogger);

Thursday, November 21, 2019

Simple CICD with CodeBuild

CICD is awesome. It automates the tasks we want to perform whenever we push code to our repository:

  • test linting
  • audit third party dependencies for security
  • compile the code
  • run unit tests
  • publish artifacts

There are a lot of great options for implementing CICD too. On-prem systems like Jenkins and Team City, and SAAS systems like Travis, CircleCI, and Github Actions. For my personal projects I wanted a SAAS solution that triggers on github PR's and integrates with AWS, so I setup a CodeBuild process that I'm pretty happy with.

Codebuild Setup

Setting up codebuild is straight forward if you're comfortable with cloudformation.

  • save credentials
  • setup IAM roles for codebuild
  • setup a build for each github repository
  • add a build conifiguration to each github repository

save credentials

Codebuild can access credentials in AWS secrets manager for interacting with non-AWS systems like github and npm.org. The misc-stuff github repo has helpers for common operations tasks like managing secrets.

git clone https://github.com/frickjack/misc-stuff.git
export LITTLE_HOME="$(pwd)/misc-stuff"
alias little="bash '$(pwd)/misc-stuff/AWS/little.sh'"

little help secret
little secret create the/secret/key "$(cat secretValue.json)"

codebuild IAM setup

Codebuild needs IAM credentials to allocate build resources. This cloudformation template sets up a standard role that we'll pass to our builds:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
      "GithubToken": {
        "Type": "String",
        "Description": "arn of secretsmanager secret for access token",
        "ConstraintDescription": "secret arn",
        "AllowedPattern": "arn:.+"
      }
    },
    "Resources": {
      "CodeBuildRole": {
        "Type": "AWS::IAM::Role",
        "Properties": {
          "RoleName" : "littleCodeBuild",
          "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [{
              "Effect": "Allow",
              "Principal": { "Service": ["codebuild.amazonaws.com"] },
              "Action": ["sts:AssumeRole"]
            }]
          },
          "Policies": [{
            "PolicyName": "CodebuildPolicy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Sid": "CloudWatchLogsPolicy",
                  "Effect": "Allow",
                  "Action": [
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents"
                  ],
                  "Resource": [
                    "*"
                  ]
                },
                {
                  "Sid": "CodeCommitPolicy",
                  "Effect": "Allow",
                  "Action": [
                    "codecommit:GitPull"
                  ],
                  "Resource": [
                    "*"
                  ]
                },
                {
                  "Sid": "S3Policy",
                  "Effect": "Allow",
                  "Action": [
                    "s3:GetObject",
                    "s3:GetObjectVersion",
                    "s3:PutObject",
                    "s3:GetBucketAcl",
                    "s3:GetBucketLocation"
                  ],
                  "Resource": [
                    "*"
                  ]
                },
                {
                  "Sid": "SsmPolicy",
                  "Effect": "Allow",
                  "Action": [
                    "ssm:GetParameters",
                    "secretsmanager:Get*"
                  ],
                  "Resource": [
                    "*"
                  ]
                },
                {
                  "Sid": "ECRPolicy",
                  "Effect": "Allow",
                  "Action": [
                    "ecr:BatchCheckLayerAvailability",
                    "ecr:GetDownloadUrlForLayer",
                    "ecr:BatchGetImage",
                    "ecr:GetAuthorizationToken"

                  ],
                  "Resource": [
                    "*"
                  ]
                },
                {
                  "Sid": "CFPolicy",
                  "Effect": "Allow",
                  "Action": [
                    "cloudformation:ValidateTemplate"
                  ],
                  "Resource": [
                    "*"
                  ]
                },
                {
                  "Sid": "LambdaLayerPolicy",
                  "Effect": "Allow",
                  "Action": [
                    "lambda:PublishLayerVersion",
                    "lambda:Get*",
                    "lambda:List*",
                    "lambda:DeleteLayerVersion",
                    "iam:List*"
                  ],
                  "Resource": [
                    "*"
                  ]
                }
              ]
            }
          }
        ]}
      },
      "GithubCreds": {
        "Type" : "AWS::CodeBuild::SourceCredential",
        "Properties" : {
            "AuthType" : "PERSONAL_ACCESS_TOKEN",
            "ServerType" : "GITHUB",
            "Token": { "Fn::Join" : [ "", [ "{{resolve:secretsmanager:", { "Ref": "GithubToken" }, ":SecretString:token}}" ]] }
          }
      }
    },

    "Outputs": {
    }
}

We can use the little tool mentioned earlier to deploy the stack:

little stack create "$LITTLE_HOME/AWS/db/cloudformation/YourAccount/cell0/cicd/cicdIam/stackParams.json"

register a new build

This cloudformation template registers a new build that runs for pull requests and tag events on a github repository. In addition to the primary repo the build also pulls in a secondary support repository that hosts our automation scripts.

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters": {
      "ProjectName": {
        "Type": "String",
        "Description": "name of the build project - also tag",
        "ConstraintDescription": "should usually be the domain of the github repository"
      },
      "ServiceRole": {
        "Type": "String",
        "Description": "arn of IAM role for codebuild to assume",
        "ConstraintDescription": "IAM role arn",
        "AllowedPattern": "arn:.+"
      },
      "GithubRepo": {
        "Type": "String",
        "Description": "url of github source repo",
        "ConstraintDescription": "https://github.com/ repo url",
        "AllowedPattern": "https://github.com/.+"
      },
      "SupportRepo": {
        "Type": "String",
        "Description": "url of github secondary repo - default https://github.com/frickjack/misc-stuff.git",
        "Default": "https://github.com/frickjack/misc-stuff.git",
        "ConstraintDescription": "https://github.com/ repo url",
        "AllowedPattern": "https://github.com/.+"
      }
    },

    "Resources": {
        "CodeBuild": {
            "Type" : "AWS::CodeBuild::Project",
            "Properties" : {
                "Artifacts" : {
                  "Type": "NO_ARTIFACTS"
                },
                "BadgeEnabled" : true,
                "Description" : "build and test little-elements typescript project",
                "Environment" : {
                  "ComputeType" : "BUILD_GENERAL1_SMALL",
                  "EnvironmentVariables" : [ 
                    {
                      "Name" : "LITTLE_EXAMPLE",
                      "Type" : "PLAINTEXT",
                      "Value" : "ignore"
                    }
                   ],
                  "Image" : "aws/codebuild/standard:2.0",
                  "Type" : "LINUX_CONTAINER"
                },
                "Name" : { "Ref": "ProjectName" },
                "QueuedTimeoutInMinutes" : 30,
                "SecondaryArtifacts" : [],
                "ServiceRole" : { "Ref": "ServiceRole" },
                "Source" : {
                  "Type": "GITHUB",
                  "Location": { "Ref" : "GithubRepo" },
                  "GitCloneDepth": 2,
                  "ReportBuildStatus": true
                },
                "SecondarySources": [
                  {
                    "Type": "GITHUB",
                    "Location": { "Ref" : "SupportRepo" },
                    "GitCloneDepth": 1,
                    "SourceIdentifier": "HELPERS"
                  }
                ],
                "Tags": [
                    {
                        "Key": "org",
                        "Value": "applications"
                    },
                    {
                        "Key": "project",
                        "Value": { "Ref": "ProjectName" }
                    },
                    {
                        "Key": "stack",
                        "Value": "cell0"
                    },
                    {
                        "Key": "stage",
                        "Value": "dev"
                    },
                    {
                      "Key": "role",
                      "Value": "codebuild"
                    }
                ],
                "TimeoutInMinutes" : 10,
                "Triggers" : {
                  "FilterGroups" : [ 
                    [
                      {
                        "ExcludeMatchedPattern" : false,
                        "Pattern" : "PULL_REQUEST_CREATED, PULL_REQUEST_UPDATED, PULL_REQUEST_REOPENED, PULL_REQUEST_MERGED",
                        "Type" : "EVENT"
                      }
                    ],
                    [
                        {
                          "ExcludeMatchedPattern" : false,
                          "Pattern" : "PUSH",
                          "Type" : "EVENT"
                        },
                        {
                          "ExcludeMatchedPattern" : false,
                          "Pattern" : "^refs/tags/.*",
                          "Type" : "HEAD_REF"
                        }
                    ]
                   ],
                  "Webhook" : true
                }
            }
        }
    },

    "Outputs": {
    }
}

We can use the little tool mentioned earlier to deploy the stack:

little stack create "$LITTLE_HOME/AWS/db/cloudformation/YourAccount/cell0/cicd/nodeBuild/little-elements/stackParams.json"

configure the repository

Codebuild expects a buildspec.yaml file in the code repository to contain the commands for a build. This build file runs a typescript compile, unit tests, dependency security audit, and linting check. If the build trigger is a tagging event, then the build goes on to publish the build's assets as a lambda layer and https://npm.org package.

# see https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-codebuild-devicefarm.html
version: 0.2

env:
  variables:
    LITTLE_INTERACTIVE: "false"
  parameter-store:
    NPM_TOKEN: "/aws/reference/secretsmanager/applications/cicd/cell0/dev/npmjs-token"
phases:
  install:
    runtime-versions:
      nodejs: 10
    commands:
      - echo "Entered the install phase - jq already installed ..."
      #- apt-get update -y
      #- apt-get install -y jq
  pre_build:
    commands:
      - echo "HOME is $HOME, CODEBUILD_SRC_DIR is $CODEBUILD_SRC_DIR, CODEBUILD_SRC_DIR_HELPERS is $CODEBUILD_SRC_DIR_HELPERS, pwd is $(pwd)"
      - echo "//registry.npmjs.org/:_authToken=$(echo "$NPM_TOKEN" | jq -e -r .token)" > "$HOME/.npmrc"
      - mkdir -p "$HOME/.aws"; /bin/echo -e "[default]\nregion = us-east-2\noutput = json\ncredential_source = Ec2InstanceMetadata\n" | tee "$HOME/.aws/config"
      - npm ci
      - pip install yq --upgrade
  build:
    commands:
      - npm run build
      - npm run lint
      - npm audit --audit-level=high
      - npm run test
  post_build:
    commands:
      - echo "CODEBUILD_WEBHOOK_TRIGGER == $CODEBUILD_WEBHOOK_TRIGGER"
      # checkout a branch, so lambda publish goes there
      - git checkout -b "cicd-$CODEBUILD_WEBHOOK_TRIGGER"
      - BUILD_TYPE="$(echo $CODEBUILD_WEBHOOK_TRIGGER | awk -F / '{ print $1 }')"
      - echo "BUILD_TYPE is $BUILD_TYPE"
      # publish lambda layers for pr's and tags
      - if test "$BUILD_TYPE" = pr || test "$BUILD_TYPE" = tag; then bash "$CODEBUILD_SRC_DIR_HELPERS/AWS/little.sh" lambda update "$CODEBUILD_SRC_DIR"; fi
      # publish on tag events - see https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html
      - if test "$BUILD_TYPE" = tag; then npm publish . --tag cicd; fi

Saturday, November 16, 2019

Publish Web Content to S3

I host a little website at https://apps.frickjack.com by publishing static web files to an S3 bucket behind a cloudfront CDN. One way to improve the performance of an S3 web site is to configure each file's (s3 object) content encoding (gzip'ing the file if required), content type, and cache control at upload time (S3 natively specifies an Etag).

Here's a little script that makes that easy (also at https://github.com/frickjack/misc-stuff/blob/master/AWS/bin/s3web.sh), but rather than copy the script - you can install it like this:

git clone https://github.com/frickjack/misc-stuff.git
alias little="bash $(pwd)/Code/misc-stuff/AWS/little.sh"

little help s3web
little s3web publish localFolder s3Path [--dryrun]

Given a local source folder with html and other web content, then little s3web publish localFolder s3://destiation does the following:

  • html files are gzipped, and annotated with headers content-encoding gzip, mime-type, cache for one hour
  • css, js, svg, map, json, md files are gzipped, and annotated with content-encoding, mime-type, and cache for 1000 hours
  • png, jpg, wepb files are not gzipped, and annotated with mime-type and cache for 1000 hours

This caching policy assumes that html files may periodically change to reference new assets (javascript, css, etc), but the referenced assets are immutable. For example, https://apps.frickjack.com loads javascript and CSS assets from a versioned subpath, and every update to https://apps.frickjack.com publishes a new folder of assets.

#
# helpers for publishing web content to S3
#

source "${LITTLE_HOME}/lib/bash/utils.sh"

# lib ------------------------

#
# Determine mime type for a particular file name
#
# @param path
# @return echo mime type
#
s3webPath2ContentType() {
    local path="$1"
    shift
    case "$path" in
        *.html)
            echo "text/html; charset=utf-8"
            ;;
        *.json)
            echo "application/json; charset=utf-8"
            ;;
        *.js)
            echo "application/javascript; charset=utf-8"
            ;;
        *.mjs)
            echo "application/javascript; charset=utf-8"
            ;;
        *.css)
            echo "text/css; charset=utf-8"
            ;;
        *.svg)
            echo "image/svg+xml; charset=utf-8"
            ;;
        *.png)
            echo "image/png"
            ;;
        *.jpg)
            echo "image/jpeg"
            ;;
        *.webp)
            echo "image/webp"
            ;;
        *.txt)
            echo "text/plain; charset=utf-8"
            ;;
        *.md)
            echo "text/markdown; charset=utf-8"
            ;;
        *)
            echo "text/plain; charset=utf-8"
            ;;
    esac
}

#
# Get the npm package name from the `package.json`
# in the current directory
#
s3webPublish() {
    local srcFolder
    local destPrefix
    local dryRun=off

    if [[ $# -lt 2 ]]; then
        gen3_log_err "s3web publish takes 2 arguments"
        little help s3web
        return 1
    fi
    srcFolder="$1"
    shift
    destPrefix="$1"
    shift
    if [[ $# -gt 0 && "$1" =~ ^-*dryrun ]]; then
        shift
        dryRun=on
    fi
    if [[ ! -d "$srcFolder" ]]; then
        gen3_log_err "invalid source folder: $srcFolder"
        return 1
    fi
    if ! [[ "$destPrefix" =~ ^s3://.+ ]]; then
        gen3_log_err "destination prefix should start with s3:// : $destPrefix"
        return 1
    fi
    local filePath
    local ctype
    local encoding
    local cacheControl
    local errCount=0
    local gzTemp
    local commandList
    (
        commandList=()
        cd "$srcFolder"
        gzTemp="$(mktemp "$XDG_RUNTIME_DIR/gzTemp_XXXXXX")"
        find . -type f -print | while read -r filePath; do
            ctype="$(s3webPath2ContentType "$filePath")"
            encoding="gzip"
            cacheControl="max-age=3600000, public"
            destPath="${destPrefix%/}/${filePath#./}"
            if [[ "$ctype" =~ ^image/ && ! "$ctype" =~ ^image/svg ]]; then
                encoding=""
            fi
            if [[ "$ctype" =~ ^text/html || "$ctype" =~ ^application/json ]]; then
                cacheControl="max-age=3600, public"
            fi

            if [[ "$encoding" == "gzip" ]]; then
                gzip -c "$filePath" > "$gzTemp"
                commandList=(aws s3 cp "$gzTemp" "$destPath" --content-type "$ctype" --content-encoding "$encoding" --cache-control "$cacheControl")
            else
                commandList=(aws s3 cp "$filePath" "$destPath" --content-type "$ctype" --cache-control "$cacheControl")
            fi
            gen3_log_info "dryrun=$dryRun - ${commandList[@]}"
            if [[ "$dryRun" == "off" ]]; then
                "${commandList[@]}" 1>&2
            fi
            errCount=$((errCount + $?))
            if [[ -f "$gzTemp" ]]; then
                /bin/rm "$gzTemp"
            fi
        done
        [[ $errCount == 0 ]]
    )
}

# main ---------------------

# GEN3_SOURCE_ONLY indicates this file is being "source"'d
if [[ -z "${GEN3_SOURCE_ONLY}" ]]; then
    command="$1"
    shift

    case "$command" in
        "content-type")
            s3webPath2ContentType "$@"
            ;;
        "publish")
            s3webPublish "$@"
            ;;
        *)
            little help s3web
            ;;
    esac
fi

Friday, October 11, 2019

Executable Test Plans for BDD

There are many good libraries for developing automated software tests, but we also want to allow non developers to specify test plans and acceptance tests early in the development life cycle.

  • we want test plans and acceptance tests to help guide feature development
  • we want to track test plans as artifacts in git
  • we want to use Gherkin or some similar "living documentation" technology that allows us to link test automation with human readable test plans

Test automation frameworks like jasminejs can support manual tests via interactive helpers.

Test Plans and QA

A test plan is a living document that describes how a system should fulfill different use cases and how the system behaves under various scenarios. The system or feature design process should produce an initial test plan which guides the work of developers and testers, and helps ensure that the development process yields a system that matches its requirements. The test plan changes as the requirements for a system change.

Unit tests and integration tests are both important parts of the quality assurance process. During system development the developer team produce unit tests that run as part of the system's continuous integration process.

The QA team deploys each new build of the feature to a test environment, and caries out a test cycle that tests the behavior of the system against the system's test plan, and generates a report. The level of automation applied to a test cycle generally increases as a feature matures. A CICD system can run a fully automated test cycle that deploys each new build to a test environment, runs the test plan, and publish a report without human intervention.

Test Automation

A test plan should naturally evolve from manual checks carried out by the QA team to a fully automated test suite execute by a CICD pipeline. Test frameworks like jasminejs support automated tests, and can be leveraged to execute and generate reports for manual test cycles.

The testHelper.ts helpers allow us to write jasminejs test suites that present instructions to a human tester, then interactively collect the results of the test. This facility allows us to intermix manual and automated tests, and also support partially automated tests.

For example, one test in the testHelperSpec looks like this:

    it("allows interactive tests", ... ifInteractive(async () => {
        let result = await interactive("press enter at the prompt");
        expect(result.didPass).toBe(true);
        result = await interactive("enter \"y\" at the prompt");
        expect(result.didPass).toBe(true);
        result = await interactive("enter \"n\" at the prompt, then some explanation at the next prompt");
        expect(result.didPass).toBe(false);
        expect(result.details.length > 0).toBe(true);
    }, 3600000) as any,
    );

What's going on here?

  • ifInteractive tests the LITTLE_INTERACTIVE environment variable. If it's not false, then ifInteractive just returns its arguments (a lambda and timeout for jasmine's it method); otherwise ifInteractive returns a do-nothing test lambda for non-interactive environments
  • interactive prints the given instructions to the console for a human tester, and prompts the tester for whether the test passed or failed
  • there's also an isInteractive helper that just returns true if the LITTLE_INTERACTIVE environment variable is not false, and allows an automated test case to include optional manual elements

That's it! A simple test driven development process has a project manager specify acceptance tests in the feature definition, then a developer or tester translates those tests directly to code that fits into existing tooling for tracking changes, running tests, and generating reports. If you use jasminejs, then you can get started with the following code, otherwise it's straight forward to adapt the code to other frameworks. Good luck!


npm install @littleware/little-elements
const { interactive, ifInteractive } = require('@littleware/little-elements/commonjs/bin/testHelper.js');

Sunday, June 30, 2019

Hydrate an AWS Account

You just got your brand spankin' new AWS account, and your team is launching like a rocket into the cloud. How should you set this account thing up? Before playing with EC2 instances and S3 buckets and all the other toys at the application layer of the stack; you should figure out how you want to authenticate, authorize, and monitor users manipulating your base cloud infrastructure.

The following process describes one approach to inflate an AWS account before beginning application development. Although the steps taken here are only suitable for a single account setup for an individual or small team, the approach (setting up authentication, roles for authorization, monitoring, a simple budget, and basic alerts) generalizes for use with a multi-account AWS organization.

The Plan

Here's what we're going to do

  • login as the account's root user, and setup an IAM bootstrap user with admin privileges, so we can acquire credentials to run through a suite of account-bootstrap scripts.

  • run an accountBootstrap script to setup some basic infrastructure for deploying cloudformation stacks.

  • deploy a series of cloudformation stacks

    • IAM groups and roles for:
      • administrators - with write access to IAM and cloudtrail as well as operator permissions
      • operators - with write access to whatever services you want available for use in the account, except only read access to cloudtrail and IAM
      • developers - with access required to develop, test, deploy, and monitor applications that use infrastructure put into place by administrators and operators
    • an SNS topic for publishing alert notifications
    • a cloudtrail that logs updates to your account
    • a set of cloudwatch alarms that publish notifications to administrators (via the SNS topic described above) when various events occur:
      • IAM policy changes
      • root account access
      • budget limit exceeded
      • guard duty event
  • finally - setup an initial administrator account using the new infrastructure, and delete the temporary bootstrap user

Setup an AWS bootstrap user

Login to the root account, and do the following:

  • enable MFA on the root account
  • setup a bootstrap IAM user with MFA enabled and admin privileges:
{
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
    "Condition": {
        "Bool": {
            "aws:MultiFactorAuthPresent": "true"
        }
    }
}
  • download the access key pair for the new account, and setup ~/.aws/credentials and ~/.aws/config - ex:
[profile bootstrap-ohio]
region = us-east-2
output = json
mfa_serial = arn:aws:iam::123456789:mfa/bootstrap

Install software tools

These instructions assume your command shell has access to the following tools: bash, the jq json tool, git, and the aws cli.

  • download the cloudformation templates and helper scripts from our git repository:

    git clone https://github.com/frickjack/misc-stuff.git
  • add the little tool to your command path

    # assuming you're running a bash shell or similar
    alias little="bash $(pwd)/misc-stuff/AWS/little.sh"
    export LITTLE_HOME="$(pwd)/misc-stuff/AWS"
  • run the bootstrap script - it does the following:

    • deploys a block on s3 public access
    • creates an s3 bucket for cloudformation templates
    • sets a password policy for IAM users

ex:

export AWS_PROFILE="bootstrap-ohio"
little accountBootstrap
  • finally - prepare the inputs to our cloudformation stacks.
    • make a copy of the account-specific stack-parameters:
      cp AWS/misc-stuff/db/frickjack AWS/misc-stuff/db/YOUR-ACCOUNT
    • make whatever changes are appropriate for your account. For example - change the SNS notify e-mail in AWS/db/cloudformation/YourAccount/accountSetup/snsNotify.json
    • customize the cloudformation templates under misc-stuff/AWS/lib/cloudformation/ for your account. For example - the IamSetup.json template sets up an IAM policy that allows access to S3 and lambda and APIGateway API's, because I'm interested in those serverless technologies, but you may want to add permissions for accessing the EC2 and VPC API's.

What's the idea?

Before we start deploying stacks let's talk about the ideas we're implementing.

Authentication

The Right Way to Authenticate

First, authentication - how should a user prove who he is? AWS IAM has primitives for setting up users and groups, but that's not your best option for establishing a user's identity, because it's one more thing you need to maintain.

Instead of administering identity with users and groups in IAM under an AWS account - it's better to setup federated authentication with Google Apps or Office365 or some other identity provider that you already maintain with multi-factor auth and a password policy and all that other good stuff. If you don't already have an identity provider, then AWS has its own service, AWS SSO

While you're at it - you might setup an organization, because whatever you're doing it is bound to be wildly successful, and you'll wind up setting up multiple accounts for your galloping unicorn, and an organization helps simplify that administration.

Not the Right Way to Authenticate

If you don't already have an SSO identity provider, and you don't have someone to do it for you, then setting up an SSO and AWS federation and an AWS organization may seem like a lot of work just to manage a small team's access to AWS API's. So let's not do things the right way, but let's not be completely wrong either. We can emulate the right way.

  • require MFA for user authentication
  • enforce a password length and rotation policy
  • require rotation of user access keys
  • associate each user with at least one group
  • associate each group with an IAM role, so that a group member gains access to AWS API's by acquiring a temporary credentials via a multifactor-signed call to sts

This authentication setup ensures that access to AWS API's comes either from AWS managed temporary credentials passed directly to AWS resources like EC2 or labmda via something like the AWS metadata service, or a user must pass mutlifactor authentication to acquire a temporary token directly. Hopefully this authentication setup will protect our account from being compromised due to an exposed secret.

Authorization

Now that we have a mechanism to securely authenticate users and services that want to access AWS API's, how should we decide which privileges to grant different users? Our iamSetup cloudformation stack sets up three groups of users each associated with its own IAM role:

  • administrator
  • operator
  • developer

We restrict write permission to IAM policies to the administrators that are trained to enforce least privilege access. We want to restrict which users can disable cloudtrail, because there's no reason to do that.

The administrator group shares permissions to create other (non-IAM) AWS resources (whatever we want to allow in our account) with the group of operators. I'm not sure if it makes sense to have both an administrator group and an operator group, but one scenario might be that an administrator can setup IAM policies conditional on resource tags for a particular application or whatever, and an operator (maybe a devops specialist on a team) can then create and delete resources with the appropriate tags.

The developer group members cannot create new resources directly, but they do have permissions to deploy new versions of an application (udpate a lambda, or change the backend on an api gateway, or upgrade an EC2 AMI, or modify S3 objects - that kind of thing).

Finally - each application service has its own IAM role attached to its ECS container or EC2 instances or lambda or whatever. The administrator, operator, and developer roles should only be available to human users; each application's role grants the minimum privilege that service requires.

Tagging

A consistent tagging strategy allows everyone to easily determine the general purpose of a resource and who is responsible for the resource's allocation and billing. Something like this works, but there are many ways to do it.

"Tagging": {
        "TagSet": [
            {
                "Key": "org",
                "Value": "devops"
            },
            {
                "Key": "project",
                "Value": "infrastructure"
            },
            {
                "Key": "stack",
                "Value": "main"
            },
            {
                "Key": "stage",
                "Value": "prod"
            },
            {
              "Key": "role",
              "Value": "cloudformation-bucket"
            }
        ]
    }
  • org: which billing organization owns this resource
  • project: which project or application
  • stack: a project or application may have multiple deployments - for different clients or whatever
  • stage: a particular application stack may have multiple deployments for different stages of the development process (qa, staging, production)
  • role: what is the purpose of the resource?

Logs, Metrics, Monitoring, and Alerts

I was slow to understand what's up with cloudwatch and sns, but it's not that complicated. SNS is a pub-sub system - a client publishes something to a topic, and subscribers (lambda functions, e-mail, SQS, queues, ...) immediately receive whatever was published - no queueing or flow control - just a way to decouple systems.

Cloudwatch logs is a place to save log ("event") streams. Cloudwatch events lets listeners subscribe for notificates of various events from the AWS control plane. Cloudwatch metrics lets applications publish metrics like load, response time, number of requests, whatever. Cloudwatch alarms fire actions (lambda, SNS publication, ...) triggered by rules applied to metrics, events, and logs.

For example - our cloudformation stack sets up a notifications topic in SNS that our cloudwatch alarms publish to; and we setup alarms to send notifications when changes are made to IAM, or when the root account is accessed, or when an account approaches its budget limit, or when AWS guard duty detects something ... that kind of thing.

Deploy the stacks

Ok - let's do this thing. As the bootstrap user:

  • setup IAM groups and roles
little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/iamSetup.json"

Check if the stack came up successfully:

little stack events "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/iamSetup.json"

If not, then you can delete the stack, fix whatever the problem is, and try again:

little stack delete "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/iamSetup.json"

Similarly, you can modify a successfully deployed stack later:

little stack update "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/iamSetup.json"
  • setup a real user for yourself

The iamSetup stack creates administrator, operator, and developer groups and roles - where members of each group can assume the corresponding role. Use the AWS web console to create a user (with MFA, etc) for yourself via the console, and add the user to the administrator group. Download an access key for the new user, and configure your local ~/.aws/config, so that you can run commands with an administrator token - something like this:

[default]
region = us-east-1
output = json
mfa_serial = arn:aws:iam::012345678901:mfa/yourUser

[profile admin-ohio]
region = us-east-2
role_arn = arn:aws:iam::012345678901:role/littleware/account/user/littleAdmin
source_profile = default
mfa_serial = arn:aws:iam::012345678901:mfa/yourUser

With these credentials in place, you can run commands like the following. These tools will prompt you for an MFA code when necessary to acquire a fresh access token:

export AWS_PROFILE=admin-ohio
aws s3 ls
little env | grep AWS_

You can now deploy the following stacks as the new administrator user, and delete the bootstrap user.

  • setup cloudtrail

Update the cloudtrail parameters (AWS/db/cloudformation/YourAccount/accountSetup/cloudTrail.json) with a bucket name unique to your account - something like cloudtrail-management-$YourAccountName. You can retrieve the name of your account with aws iam list-account-aliases.

little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/cloudTrail.json"
  • setup an SNS topic

Remember to set the notification e-mail in the parameters before deploying the SNS stack; or customize the template with a subscriber for whatever notification channel (Slack, SMS, ...) you prefer. You can always add more subscribers to the topic later.

little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/snsNotify.json"
  • setup alarms

Update the stack parameter files for the alaram stacks (AWS/db/YourAccount/accountSetup/*Alarm.json) to reference the new SNS topic (aws sns list-topics) before deploying the following stacks:

little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/guardDuty.json"

little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/budgetAlarm.json"

little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/rootAccountAlarm.json"

little stack create "$LITTLE_HOME/db/cloudformation/YourAccountNameHere/accountSetup/iamAlarm.json"

Summary

We presented a simple way to secure API access in a new AWS account with authentication, authorization, a tagging strategy, a notification topic in SNS, basic cloudtrail logging, guard duty monitoring, and a few alarms. This simple setup is just a first step for a small team's journey into cloud security. A more sophisticated deployment would leverage AWS organizations and SSO. A larger organization may setup configuration rules, administrative accounts for centralized logging and alerts, and the journey goes on and on (we haven't even deployed an application yet).

Monday, December 24, 2018

EKS and AWS CNI IP Space management

AWS recently introduced its EKS managed kubernetes service which manages the kubernetes control plane (API and Etcd services), while the cluster owner administers the cluster's worker nodes in a VPC.

One of the features of EKS is the VPC CNI networking plugin that tightly integrates with AWS VPC networking, so that each kubernetes pod is assigned an IP address from the VPC CIDR range allocated to the worker node subnets. It's important to remember that EKS pods draw from the VPC IP pool when designing the VPC subnets for an EKS worker pool. When we transitioned our kubernetes infrastructure to EKS we initially allocated three /24 CIDR subnets for the EKS worker nodes. We assumed that design would allow a cluster of up to 768 nodes (256 nodes per subnet) where each node runs up to 5 pods, so up to 3840 pods with the calico CNI plugin that manages a separate IP space for pods in an overlay network; but with the AWS CNI the pods and nodes both draw IP addresses from the VPC pool of 768 IP addresses, so the cluster only supports about 625 pods on 125 nodes.

JWT Parser

I just published a new JSON web token (JWT) parser app under apps.frickjack.com. There's nothing special about this app - it's just another little vanilla-js plus custom-elements app that continues the evolution of the little-elements and little-apps code.