Showing posts with label scala. Show all posts
Showing posts with label scala. Show all posts

Sunday, July 04, 2021

gradle to sbt for scala3

Problem and Audience

Scala 3 is an overhaul of the guts of the scala language type system and compiler that was recently released (in mid 2021) for general use. Unfortunately, the scala plugin for gradle does not yet support the new scala-3 build chain, so we ported littleware to scala's sbt build tool.

Porting a gradle build to an sbt build is straight forward. Both gradle and sbt define a graph of tasks for building projects, and they define the task library (types of tasks) via third-party plugins. Both system's define the task instances in a project's build graph with a user-supplied build file written in a DSL that makes API calls against the runtime. For example, the littleware sub-project in both gradle (build.gradle) and sbt (build.sbt) are similar to each other:

sbt:

lazy val littleware = project
  .in(file("littleware"))
  .settings(
    name := "littleware",
    crossPaths := false,
    autoScalaLibrary := false,
    libraryDependencies ++= Seq(
      junit,
      guice,
      guava,
      "javax.mail" % "javax.mail-api" % "1.5.5",
      "javax" % "javaee-web-api" % "8.0",
      "org.apache.derby" % "derby" % "10.15.2.0",
      "org.apache.derby" % "derbyclient" % "10.15.2.0",
      "org.postgresql" % "postgresql" % "42.2.18",
      "mysql" % "mysql-connector-java" % "8.0.23",
      "org.javasimon" % "javasimon-core" % "4.2.0",
      "javax.activation" % "activation" % "1.1.1"
    ) ++ junitRunnerSet ++ log4jJsonSet,
  )

gradle:

project( ':littleware' ) {
    dependencies {
        implementation 'com.google.inject:guice:4.2.3:no_aop@jar'
        implementation 'junit:junit:4.13.2'
        implementation 'com.google.guava:guava:30.1-jre'
        compileOnly 'javax.mail:javax.mail-api:1.5.5'
        implementation 'javax:javaee-web-api:8.0'
        compileOnly 'org.apache.derby:derby:10.15.2.0'
        compileOnly 'org.apache.derby:derbyclient:10.15.2.0'
        compileOnly 'org.postgresql:postgresql:42.2.18'
        compileOnly 'mysql:mysql-connector-java:8.0.23'
        implementation 'org.javasimon:javasimon-core:4.2.0'
        runtimeOnly 'javax.activation:activation:1.1.1'
    }
}

I am glad that the port to SBT was straight forward, but I'm also annoyed that scala has its own build tool, sbt, rather than simply invest in gradle - which is widely used for building java, Android, and kotlin projects. It would be nice if I could just learn gradle, and use it to build scala (version 3) too. The same could be said for golang and dotnet - which also implement their own build tools rather than use gradle, but gradle depends on the jvm, so it makes more sense for languages in that ecosystem. There is a trade-off between building a custom tool chain that is finely tuned for a particular domain, or using a more generic system that has a large community of users. I expect gradle to support scala 3 in a few months anyway, so we will soon have the best of both worlds.

Summary

It was easy to port littleware's gradle build to sbt, since the two systems implement a task-graph design.

Monday, May 17, 2021

Simple Java/Scala Configuration Injection with Guice

Problem and Audience

One of the things every microservice needs is a mechanism for injecting configuration, so we developed a little json configuration helper for our littleware scala code that overlays a hierarchy of json configuration objects, and integrates with our module runtime and dependency-injection framework.

Configuration in Littleware

Littleware has a simple ServiceLoader based module runtime system that integrates with a guice dependency injection container. In practice what that means is that each java or scala jar includes a Module class that implements a simple callback interface for defining configuration injection bindings, and registering application event listeners (startup and shutdown). We have now augmented this platform with a json configuration helper that allows a module developer to provide configuration defaults on the classpath in the jar file, and the stack operator to override those defaults with a json file on an environment-defined search path or with json in an environment variable.

Here's how it works. The JsonConfigLoader provides a loadConfig helper that takes a key as an argument and returns a JsonObject (we use the gson json library).

The JsonConfigLoader also provides a bindKeys method that consumes a json object and a guice binder, converts the json to a list of (key, value) pairs, maps the values back to strings, and binds each key to its string value using guice's @Named binding facility.

So in the Module.scala (or .java) file described above, the module bootstrap code does something like this:

littleware.scala.JsonConfigLoader.loadConfig(CONFIG_KEY).map(
  {
    jsConfig =>
    littleware.scala.JsonConfigLoader.bindKeys(binder, jsConfig)
  }
)

Finally, a configuration provider can consume the bound configuration strings - like this:

@inject.Singleton()
    class ConfigProvider @inject.Inject() (
        @inject.name.Named("little.cloudmgr.sessionmgr.awsconfig") configStr:String,
        gs: gson.Gson
    ) extends inject.Provider[Config] {
        lazy val singleton: Config = {
            val js = gs.fromJson(configStr, classOf[gson.JsonObject])
            Config(
              js.getAsJsonPrimitive("oidcJwksUrl").getAsString(),
              Option(js.getAsJsonPrimitive("kmsSigningKey")).map({ _.getAsString() }),
              js.getAsJsonArray("kmsPublicKeys").asScala.map({ jsIt => jsIt.getAsJsonPrimitive().getAsString() }).toSet
            )
        }

        override def get():Config = singleton
    }

In the cloudmgr module above the configuration key is LITTLE_CLOUDMGR, so the config loader first loads littleware/config/LITTLE_CLOUDMGR.json off the classpath - which provides some developer defaults. The loader then searches the folders from the LITTLE_CONFIG_PATH environment (or system) variable until it finds a LITTLE_CLOUDMGR.json file, and it loads that, and does a shallow json merge. Finally, the config loader looks for a LITTLE_CLOUDMGR system (or environment) variable, and again merges the keys.

What does our configuration look like? We want to avoid collisions between binding keys from different modules, so the keys in a config json follow the java package reverse-dns pattern. Also, I like to have simple patterns that I can follow, so each service implementation in the module that requires configuration defines its own Config class and Provider[Config] that consumes a particular configuration key (that can be individually overriden via the configuration merge process described above). For example, the cloudmgr module has two service implementation, LocalKeySessionMgr and AwsSessionMgr, and the json configuration for the module looks like this:

{
    "little.cloudmgr.domain" : "test-cloud.frickjack.com",
    "little.cloudmgr.sessionmgr.type": "local",
    "little.cloudmgr.sessionmgr.localconfig": {
        "signingKey": { "kid": "testkey", "pem": "-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgs02I2exqJsdAoHef\n54/cjmlRvww903MKp0AOPqlRRXqhRANCAATWdeIowEmJ5lxpm7gE8GtvBnB1FBTI\nlcZHdD1FPM90oeEAraGGtnluYYEdPiJP3r29n3qFcGTgvqDAE49bc4om\n-----END PRIVATE KEY-----" }, 
        "verifyKeys": [ 
            { "kid": "testkey", "pem": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE1nXiKMBJieZcaZu4BPBrbwZwdRQU\nyJXGR3Q9RTzPdKHhAK2hhrZ5bmGBHT4iT969vZ96hXBk4L6gwBOPW3OKJg==\n-----END PUBLIC KEY-----" } 
        ],
        "oidcJwksUrl": "https://www.googleapis.com/oauth2/v3/certs" 
    }, 
    "little.cloudmgr.sessionmgr.awsconfig": {
        "kmsPublicKeys": [ 
            "alias/littleware/api/api-frickjack-com/sessMgrSigningKey", 
            "alias/littleware/api/api-frickjack-com/sessMgrOldKey" 
        ], 
        "kmsSigningKey": "alias/littleware/api/api-frickjack-com/sessMgrSigningKey", 
        "oidcJwksUrl": "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_860PcgyKN/.well-known/jwks.json"
    },
    "little.cloudmgr.sessionmgr.lambdaconfig": {
        "corsDomainWhiteList": [ ".frickjack.com" ],
        "cookieDomain": ".frickjack.com"
    }
}

Summary

We developed a little json configuration helper for our littleware scala code that overlays a hierarchy of json configuration objects, and integrates with our module runtime and dependency-injection framework.

Tuesday, April 13, 2021

AWS codebuild for scala, docker (ecr) CI

Problem and Audience

A continuous integration (CI) process that builds and tests our code, then publishes versioned deployable artifacts (docker images) is a prerequisite for deploying stable software services in the cloud. There are a wide variety of good, inexpensive CI services available, but we decided to build littleware's CI system on AWS codebuild, because it provides an easy to use serverless solution that supports the technology we build on (nodejs, java, scala, docker), and integrates well with AWS. It was straight forward for us to setup a codebuild CI process (buildspec.yml) for our little scala project given the tools we already have in place to deploy cloudformation stacks that define the codebuild project and ecr docker repository.

Overview

There were two steps to setting up our CI build: create the infrastructure, then debug and deploy the build script. The first step was easy, since we already have cloudformation templates for codebuild projects and ecr repositories that our little stack tool can deploy. For example, we deployed the codebuild project to build the littlware github repo by running:

little stack create ./stackParams.json

with this parameters file (stackParams.json):

{
    "StackName": "build-littleware",
    "Capabilities": [
        "CAPABILITY_NAMED_IAM"
    ],
    "TimeoutInMinutes": 10,
    "EnableTerminationProtection": true,
    "Parameters" : [
        {
            "ParameterKey": "PrivilegedMode",
            "ParameterValue": "true"
        },
        {
            "ParameterKey": "ProjectName",
            "ParameterValue": "cicd-littleware"
        },
        {
            "ParameterKey": "ServiceRole",
            "ParameterValue": "arn:aws:iam::027326493842:role/littleCodeBuild"
        },
        {
            "ParameterKey": "GithubRepo",
            "ParameterValue": "https://github.com/frickjack/littleware.git"
        }
    ],
    "Tags": [
            {
                "Key": "org",
                "Value": "applications"
            },
            {
                "Key": "project",
                "Value": "cicd-littleware"
            },
            {
                "Key": "stack",
                "Value": "frickjack.com"
            },
            {
                "Key": "stage",
                "Value": "dev"
            },
            {
              "Key": "role",
              "Value": "build"
            }
    ],
    "Littleware": {
        "TemplatePath": "lib/cloudformation/cicd/nodeBuild.json"
    }
}

With our infrastructure in place, we can add our build script to our github repository. There a few things to notice about our build script. First, the littleware git repo holds multiple interrelated projects - java and scala libraries and applications that build on top of them. We are currently interested in building and packaging the littleAudit/ folder (that will probably be renamed), so the build begins by moving to that folder:

  build:
    commands:
      - cd littleAudit

Next, we setup our codebuild project to run the build container in privileged mode, so our build can start a docker daemon, and build docker images:

phases:
  install:
    runtime-versions:
      # see https://github.com/aws/aws-codebuild-docker-images/blob/master/ubuntu/standard/5.0/Dockerfile
      java: corretto11
    commands:
      # see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-privilegedmode
      - nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --storage-driver=overlay &
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"

We use gradle to compile our code and run the unit test suite. The org.owasp.dependencycheck gradle plugin adds a dependencyCheckAnalyze task that checks our maven dependencies against public databases of known vulnerabilities:

  build:
    commands:
      - cd littleAudit
      - gradle build
      - gradle dependencyCheckAnalyze
      - docker build -t codebuild:frickjack .

Finally, our post-build command tags and pushes the docker image to an ecr repository. The tagging rules align with the lifecycle rules on the repository (described here and here).

  post_build:
    commands:
      - BUILD_TYPE="$(echo "$CODEBUILD_WEBHOOK_TRIGGER" | awk -F / '{ print $1 }')"
      - echo "BUILD_TYPE is $BUILD_TYPE"
      - |
        (
          little() {
              bash "$CODEBUILD_SRC_DIR_HELPERS/AWS/little.sh" "$@"
          }

          scanresult=""
          scan_in_progress() {
            local image
            image="$1"
            if ! shift; then
                echo "invalid scan image"
                exit 1
            fi
            local tag
            local repo
            tag="$(echo "$image" | awk -F : '{ print $2 }')"
            repo="$(echo "$image" | awk -F : '{ print $1 }' | cut -d / -f 2-)"
            scanresult="$(little ecr scanreport "$repo" "$tag")"
            test "$(echo "$scanresult" | jq -e -r .imageScanStatus.status)" = IN_PROGRESS
          }

          TAGSUFFIX="$(echo "$CODEBUILD_WEBHOOK_TRIGGER" | awk -F / '{ suff=$2; gsub(/[ @/]+/, "_", suff); print suff }')"
          LITTLE_REPO_NAME=little/session_mgr
          LITTLE_DOCKER_REG="$(little ecr registry)" || exit 1
          LITTLE_DOCKER_REPO="${LITTLE_DOCKER_REG}/${LITTLE_REPO_NAME}"

          little ecr login || exit 1
          if test "$BUILD_TYPE" = pr; then
            TAGNAME="${LITTLE_DOCKER_REPO}:gitpr_${TAGSUFFIX}"
            docker tag codebuild:frickjack "$TAGNAME"
            docker push "$TAGNAME"
          elif test "$BUILD_TYPE" = branch; then
            TAGNAME="${LITTLE_DOCKER_REPO}:gitbranch_${TAGSUFFIX}"
            docker tag codebuild:frickjack "$TAGNAME"
            docker push "$TAGNAME"
          elif test "$BUILD_TYPE" = tag \
            && (echo "$TAGSUFFIX" | grep -E '^[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}$' > /dev/null); then
            # semver tag
            TAGNAME="${LITTLE_DOCKER_REPO}:gitbranch_${TAGSUFFIX}"
            if ! docker tag codebuild:frickjack "$TAGNAME"; then
              echo "ERROR: failed to tag image with $TAGNAME"
              exit 1
            fi
            ...

If the CI build was triggered by a semver git tag, then it waits for the ecr image scan to complete successfully before tagging the docker image for production use:

       ...
          elif test "$BUILD_TYPE" = tag \
            && (echo "$TAGSUFFIX" | grep -E '^[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}$' > /dev/null); then
            # semver tag
            TAGNAME="${LITTLE_DOCKER_REPO}:gitbranch_${TAGSUFFIX}"
            if ! docker tag codebuild:frickjack "$TAGNAME"; then
              echo "ERROR: failed to tag image with $TAGNAME"
              exit 1
            fi
            # see https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_ImageScanStatus.html
            docker push "$TAGNAME" || exit 1
            count=0
            sleep 10

            while scan_in_progress "$TAGNAME" && test "$count" -lt 50; do
              echo "Waiting for security scan - sleep 10"
              count=$((count + 1))
              sleep 10
            done
            echo "Got image scan result: $scanresult"
            if ! test "$(echo "$scanresult" | jq -e -r .imageScanStatus.status)" = COMPLETE \
               || ! test "$(echo "$scanresult" | jq -e -r '.imageScanFindingsSummary.findingSeverityCounts.HIGH // 0')" = 0 \
               || ! test "$(echo "$scanresult" | jq -e -r '.imageScanFindingsSummary.findingSeverityCounts.CRITICAL // 0')" = 0; then
               echo "Image $TAGNAME failed security scan - bailing out"
               exit 1
            fi
            SEMVER="${LITTLE_DOCKER_REPO}:${TAGSUFFIX}"
            docker tag "$TAGNAME" "$SEMVER"
            docker push "$SEMVER"
          else
            echo "No docker publish for build: $BUILD_TYPE $TAGSUFFIX"
          fi

Summary

A continuous integration (CI) process that builds and tests our code, then publishes versioned deployable artifacts (docker images) is a prerequisite for deploying stable software services in the cloud. Our codebuild CI project builds and publishes the docker images that we will use to deploy our "little session manager" service as a lambda behind an API gateway (but we're still working on that).

Friday, April 02, 2021

Sign and Verify JWT with ES256

Problem and Audience

A developer of a system that uses json web tokens (JWT) to authenticate HTTP API requests needs to generate asymmetric cryptographic keys, load the keys into code, then use the keys to sign and validate tokens.

We are building a multi-tenant system that implements a hierarchy where each tenant (project) may enable one or more api's. An end user authenticates with the global system (OIDC authentication client) via a handshake with a Cognito identity provider, then acquires a short lived session token to interact with a particular api under a particular project (OIDC resource server).

It would be nice to simply implement the session token as a Cognito OIDC access token, but our system has a few requirements that push us to manage our own session tokens for now. First, each (project, api) pair is effectively an OIDC resource server in our model, and projects and api's are created dynamically, so managing the custom JWT claims with Cognito resource servers would be messy.

Second, we want to be able to support robot accounts at the project level, and a Cognito mechanism to easily provision robot accounts and tokens is not obvious to us. So we decided to manage our "session" tokens in the application, and rely on Cognito to federate identity providers for user authentication.

JWTS with ES256

I know very little about cryptography, authentication, and authorization; but fortunately people that know more than me share their knowledge online. Scott Brady's bLog gives a nice overview of JWT signing. We want to sign and verify JWTs in scala using the elliptic curve ES256 algorithm - which improves on RSA256 in a few ways, and is widely supported.

There are different ways to generate an elliptic curve sha-256 key pair, but EC keys saved to pem files are supported by multiple tools, and are easy to save to configuration stores like AWS SSM parameter store or secrets manager.

This bash function uses openssl to generate keys in pem files.

#
# Bash function to generate new ES256 key pair
#
newkey() {
    local kid=${1:-$(date +%Y%m)}
    local secretsFolder=$HOME/Secrets/littleAudit

    (
        mkdir -p "$secretsFolder"
        cd "$secretsFolder" || return 1
        if [[ ! -f ec256-key-${kid}.pem ]]; then
          openssl ecparam -genkey -name prime256v1 -noout -out ec256-key-${kid}.pem
        fi
        # convert the key to pkcs8 format
        openssl pkcs8 -topk8 -nocrypt -in ec256-key-${kid}.pem -out ec256-pkcs8-key-${kid}.pem
        # extract the public key
        openssl ec -in ec256-pkcs8-key-${kid}.pem -pubout -out ec256-pubkey-${kid}.pem
    )
}

Load the keys into code

Now that we have our keys - we need to load them into our scala application.

class KeyHelper @inject.Inject() (
  gs: gson.Gson, 
  ecKeyFactory:KeyHelper.EcKeyFactory, 
  rsaKeyFactory:KeyHelper.RsaKeyFactory
  ) {    
    /**
     * @return pem input with pem file prefix/suffix and empty space removed
     */
    def decodePem(pem:String): String = {
      pem.replaceAll(raw"-----[\w ]+-----", "").replaceAll("\\s+", "")
    }


    def loadPublicKey(kid:String, pemStr:String):SessionMgr.PublicKeyInfo = {
      val key = ecKeyFactory.generatePublic(decodePem(pemStr))
      SessionMgr.PublicKeyInfo(kid, "ES256", key)
    }


    def loadPrivateKey(kid:String, pemStr:String):SessionMgr.PrivateKeyInfo = {
      val key = ecKeyFactory.generatePrivate(decodePem(pemStr))
      SessionMgr.PrivateKeyInfo(kid, "ES256", key)
    }

    /**
     * Load keys from a jwks url like 
     *    https://www.googleapis.com/oauth2/v3/certs
     */
    def loadJwksKeys(jwksUrl:java.net.URL): Set[SessionMgr.PublicKeyInfo] = {
      val jwksStr = {
        val connection = jwksUrl.openConnection()
        connection.setRequestProperty("Accept-Charset", KeyHelper.utf8)
        connection.setRequestProperty("Accept", "application/json")
        val response = new java.io.BufferedReader(new java.io.InputStreamReader(connection.getInputStream(), KeyHelper.utf8))
        try {
            littleware.base.Whatever.get().readAll(response)
        } finally {
            response.close()
        }
      }

      gs.fromJson(jwksStr, classOf[gson.JsonObject]).getAsJsonArray("keys").asScala.map(
          { 
            json:gson.JsonElement =>
            val jsKeyInfo = json.getAsJsonObject()
            val kid = jsKeyInfo.getAsJsonPrimitive("kid").getAsString()
            val n = jsKeyInfo.getAsJsonPrimitive("n").getAsString()
            val e = jsKeyInfo.getAsJsonPrimitive("e").getAsString()
            val pubKey = rsaKeyFactory.generatePublic(n, e)
            SessionMgr.PublicKeyInfo(kid, "RSA256", pubKey)
          }
      ).toSet 
    }
}

object KeyHelper {
    val utf8 = "UTF-8"

    /**
     * Little injectable key factory hard wired to use X509 key spec for public key
     */
    class EcKeyFactory {
        val keyFactory = java.security.KeyFactory.getInstance("EC")
        val b64Decoder = java.util.Base64.getDecoder()

        def generatePublic(base64:String):ECPublicKey = {
            val bytes = b64Decoder.decode(base64.getBytes(utf8))
            val spec = new X509EncodedKeySpec(bytes)

            keyFactory.generatePublic(spec).asInstanceOf[ECPublicKey]
        }

        def generatePrivate(base64:String):ECPrivateKey = {
            val bytes = b64Decoder.decode(base64.getBytes(utf8))
            val spec = new PKCS8EncodedKeySpec(bytes)

            keyFactory.generatePrivate(spec).asInstanceOf[ECPrivateKey]
       }
    }

    /**
     * Little injectable key factory hard wired for RSA jwks decoding
     * See: https://github.com/auth0/jwks-rsa-java/blob/master/src/main/java/com/auth0/jwk/Jwk.java
     */
    class RsaKeyFactory {
        private val keyFactory = java.security.KeyFactory.getInstance("RSA")
        private val b64Decoder = java.util.Base64.getUrlDecoder()

        def generatePublic(n:String, e:String):RSAPublicKey = {
            val modulus = new java.math.BigInteger(1, b64Decoder.decode(n))
            val exponent = new java.math.BigInteger(1, b64Decoder.decode(e))
            keyFactory.generatePublic(new RSAPublicKeySpec(modulus, exponent)).asInstanceOf[RSAPublicKey]
        }
    }
}

Sign and verify JWTs

Now that we have loaded our keys, we can use them to sign and verify JWTs. Okta has published open source code for working with JWTs , Auth0 has published open source code for working with JWK, and AWS KMS supports elliptic curve digital signing algorithms with asymmetric keys.

import com.google.inject
// see https://github.com/jwtk/jjwt#java-jwt-json-web-token-for-java-and-android
import io.{jsonwebtoken => jwt}
import java.security.{ Key, PublicKey }
import java.util.UUID
import scala.util.Try

import littleware.cloudmgr.service.SessionMgr
import littleware.cloudmgr.service.SessionMgr.InvalidTokenException
import littleware.cloudmgr.service.littleModule
import littleware.cloudutil.{ LRN, Session }

/**
 * @param signingKey for signing new session tokens
 * @param verifyKeys for verifying the signature of session tokens
 */
@inject.ProvidedBy(classOf[LocalKeySessionMgr.Provider])
@inject.Singleton()
class LocalKeySessionMgr (
    signingKey: Option[SessionMgr.PrivateKeyInfo],
    sessionKeys: Set[SessionMgr.PublicKeyInfo],
    oidcKeys: Set[SessionMgr.PublicKeyInfo],
    issuer:String,
    sessionFactory:inject.Provider[Session.Builder]
    ) extends SessionMgr {

    val resolver = new jwt.SigningKeyResolverAdapter() {
        override def resolveSigningKey(jwsHeader:jwt.JwsHeader[T] forSome { type T <: jwt.JwsHeader[T] }, claims:jwt.Claims):java.security.Key = {
            val kid = jwsHeader.getKeyId()
            (
                {
                    if (claims.getIssuer() == issuer) {
                        sessionKeys
                    } else {
                        oidcKeys
                    }
                }
            ).find(
                { it => it.kid == kid }
            ).map(
                { _.pubKey }
            ) getOrElse {
                throw new SessionMgr.InvalidTokenException(s"invalid auth kid ${kid}")
            }
        }
    }

    ...

    def jwsToClaims(jwsIdToken:String):Try[jwt.Claims] = Try(
        { 
            jwt.Jwts.parserBuilder(
            ).setSigningKeyResolver(resolver
            ).build(
            ).parseClaimsJws(jwsIdToken
            ).getBody()
        }
    ).flatMap( claims => Try( {
                    Seq("email", jwt.Claims.EXPIRATION, jwt.Claims.ISSUER, jwt.Claims.ISSUED_AT, jwt.Claims.AUDIENCE).foreach({
                        key =>
                        if(claims.get(key) == null) {
                            throw new InvalidTokenException(s"missing ${key} claim")
                        }
                    })
                    claims
                }
            )
    ).flatMap(
        claims => Try(
            {
                if (claims.getExpiration().before(new java.util.Date())) {
                    throw new InvalidTokenException(s"auth token expired: ${claims.getExpiration()}")
                }
                claims
            }
        )
    )


    def sessionToJws(session:Session):String = {
        val signingInfo = signingKey getOrElse { throw new UnsupportedOperationException("signing key not available") }
        jwt.Jwts.builder(
        ).setHeaderParam(jwt.JwsHeader.KEY_ID, signingInfo.kid
        ).setClaims(SessionMgr.sessionToClaims(session)
        ).signWith(signingInfo.privKey
        ).compact()
    }

    def jwsToSession(jws:String):Try[Session] = jwsToClaims(jws
        ) map { claims => SessionMgr.claimsToSession(claims) }

    ...
}

This code is all online under this github repo, but is in a state of flux.

Summary

To sign and verify JWTs we need to generate keys, load the keys into code, and use the keys to sign and verify tokens. We plan to add support for token signing with AWS KMS soon.

Saturday, March 20, 2021

A little framework for scala builders

Many developers use immutable data structures in application code, and leverage the builder pattern to construct the application's initial state, and progress to new versions of the state in response to user inputs and other side effects. The scala programming language supports immutable data structures well, but does not provide a native implementation of the builder pattern. The following describes a simple but useful scala builder framework that leverages re-usable data-validation lambda functions.

A Scala Framework for Builders

The following are typical examples of an immutable case class in scala:

/**
 * Little resource name URI:
 * "lrn://${cloud}/${api}/${project}/${resourceType}/${drawer}/${path}"
 */
trait LRN {
    val cloud: String
    val api: String
    val projectId: UUID
    val resourceType: String
    val drawer: String
    val path: String
}

/**
 * Path based sharing - denormalized data
 */
case class LRPath(
    cloud: String,
    api: String,
    projectId: UUID,
    resourceType: String,
    drawer: String,
    path: String
) extends LRN {
}

/**
 * Id based sharing - normalized data
 */
case class LRId(
    cloud: String,
    api: String,
    projectId: UUID,
    resourceType: String,
    resourceId: UUID
) extends LRN {
    override val drawer = ":"
    val path = resourceId.toString()
}

Builders like the following simplify object construction and validation (compared to passing all the property values to the constructor).

object LRN {
    val zeroId:UUID = UUID.fromString("00000000-0000-0000-0000-000000000000")

    trait Builder[T <: LRN] extends PropertyBuilder[T] {
        val cloud = new Property("") withName "cloud" withValidator dnsValidator
        val api = new Property("") withName "api" withValidator LRN.apiValidator
        val projectId = new Property[UUID](null) withName "projectId" withValidator notNullValidator
        val resourceType = new Property("") withName "resourceType" withValidator LRN.resourceTypeValidator
        val path = new Property("") withName "path" withValidator pathValidator

        def copy(lrn:T):this.type = this.projectId(lrn.projectId).api(lrn.api
            ).cloud(lrn.cloud).resourceType(lrn.resourceType
            ).path(lrn.path)

        def fromSession(session:Session): this.type = this.cloud(session.lrp.cloud
            ).api(session.api
            ).projectId(session.projectId
            )
    }

    class LRPathBuilder extends Builder[LRPath] {        
        val drawer = new Property("") withName "drawer" withValidator drawerValidator

        override def copy(other:LRPath) = super.copy(other).drawer(other.drawer)

        def build():LRPath = {
            validate()
            LRPath(cloud(), api(), projectId(), resourceType(), drawer(), path())
        }
    }

    class LRIdBuilder extends Builder[LRId] {
        def build():LRId = {
            validate()
            LRId(cloud(), api(), projectId(), resourceType(), UUID.fromString(path()))
        }
    }

    def apiValidator = rxValidator(raw"[a-z][a-z0-9-]+".r)(_, _)

    def drawerValidator(value:String, name:String) = rxValidator(raw"([\w-_.*]+:)*[\w-_.*]+".r)(value, name) orElse {
        if (value.length > 1000) {
            Some(s"${name} is too long: ${value}")
        } else {
            None
        }
    }

    def pathValidator(value:String, name:String) = pathLikeValidator(value, name) orElse {
        if (value.length > 1000) {
            Some(s"${name} is too long: ${value}")
        } else {
            None
        }
    }

    def resourceTypeValidator = rxValidator(raw"[a-z][a-z0-9-]{1,20}".r)(_, _)

    // ...
}

This builder implementation does not leverage the type system to detect construction errors at compile time (this blog shows an approach with phantom types), but it is composable in a straight forward way. A couple fun things about this implementation are that it leverages the builder pattern to define the properties in a builder (new Property... withName ... withValidator ...), and the setters on the nested property class return the parent Builder type, so we can write code like this:

    @Test
    def testLRNBuilder() = try {
        val builder = builderProvider.get(
        ).cloud("test.cloud"
        ).api("testapi"
        ).drawer("testdrawer"
        ).projectId(LRN.zeroId
        ).resourceType("testrt"
        ).path("*")

        val lrn = builder.build()
        assertTrue(s"api equal: ${lrn.api} ?= ${builder.api()}", lrn.api == builder.api())
    } catch basicHandler

Unfortunately, the code (in https://github.com/frickjack/littleware under littleAudit/ and littleScala/) is in a state of flux, but the base PropertyBuilder can be copied into another code base - something like this:

/**
 * Extends Validator with support for some scala types
 */
trait LittleValidator extends Validator {
  @throws(classOf[ValidationException])
  override def validate():Unit = {
    val errors = checkSanity()
    if ( ! errors.isEmpty ) {
      throw new ValidationException(
        errors.foldLeft(new StringBuilder)( (sb,error) => { sb.append( error ).append( littleware.base.Whatever.NEWLINE ) } ).toString
      )
    }
  }


  /**
   * Same as checkIfValid, just scala-friendly return type
   */
  def checkSanity():Iterable[String]
}

trait PropertyBuilder[B] extends LittleValidator {
  builder =>
  import PropertyBuilder._
  type BuilderType = this.type

  import scala.collection.mutable.Buffer

  /**
   * List of properties allocated under this class - used by isReady() below -
   * use with caution.
   */
  protected val props:Buffer[Property[_]] = Buffer.empty

  /**
   * Default implementation is props.flatMap( _.checkSanity ) 
   */
  def checkSanity():Seq[String] = props.toSeq.flatMap( _.checkSanity() )

  override def toString():String = props.mkString(",")   

  def copy(value:B):BuilderType
  def build():B

  /**
   * Typical property, so build has things like
   *     val a = new Property(-1) withName "a" withValidator { x => ... }
   *
   * Note: this type is intertwined with PropertyBuilder - don't
   *    try to pull it out of a being a subclass - turns into a mess
   */
  class Property[T](
      var value:T
    ) extends LittleValidator {    
    type Validator = (T,String) => Option[String]

    def apply():T = value

    var name:String = "prop" + builder.props.size
    var validator:Validator  = (_, _) => None

    override def checkSanity() = validator(this.value, this.name)
    def withValidator(v:Validator):this.type = {
      validator = v
      this
    }

    def withName(v:String):this.type = {
      this.name = v
      this
    }

    override def toString():String = "" + name + "=" + value + " (" + checkSanity().mkString(",") + ")"

    /** Chainable assignment */
    def apply(v:T):BuilderType = { value = v; builder }

    builder.props += this 
  }

  /**
   * Property accepts multiple values
   */
  class BufferProperty[T] extends Property[Buffer[T]](Buffer.empty) {
    def add( v:T ):BuilderType = { value += v; builder }
    def addAll( v:Iterable[T] ):BuilderType = { value ++= v; builder }
    def clear():BuilderType = { value.clear(); builder; }

    def withMemberValidator(memberValidator:(T,String) => Option[String]):this.type =
      withValidator(
        (buff, propName) => buff.view.flatMap({ it => memberValidator(it, propName) }).headOption
      )  
  }  

  class OptionProperty[T] extends Property[Option[T]](None) {
    def set(v:T):BuilderType = { value = Option(v); builder }

    def withMemberValidator(memberValidator:(T,String) => Option[String]):this.type =
      withValidator(
        (option, propName) => option.flatMap({ it => memberValidator(it, propName) })
      )  

  }
}

object PropertyBuilder {  
  /** littleware.scala.Messages resource bundle */
  val rb = java.util.ResourceBundle.getBundle( "littleware.scala.Messages" )

  def rxValidator(rx:Regex)(value:String, name:String):Option[String] = {
    if (null == value || !rx.matches(value)) {
      Some(s"${name}: ${value} !~ ${rx}")
    } else {
      None
    }
  }

  def notNullValidator(value:AnyRef, name:String):Option[String] = {
    if (null == value) {
      Some(s"${name}: is null")
    } else {
      None
    }
  }

  def positiveIntValidator(value:Int, name:String):Option[String] = {
    if (value <= 0) {
      Some(s"${name}: is not positive")
    } else {
      None
    }
  }

  def positiveLongValidator(value:Long, name:String):Option[String] = {
    if (value <= 0) {
      Some(s"${name}: is not positive")
    } else {
      None
    }
  }

  def dnsValidator = rxValidator(raw"([\w-]{1,40}\.){0,10}[\w-]{1,40}".r)(_, _)
  def emailValidator = rxValidator(raw"[\w-_]{1,20}@\w[\w-.]{1,20}".r)(_, _)
  def pathLikeValidator = rxValidator(raw"([\w-:_.*]{1,255}/){0,20}[\w-:_.*]{1,255}".r)(_, _)


}

Saturday, May 04, 2013

scala REPL classpath hack

While writing code I often want to take a quick look at the behavior of some class or module, and might wind up writing a short test program just to verify that some method does what I expect or that I'm obeying some DSL's state machine or whatever. One of the great things about Scala (and groovy, javascript, clojure, ...) is its REPL, which allows me to test little blocks of code without the hassle of compiling a test file.

Unfortunately, Scala's REPL launch scripts (scala.bat, scala.sh) do not provide command-line options for manipulating the REPL's classpath, so the REPL can't see the jars IVY assembles for my project ( javasimon, joda-time, guava, gson, guice, ... ).

Anyway - it's easy to just copy %SCALA_HOME%/bin/scala.bat (or ${SCALA_HOME}/scala if not on Windows - whatever), and modify it, so the REPL runs with the same classpath as your project, and at the same time set the heap size, logging config-file system property, and whatever else is needed. For example, here's the shell.bat launch-script from a project I was playing with recently.

@echo off

if not defined JAVACMD set JAVACMD=java.exe
if defined JAVA_HOME set JAVACMD="%JAVA_HOME%\bin\java.exe"

@rem echo "mojo:  .\runTestCase.bat 2>&1 | out-file output.txt utf8"


%JAVACMD% -Xmx1024m -Xms1024m 
   "-Djava.util.logging.config.file=%~dps0logging.properties"  
   -cp "%SCALA_HOME%\lib\*;%~dps0\lib\*;%~dps0\*;." 
   scala.tools.nsc.MainGenericRunner -usejavacp %*

The "mojo" line is a reminder to me that
bla 2>&1 | out-file output.txt utf8
is the Windows powersell equivalent of
bla > output.txt 2>&1
in bash shell - something like that. I'm terrible in bash, and worse in powershell, but I try to at least be able to send output to a file, grep, and run commands in a loop ... ugh!

Anyway - once the classpath is all set, then it's REPL playtime!

> .\cli\shell.bat
Welcome to Scala version 2.10.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_0
9).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.joda.{time => jtime}
import org.joda.{time=>jtime}
                                                    ^

scala> val t = jtime.DateTime.parse( jtime.DateTime.now().toString() )
t: org.joda.time.DateTime = 2013-05-03T16:34:48.496-05:00

scala> import com.google.common.{io => gio}
import com.google.common.{io=>gio}


scala> val UTF8 = java.nio.charset.Charset.forName( "UTF-8" )
UTF8: java.nio.charset.Charset = UTF-8

scala> gio.Files.write( jtime.DateTime.now().toString(), 
         new java.io.File( "bla.txt" ), UTF8 )

scala> val t = jtime.DateTime.parse( 
      gio.Files.toString( new java.io.File( "bla.txt" ), UTF8 ) 
    )
t: org.joda.time.DateTime = 2013-05-03T16:37:35.561-05:00

scala>

Tuesday, April 16, 2013

paging s3 listObjects as a scala stream

One of Scala's fun features is its lazy list Stream class. I guess for a Haskell programmer a lazy list is not a big deal, but it's cool for the rest of us java, ruby, python, javascript, C#, ... programmers.

Anyway - I plan to host some web content on S3, so I've been writing some scala code using S3's java API to automate some tasks. One of the patterns S3 employs is to "page" large result sets, and scala streams provide a natural way to load the pages on demand - something like this:

      lazy val s3Listing:Stream[s3.model.ObjectListing] = 
          s3Client.listObjects( 
            new s3.model.ListObjectsRequest( folder.getHost, queryFolderPath, null, "/", null ) 
          ) #:: s3Listing.takeWhile( _.isTruncated ).map( (part) => s3Client.listNextBatchOfObjects( part ) )

Pretty cool, but, unfortunately, Stream has some quirk where its flatMap method can easily overflow the stack. For example - the following code for assembling the directories in a file system explodes:

          lazy val folderStream:Stream[jio.File] = new java.io.File( folder.path.getPath ).getCanonicalFile #:: 
            folderStream.flatMap( (d) => { d.listFiles().filter( (sd) => sd.isDirectory ) } )

We can construct a stream, and avoid calling flatMap with a recursive method that manages its own stack.

          def depthFirst( stack:List[jio.File] ):Stream[jio.File] = {
            stack match {
              case head :: tail => {
                  head #:: depthFirst( tail ++ head.listFiles.filter( _.isDirectory ) )
              }
              case Nil => Stream.empty
            }
          }

          depthFirst( List( root ) )

Saturday, January 05, 2013

scala 2.10 netbeans ant fix

I just upgraded the scala install on my laptop from 2.9.x to the latest 2.10.0 release candidate, and was depressed to discover that broke my crazy netbeans derived ant build scripts (ugh!).

> ant clean
Buildfile: C:\Users\pasquini\Documents\Code\littleware\WIP\littleware\webapp\lit
tleId\littleId\build.xml

config-check:

download-ivy:

install-ivy:

resolveIfNecessary:

-pre-init:

-init-private:

BUILD FAILED
C:\Users\pasquini\Documents\Code\littleware\WIP\littleware\webapp\littleId\littl
eId\nbproject\build-impl.xml:50: The following error occurred while executing th
is line:
jar:file:/C:/Program%20Files/Scala/scala-2.10.0-RC5/lib/scala-compiler.jar!/scal
a/tools/ant/antlib.xml:5: taskdef A class needed by class scala.tools.ant.FastSc
alac cannot be found: scala/reflect/internal/settings/MutableSettings$SettingVal
ue
 using the classloader AntClassLoader[C:\Program Files\Scala\scala-2.10.0-RC5\li
b\scala-compiler.jar;C:\Program Files\Scala\scala-2.10.0-RC5\lib\scala-library.j
ar]

Total time: 2 seconds

Fortunately - the fix was easy. Scala 2.10.x pushes some classes that its ant task depends on to a new scala-reflect jar file, so I added that to the classpath in one of the xml blocks in nbproject/build-impl.xml, and I was back in business:

        <property name="scala.compiler" value="${scala.home}/lib/scala-compiler.jar"/>
        <property name="scala.library" value="${scala.home}/lib/scala-library.jar"/>
        <property name="scala.lib" value="${scala.home}/lib"/>
        <taskdef resource="scala/tools/ant/antlib.xml">
            <classpath>
                <pathelement location="${scala.compiler}"/>
                <pathelement location="${scala.library}"/>
                <pathelement location="${scala.lib}/scala-reflect.jar"/>
            </classpath>
        </taskdef>

Anyway - scala 2.10 looks to be a great release. A lot of people are excited about the "macro" system, but I've been working with the new akka actors, and also look forward to playing around with the futures and promises APIs.

Saturday, August 25, 2012

scala companion package

I wrote a scala trait the other day with a "companion package" instead of the usual "companion object". I was pretty happy with how the code came together, so thought I'd write a quick post to my poor neglected bLog.

I often write a scala trait that defines an interface to some tool that manages access to a data repository or manipulates data in some way. I usually define the interface traits to different tools provided by some module "bla" in a "bla.controller" sub-package. I define bla's data model in a "model" sub-package, and classes that simply support a tool's interface in the tool's companion object, so something like this:

package bla
package controller

trait Tool {
    def get( id:String ):model.Data
    def search( query:String ):Tool.SearchResult
    ...
}

object Tool {
    case class SearchResult( query:String, matches:Iterable[model.IndexEntry] ) {}

    object SomeEnum extends Enumeration {
       val A, B, C = Value
    }

    trait Builder {
       var creds:Credentials = null
       def creds( value:Credentials ):this.type = {creds = value; this }
       
       var endPoint:URL = new URL( "http://default.end.point" )
       def endPoint( value:URL ):this.type = { endPoint = value; this }

       ...
       def build():Tool
    }

    /** Factory necessary if not running in IOC container - ugh */
    object Factory extends Supplier[Builder] {
        def get():Builder = internal.SimpleTool.Factory.get()
    }
}

Anyway - that pattern works great most of the time, but the other day I was working on a tool with several supporting types in the companion object, and the Tool.scala file was growing fatter than I like. Fortunately - it turns out we can accomplish the same semantics as a companion object with a companion package like this:

package bla
package controller

trait Tool { ... }

package Tool {
    // ...
    object Factory extends Supplier[Builder] { ... }
}

The companion package approach has the advantage that we can split out some of the types defined in the companion package to their own files, so we could have a bla/DataHelper/Factory.scala like this:

package bla
package controller
package Tool

object Factory extends Supplier[Tool] { ... }

The companion package can define "static" constants in a package object, and we can move the tool-specific implementation classes to its own bla.controller.Tool.internal package rather than the shared bla.controller.internal package.

Finally - changing companion object code to the companion package pattern does not require changes to client code (code that uses Tool) - just a recompile.

Anyway - I like the way things fit together with the companion package pattern.