EKS and AWS CNI IP Space management

AWS recently introduced its EKS managed kubernetes service which manages the kubernetes control plane (API and Etcd services), while the cluster owner administers the cluster's worker nodes in a VPC.

One of the features of EKS is the VPC CNI networking plugin that tightly integrates with AWS VPC networking, so that each kubernetes pod is assigned an IP address from the VPC CIDR range allocated to the worker node subnets. It's important to remember that EKS pods draw from the VPC IP pool when designing the VPC subnets for an EKS worker pool. When we transitioned our kubernetes infrastructure to EKS we initially allocated three /24 CIDR subnets for the EKS worker nodes. We assumed that design would allow a cluster of up to 768 nodes (256 nodes per subnet) where each node runs up to 5 pods, so up to 3840 pods with the calico CNI plugin that manages a separate IP space for pods in an overlay network; but with the AWS CNI the pods and nodes both draw IP addresses from the VPC pool of 768 IP addresses, so the cluster only supports about 625 pods on 125 nodes.

JWT Parser

I just published a new JSON web token (JWT) parser app under apps.frickjack.com. There's nothing special about this app - it's just another little vanilla-js plus custom-elements app that continues the evolution of the little-elements and little-apps code.

little-elements webapp shell

I recently updated apps.frickjack.com to use unbundled javascript modules to load custom elements and style rules. It worked great, except I soon noticed that initial page load would present an un-styled, no-custom-element rendering of the page until the javascript modules finished loading asynchronously... ugh.

I worked around the problem by implementing a simple application shell. The html pages under https://apps.frickjack.com now extend a compile-time nunjucks template, basicShell.html.njk (npm install @littleware/little-elements). The shell hides the page content with a style rule, maybe loads the webcomponentsjs polyfill, and renders a "Loading ..." message if the main.js javascript module doesn't load fast enough. I also extended the main javascript module on each page to signal the shell when it has finished rendering the initial page content. For example - indexMain.js for https://apps.frickjack.com/index.html looks like this:

import './headerSimple/headerSimple.js';

if ( window['littleShell'] ) {
    window['littleShell'].clear();
}

The index.html nunjucks template looks like this:

{% extends "little-elements/lib/styleGuide/shell/basicShell.html.njk" %}

{% block head %}
<title>frickjack.com</title>
{% endblock %}

{% block content %}
    <lw-header-simple title="Home">
    </lw-header-simple>
    ...
   <script src="{{ jsroot }}/@littleware/little-apps/lib/indexMain.js" type="module"></script>
{% endblock %}

A visitor to https://apps.frickjack.com may now briefly see a "Loading" screen for the first page load, but subsequent page loads generally load fast enough from cache that the shell doesn't bother to show the "Loading" screen. I can improve on this by extending the shell with a service worker - that will be my next project.

I envy gmail's loading screen that renders a crazy animation of a googley envelope. Fancy!

jasminejs tests with es2015 modules

Running jasminejs tests of javascript modules in a browser is straight forward, but requires small adjustments, since a module loads asynchronously, while jasmine's default runtime assumes code loads synchronously. The approach I take is the same as the one used to test requirejs AMD modules:

• customize jasmine boot, so that it does not automatically launch test execution on page load
• write a testMain.js script that imports all the spec's for your test suite, then launches jasmine's test execution

For example, this is the root test suite (testMain.ts) for the @littleware/little-elements module:

import './test/spec/utilSpec.js';
import './arrivalPie/spec/arrivalPieSpec.js';
import './styleGuide/spec/styleGuideSpec.js';
import {startTest} from './test/util.js';

startTest();

The startTest() function is a little wrapper that detects whether karmajs is the test runtime, since the test bootstrap process is a little different in that scenario. The karmajs config file should also annotate javascript module files with a module type. For example, here is an excerpt from little-element's karma.conf.js:

    files: [
      'lib/test/karmaAdapter.js',
      { pattern: 'lib/arrivalPie/**/*.js', type: 'module', included: false },
      { pattern: 'lib/styleGuide/**/*.js', type: 'module', included: false },
      { pattern: 'lib/test/**/*.js', type: 'module', included: false },
      { pattern: 'lib/testMain.js', type: 'module', included: true },
      { pattern: 'node_modules/lit-html/*.js', type: 'module', included: false }
    ],

Feel free to import the @littleware/little-elements module into your project if it will be helpful!

little-elements

Last week I rolled out an update to apps.frickjack.com that changes the plumbing of the site.

• The javascript is now organized as unbundled es2015 modules
• The site also leverages javascript modules to manage html templates and CSS with lit-html

I also setup my first module on npm (here) to track UX components that could be shared between projects - which was fun :-)

wait for kubernetes pods

First, there's probably a better way to do this with kubectl rollout, but I didn't know that existed till last week.

Anyway ... kubernetes' API is asynchronous, so if an operator that issues a series of kubernetes commands to update various deployments wants to wait till all the pods in those deployments are up and running, then he may take advantage of a script like: kube-wait4-pods

(
    # If new pods are still rolling/starting up, then wait for that to finish
    COUNT=0
    OK_COUNT=0
    # Don't exit till we get 2 consecutive readings with all pods running.
    while [[ "$OK_COUNT" -lt 2 ]]; do
      g3kubectl get pods
      if [[ 0 == "$(g3kubectl get pods -o json |  jq -r '[.items[] | { name: .metadata.generateName, phase: .status.phase, waitingContainers: [ try .status.containerStatuses[] | { waiting:.state|has("waiting"), ready:.ready}|(.waiting==true or .ready==false)|select(.) ]|length }] | map(select(.phase=="Pending" or .phase=="Running" and .waitingContainers > 0)) | length')" ]]; then
        let OK_COUNT+=1
      else
        OK_COUNT=0
      fi
      
      if [[ "$OK_COUNT" -lt 2 ]]; then
        echo ------------
        echo "INFO: Waiting for pods to exit Pending state"
        let COUNT+=1
        if [[ COUNT -gt 30 ]]; then
          echo -e "$(red_color "ERROR:") pods still not ready after 300 seconds"
          exit 1
        fi
        sleep 10
      fi
    done
)

For example - this Jenkins pipeline deploys a new stack of services to a QA environment, then waits till the new versions of pods are deployed before running through an integration test suite.

...
    stage('K8sDeploy') {
      steps {
        withEnv(['GEN3_NOPROXY=true', "vpc_name=$env.KUBECTL_NAMESPACE", "GEN3_HOME=$env.WORKSPACE/cloud-automation"]) {
          echo "GEN3_HOME is $env.GEN3_HOME"
          echo "GIT_BRANCH is $env.GIT_BRANCH"
          echo "GIT_COMMIT is $env.GIT_COMMIT"
          echo "KUBECTL_NAMESPACE is $env.KUBECTL_NAMESPACE"
          echo "WORKSPACE is $env.WORKSPACE"
          sh "bash cloud-automation/gen3/bin/kube-roll-all.sh"
          sh "bash cloud-automation/gen3/bin/kube-wait4-pods.sh || true"
        }
      }
    }
...

BTW - the code referenced above is a product of the great team of developers at the Center for Data Intensive Science.

nginx reverse-proxy tricks for jwt and csrf

I've been doing devops work over the last few months at CDIS. One of the tasks I worked on recently was to extend the nginx configuration (on github as a kubernetes configmap here) for the gen3 data commons stack.

First, we added end-user data to the proxy log by extracting user information from the JWT token in the auth cookie. We took advantage of the nginscript (nginscript is a subset of javascript) to import some standard javascript code (also in the kubernetes configmap) to parse the JWT.

function userid(req, res) {
  var token = req.variables["access_token"];
  var user = "uid:null,unknown@unknown";

  if (token) {
    // note - raw token is secret, so do not expose in userid
    var raw = atob((token.split('.')[1] || "").replace('-', '+').replace('_', '/'));
    if (raw) {
      try {
        var data = JSON.parse(raw);
        if (data) {
          if (data.context && data.context.user && data.context.user.name) {
            user = "uid:" + data.sub + "," + data.context.user.name;
          }
        }
      } catch (err) {}
    }
  }
  return user;
}

Next, we added a CSRF guard that verifies that if a CSRF cookie is present, then it matches a CSRF header. The logic is a little clunky (in github here), because of the restrictions on conditionals in nginx configuration, but it basically looks like the sample configuration below - where $access_token is the JWT auth token (either from a header from non-web clients, or a cookie for web browser clients), and an "ok-SOMETHING" $csrf_check is only required for browser clients. Finally - we do not enforce the HTTP-header based CSRF check in the proxy for endpoints that may be accessed by traditional HTML form submissions (that embed the token in the form body), or for endpoints that are accessed by third party browser frontends (like jupyterhub) that may implement a CSRF guard in a different way. Finally, there's also a little bit of logic that auto-generates a CSRF cookie if it's not present - a javascript client in the same domain can read the cookie to get the value to put in the X-CSRF header.

          set $access_token "";
          set $csrf_check "ok-tokenauth";
          if ($cookie_access_token) {
              set $access_token "bearer $cookie_access_token";
              # cookie auth requires csrf check
              set $csrf_check "fail";
          }
          if ($http_authorization) {
              # Authorization header is present - prefer that token over cookie token
              set $access_token "$http_authorization";
          }

          #
          # CSRF check
          # This block requires a csrftoken for all POST requests.
          #
          if ($cookie_csrftoken = $http_x_csrf_token) {
            # this will fail further below if cookie_csrftoken is empty
            set $csrf_check "ok-$cookie_csrftoken";
          }
          if ($request_method != "POST") {
            set $csrf_check "ok-$request_method";
          }
          if ($cookie_access_token = "") {
            # do this again here b/c empty cookie_csrftoken == empty http_x_csrf_token - ugh  
            set $csrf_check "ok-tokenauth";
          }
          ...
          location /index/ {
              if ($csrf_check !~ ^ok-\S.+$) {
                return 403 "failed csrf check";
              }
              if ($cookie_csrftoken = "") {
                add_header Set-Cookie "csrftoken=$request_id$request_length$request_time$time_iso8601;Path=/";
              }

              proxy_pass http://indexd-service/;
          }
          ...