Write an extensible library

Jsonnet gives us a lot of freedom to organize our libraries, there is no right or wrong, however a well-organized library can get you a long way. While applying common software development best-practices, we'll come up with an extensible library to deploy a webserver on Kubernetes.

Objectives

Write object-oriented with 'mixin' functions
Develop for extensibility with ::, +: and objects rather than arrays
Properly use keywords such as local, super, self, null and $
Know how to avoid common pitfalls

Lesson

Creating an extensible library

Let's start with a simple Deployment of a webserver:

{
  apiVersion: 'apps/v1',
  kind: 'Deployment',
  metadata: {
    name: 'webserver',
  },
  spec: {
    replicas: 1,
    template: {
      spec: {
        containers: [
          {
            name: 'httpd',
            image: 'httpd:2.4',
          },
        ],
      },
    },
  },
}

// example1.jsonnet

Try example1.jsonnet in Jsonnet Playground

A Deployment needs a number of configuration options, most importantly a unique name and an array of containers

The name attribute exists on both the metadata and the first container. To refer to these without ambiguity we can use a dot-notation, so referring can become more explicit with metadata.name and spec.template.spec.containers[0].name.

Let's wrap this into a small webserver library and parameterize the name because 'webserver' may be a bit too generic:

local webserver = {
  new(name, replicas=1): {
    apiVersion: 'apps/v1',
    kind: 'Deployment',
    metadata: {
      name: name,
    },
    spec: {
      replicas: replicas,
      template: {
        spec: {
          containers: [
            {
              name: 'httpd',
              image: 'httpd:2.4',
            },
          ],
        },
      },
    },
  },
};

webserver.new('wonderful-webserver')

// example2.jsonnet

Try example2.jsonnet in Jsonnet Playground

The local keyword makes this part of the code only available within this file, it is often used for importing libraries from other files, for example local myapp = import 'myapp.libsonnet';.

The Deployment is wrapped into a new() function with a name and an optional replicas arguments, this configures metadata.name and spec.replicas respectively.

Let's add another function to modify the image of the httpd container:

local webserver = {
  new(name, replicas=1): {
    apiVersion: 'apps/v1',
    kind: 'Deployment',
    metadata: {
      name: name,
    },
    spec: {
      replicas: replicas,
      template: {
        spec: {
          containers: [
            {
              name: 'httpd',
              image: 'httpd:2.4',
            },
          ],
        },
      },
    },
  },

  withImage(image): {
    local containers = super.spec.template.spec.containers,
    spec+: {
      template+: {
        spec+: {
          containers: [
            if container.name == 'httpd'
            then container { image: image }
            else container
            for container in containers
          ],
        },
      },
    },
  },
};

webserver.new('wonderful-webserver')
+ webserver.withImage('httpd:2.5')

// example3.jsonnet

Try example3.jsonnet in Jsonnet Playground

withImage is an optional 'mixin' function to modify the Deployment, notice how the new() function did not have to change to make this possible. The function is intended to be concatenated to the Deployment object created by new(), it uses the super keyword to access the container attribute.

As the container attribute is an array, it is not intuitive to modify an single entry. We have to loop over the array, find the matching container and apply a patch. This is quite verbose and hard to read.

Let's make the container a bit more accessible by moving it out of the Deployment:

local webserver = {
  new(name, replicas=1): {
    local base = self,

    container:: {
      name: 'httpd',
      image: 'httpd:2.4',
    },

    deployment: {
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: {
        name: name,
      },
      spec: {
        replicas: replicas,
        template: {
          spec: {
            containers: [
              base.container,
            ],
          },
        },
      },
    },
  },

  withImage(image): {
    container+: { image: image },
  },
};

webserver.new('wonderful-webserver')
+ webserver.withImage('httpd:2.5')

// example4.jsonnet

Try example4.jsonnet in Jsonnet Playground

This makes the code a lot more succinct, no more loops and conditionals needed. The code now reads more like a declarative document.

This introduces the :: syntax, it hides an attribute from the final output but allows for future changes to be applied to them. The withImage function uses +:, this concatenates the image patch to the container attribute, using a single colon it maintains the same hidden visibility as the Deployment object has defined.

The local base variable refers to the self keyword which returns the current object (first curly brackets it encounters). The deployment then refers to self.container, as self is late-bound any changes to container will be reflected in deployment.

To expose the webserver, a port is configured below. Now imagine that you are not the author of this library and want to change the ports attribute.

local webserver = {
  new(name, replicas=1): {
    local base = self,

    container:: {
      name: 'httpd',
      image: 'httpd:2.4',
      ports: [{
        containerPort: 80,
      }],
    },

    deployment: {
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: {
        name: name,
      },
      spec: {
        replicas: replicas,
        template: {
          spec: {
            containers: [
              base.container,
            ],
          },
        },
      },
    },
  },

  withImage(image): {
    container+: { image: image },
  },
};

webserver.new('wonderful-webserver')
+ webserver.withImage('httpd:2.5')
+ {
  container+: {
    ports: [{
      containerPort: 8080,
    }],
  },
}

// example6.jsonnet

Try example6.jsonnet in Jsonnet Playground

The author has not provided a function for that however, unlike Helm charts, it is not necessary to fork the library to make this change. Jsonnet allows the user to change any attribute after the fact by concatenating a 'patch'. The container+: will maintain the visibility of the container attribute while ports: will change the value of container.ports.

This trait of Jsonnet keeps a balance between library authors providing a useful library and users to extend it easily. Authors don't need to think about every use case out there, they can apply YAGNI and keep the library code terse and maintainable without sacrificing extensibility.

Common pitfalls when creating libraries

Builder pattern

Avoid the 'builder' pattern:

local webserver = {
  new(name, replicas=1): {
    local base = self,

    container:: {
      name: 'httpd',
      image: 'httpd:2.4',
    },

    deployment: {
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: {
        name: name,
      },
      spec: {
        replicas: replicas,
        template: {
          spec: {
            containers: [
              base.container,
            ],
          },
        },
      },
    },

    withImage(image):: self + {
      container+: { image: image },
    },
  },
};

webserver.new('wonderful-webserver').withImage('httpd:2.5')

// pitfall1.jsonnet

Try pitfall1.jsonnet in Jsonnet Playground

Notice the odd withImage():: self + {} structure within new().

This practice nests functions in the newly created object, allowing the user to 'chain' functions to modify self. However this comes at a performance impact in the Jsonnet interpreter and should be avoided.

`_config` and `_images` pattern

A common pattern involves libraries that use the _config and _images keys. This supposedly attempts to differentiate between 'public' and 'private' APIs on libraries. However the underscore prefix has no real meaning in Jsonnet, at best it is a convention with implied meaning.

Applying the convention to above library would make it look like this:

local webserver = {
  local base = self,

  _config:: {
    name: error 'provide name',
    replicas: 1,
  },

  _images:: {
    httpd: 'httpd:2.4',
  },

  container:: {
    name: 'httpd',
    image: base._images.httpd,
  },

  deployment: {
    apiVersion: 'apps/v1',
    kind: 'Deployment',
    metadata: {
      name: base._config.name,
    },
    spec: {
      replicas: base._config.replicas,
      template: {
        spec: {
          containers: [
            base.container,
          ],
        },
      },
    },
  },
};

webserver {
  _config+: {
    name: 'wonderful-webserver',
  },
  _images+: {
    httpd: 'httpd:2.5',
  },
}

// pitfall2.jsonnet

Try pitfall2.jsonnet in Jsonnet Playground

This convention attempts to provide a 'stable' API through the _config and _images parameters, implying that patching other attributes will not be supported. However the 'public' attributes (indicated by the _ prefix) are not more public or private than the 'private' attributes as they exists the same space. To make the name parameter a required argument, an error is returned if it is not set in _config.

It is comparable to the values.yaml in Helm charts, however Jsonnet does not face the same limitations and as mentioned before users can modify the final output after the fact either way.

This pattern also has an impact on extensibility. When introducing a new attribute, the author needs to take into account that users might not want the same default.

local webserver = {
  local base = self,

  _config:: {
    name: error 'provide name',
    replicas: 1,
    imagePullPolicy: null,
  },

  _images:: {
    httpd: 'httpd:2.4',
  },

  container:: {
    name: 'httpd',
    image: base._images.httpd,
  } + (
    if base._config.imagePullPolicy != null
    then { imagePullPolicy: base._config.imagePullPolicy }
    else {}
  ),

  deployment: {
    apiVersion: 'apps/v1',
    kind: 'Deployment',
    metadata: {
      name: base._config.name,
    },
    spec: {
      replicas: base._config.replicas,
      template: {
        spec: {
          containers: [
            base.container,
          ],
        },
      },
    },
  },
};

webserver {
  _config+: {
    name: 'wonderful-webserver',
    imagePullPolicy: 'Always',
  },
  _images+: {
    httpd: 'httpd:2.5',
  },
}

// pitfall3.jsonnet

Try pitfall3.jsonnet in Jsonnet Playground

This can be accomplished with imperative statements, however these pile up over time and make the library brittle and hard to read. In this example the default for imagePullPolicy is null, the author avoids adding an additional boolean parameter (_config.imagePullPolicyEnabled for example) with the drawback that no default value can be provided.

In the object-oriented library this can be done with a new function:

local webserver = {
  new(name, replicas=1): {
    local base = self,

    container:: {
      name: 'httpd',
      image: 'httpd:2.4',
      ports: [{
        containerPort: 80,
      }],
    },

    deployment: {
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: {
        name: name,
      },
      spec: {
        replicas: replicas,
        template: {
          spec: {
            containers: [
              base.container,
            ],
          },
        },
      },
    },
  },

  withImage(image): {
    container+: { image: image },
  },

  withImagePullPolicy(policy='Always'): {
    container+: { imagePullPolicy: policy },
  },
};

webserver.new('wonderful-webserver')
+ webserver.withImage('httpd:2.5')
+ webserver.withImagePullPolicy()

// example7.jsonnet

Try example7.jsonnet in Jsonnet Playground

The withImagePullPolicy() function provides a more declarative approach to configure this new option. In contrast to the approach above this new feature does not have to modify the existing code, keeping a strong separation of concerns and reduces the risk of introducing bugs.

At the same time functions provide a clean API for the end user and the author alike, replacing the implied convention with declarative statements with required and optional arguments. Calling the function implies that the user wants to set a value, the optional arguments provides a default value Always to get the user going.

Use of `$`

As you might have noticed, the $ keyword is not used in any of these examples. In many libraries it is used to refer to variables that still need to be set.

local webserver1 = {
  _images:: {
    httpd: 'httpd:2.4',
  },
  webserver1: {
    apiVersion: 'apps/v1',
    kind: 'Deployment',
    metadata: {
      name: 'webserver1',
    },
    spec: {
      replicas: 1,
      template: {
        spec: {
          containers: [{
            name: 'httpd',
            image: $._images.httpd,
          }],
        },
      },
    },
  },
};

local webserver2 = {
  _images:: {
    httpd: 'httpd:2.5',
  },
  webserver2: {
    apiVersion: 'apps/v1',
    kind: 'Deployment',
    metadata: {
      name: 'webserver2',
    },
    spec: {
      replicas: $._config.httpd_replicas,
      template: {
        spec: {
          containers: [{
            name: 'httpd',
            image: $._images.httpd,
          }],
        },
      },
    },
  },
};

webserver1 + webserver2 + {
  _config:: {
    httpd_replicas: 1,
  },
}

// pitfall4.jsonnet

Try pitfall4.jsonnet in Jsonnet Playground

This pattern makes it hard to determine which library is consuming which attribute. On top of that libraries can influence each other unintentionally.

In this example:

_config.httpd_replicas is only consumed by webserver2 while it seems to apply to both.
_image.httpd is set on both libraries, however webserver2 overrides the image of webserver1 as it was concatenated later.

This practice comes from an anti-pattern to merge several libraries on top of each other and refer to attributes that need to be set elsewhere. Or in other words, $ promotes the concept known as 'globals' in other programming libraries. It is best to avoid this as it leads to spaghetti code.

Conclusion

By following an object-oriented approach, it is possible to build extensible jsonnet libraries. They can be extended infinitely and in such a way that it doesn't impact existing uses, providing backwards compatibility.

The pitfalls show a few patterns that exist in the wild but should be avoided and refactored as they become unsustainable in the long term.

index next »