Solum/FeatureBlueprints/BuildingSourceIntoDeploymentArtifacts

Blueprint: https://blueprints.launchpad.net/solum/+spec/lang-pack Drafted by: Clayton Coleman

Objectives

1. Support turning user code into immutable server images that can be run, scaled, and deployed in an OpenStack environment easily.
2. Allow consumers to be extremely precise about dependencies, down to the kernel where desired.
3. Create a simple model that can fit a wide range of build and deployment workflows (all workflows?)
4. Support the following existing models - be simple enough to encompass all, but avoid hiding complexity and features
   1. vm images (linux or windows)
   2. docker containers
   3. heroku build packs
   4. openshift cartridges
5. Offer simple flows for most use cases, and allow complex composition of source code and runtime environments where necessary

All flows take a base image (docker container, other container, or vm), inject some artifacts (source code, binaries, static config), execute a command inside a running instance based on that image, and then initialize an execution environment. The resulting snapshot (and metadata) becomes a deployment artifact that can be started in an openstack environment and have network traffic routed to it. In general, Solum expects to closely link a source code repository or input binary file format to a process that creates executable isolated processes, but developers should be free to implement arbitrary complexity.

Each base image defines a contract for the source repository input it expects and how that source will be used. A Java image might expect to receive a standard maven project, would build that project using "mvn", and then expect to receive a WAR file that can be started with Tomcat. Advanced composition is out of scope of this reference document.

The process of transforming a base image into a deployment artifact is called **preparation** - it can include a number of traditional steps such as build, test, and deployment onto a filesystem. Since the input and process of preparation may be organization or technology specific, Solum expects a deployable artifact to be a runnable image (vm or otherwise).

Inputs:

• A base image that can be started via a hypervisor or in a linux container
  • image is assumed to have a writable filesystem
• Arbitrary input parameters, source code, or binary artifacts
  • must have some sort of contract as to how it is placed inside the building instance
• A known command (or set of commands) in that that can be used to

Outputs

• A mutated filesystem that can support an execution environment of the injected inputs (ie run a built WAR)
  • stored in glance and associated with the consuming application
• Sufficient information to allow a consumer to interact with the container
  • network ports, the type of protocol those ports support, and any specific info to allow firewalls, load balancing, or port mapping
  • for docker containers, a run command
  • for vms, one or more services that run on startup

Example of Java running in a Docker container

1. author creates a docker file with java on Ubuntu 12 via apt-get
2. author generates an image from that docker file and uploads it to the local glance server
3. a Solum application is constructed pointing to this image as the basis for a build
4. when the user pushes code to their git repo, a post receive hook notifies Solum to build commit ABC
5. Solum starts a container via nova/container service based on the image, and bind mounts a tar of the repo at commit ABC to a certain location
6. Solum instructs the container to execute a known command (or uses the default run command) that should build the source in the tarball
7. the command in the container unzips the tar, runs a maven build, deletes any temporary files, and moves the resulting WAR into a directory where a system service will run it on port 8080
8. when the command completes successfully, Solum creates a new docker image via "docker commit" and ensures the proper metadata is set so that the WAR will be running when that image is started
9. Solum registers the image with Glance and triggers a new deployment using that ID

Example of Python running on a Windows vm

1. author creates a Windows 7 vm image that contains an installed Python distribution
2. author uploads that image to glance
3. a Solum application is constructed pointing to this image as the basis for a build
4. when the user pushes code to their git repo, a post receive hook notifies Solum to build commit ABC
5. Solum starts a vm via nova based on the image, with a user data script that downloads the tarball for the repo at commit ABC and executes a known command that should build the source in the tarball
6. the command in the vm unzips the tar, runs a pip install, and then defines a windows service that will start on port 8080 at system start and run the provided source as a wsgi app.
7. when the command completes successfully, Solum creates a new image via snapshot
8. Solum registers the image with Glance and triggers a new deployment using that ID

Example of a Heroku buildpack running on a Docker container

1. author creates a docker image with Ubuntu 12 and the set of packages that Heroku makes available (python 2.7, etc)
   1. In that image, the author creates a command that should work for any buildpack
2. author uploads that image to glance
3. a Solum application is constructed pointing to this image as the basis for a build, and includes an input parameter to a specific buildpack
4. when the user pushes code to their git repo, a post receive hook notifies Solum to build commit ABC
5. Solum starts a container via nova/container service based on the image, and bind mounts a tar of the repo at commit ABC to a certain location
6. Solum instructs the container to execute the buildpack step
7. the command in the container unzips the source tar, download/clones the provided buildpack based on an input parameter, unzips it to a temporary directory, and executes the buildpack bin/compile step against the source
8. Since Heroku depends on a Procfile, create a Procfile if necessary, install the foreman gem, and then sets the run command to execute foreman against the home directory
9. when the command completes successfully, Solum creates a new docker image via "docker commit" and ensures the proper metadata is set so that the procfile/foreman will be running when that image is started
10. Solum registers the image with Glance and triggers a new deployment using that ID

Updating deployment units and base images

There are three types of updates that are generally applied that might cause a deployment unit to need to be updated. They are

1. user application update
2. image author update that changes how a build is run, fixes a bug, or updates a key dependency
3. operator security update of a package in an image

The first is part of the Solum deployment flow and is described in other documents.

An image author may wish to make new versions of their base images available - that process may include a registry of base images included with Solum along with the necessary API to add, update, and delete images. This topic may be covered in future blueprints.

The need to periodically apply security or bug fixes naturally leads to a model where both an operator and author may generate an image on their own terms. In some scenarios, the operator may allow images to be uploaded directly, but by doing so may lose the ability to offer developers a controlled security update stream. As a consequence of operators needing to propagate security updates to a large number of images (the base images, the deployments), this blueprint describes how that might occur:

1. Operator identifies a security update to a set of packages or products
2. Operator identifies the set of affected base images that it knows how to recreate
   1. For each base image, run a recreation step that applies the security update, or refreshes the image from an upstream source
      1. For each application that utilizes those base images, either notify the application owner, or trigger a new preparation and deployment of the application
3. Operator identifiers the set of base images and external deployment artifacts that it cannot recreate
   1. For each base image, notify the application owner

The need to refresh or recreate an image to take security updates implies that creating and maintaining images should emphasize the recipe and processes by which an image is created rather than the actual binary images themselves. For docker containers, a dockerfile is a natural mechanism by which such recipes can be distributed, and the operator may wish to offer base images they create themselves. For virtual machines, base images and well defined user scripts can offer similar flexibility. It is hoped that communities of base image authors

User selection

This blueprint does not define the method by which developers browse, select, and use base images to create their deployment units. It is reasonable to assume a set of registries both public and operator maintained that add user facing metadata (names, descriptions, categorization) and enhanced capabilities to the images themselves.

Naming

Interoperator portability