StarlingX/Containers/Applications/app-vault

Application: vault-armada-app

Source

• Code Repository
• Gerrit Reviews

Building

• From the Debian Build environment:

VAULT_PKGS="python3-k8sapp-vault,vault-helm,vault-manager-helm,stx-vault-helm"
build-pkgs -c -p $VAULT

The packages contain:

• python3-k8sapp-vault - sysinv integrations; helm and lifecycle
• vault-helm - the build of upstream vault helm chart
• vault-manager-helm - the build of Starlingx vault-manager helm chart
• stx-vault-helm - the Starlingx application; metadata, fluxcd yaml

The final package stx-vault-helm contains the output of the others. The stx-vault-helm package is installed on the ISO. The installed application tarball is '/usr/local/share/applications/helm/vault-*.tgz'

Testing

Vault sanity should typically include:

• application lifecycle: upload, apply, remove, abort, delete, update, and helm overrides
• Vault with replicas=1 (AIO-SX) and replicas=3 (AIO-DX + worker or standard controller)
• Configure vault with CLI and REST API
• Workflow for applications that are Vault aware (REST API) and unaware (vault injector)
• Pod recovery
• Vault backup and restore
• Vault-manager rekey feature
• Download images from private registry; network isolation test

Application Lifecycle

Refer to Application Commands and Helm Overrides

Typically changes for the following should be accompanied with testing for the application lifecycle:

• Sysinv integrations, package python3-k8sapp-vault
• New and updated helm charts or chart overrides (values.yaml)

And also at the end of each release cycle the sanity for application lifecycle should be asserted.

Vault with replicas 1 and 3

When the vault app is applied on AIO-SX, the vault server statefulset is configured automatically with replicas=1. When there are three or more controller and worker nodes provisioned in the cluster then the vault server statefulset will be configured automatically with replicas=3.

Sanity for replicas=3 should include vault HA - observation that a standby vault server becomes active when the active server is restarted gracefully.

Refer also to "Pod Recovery" section.

Configure using CLI and REST API

Refer to the following documents for examples of configuring the vault using CLI and REST API:

• Configure Vault Using the Vault CLI
• Configure Vault Using the Vault REST API

Follow each configuration with sanity of the workflows in Vault Aware and Unaware test section.

Vault Aware and Unaware

Refer to the following documents for examples of these workflows:

• Vault Aware
• Vault Unaware

Refer also to Configure Vault Using the Vault REST API which includes examples relevant to a vault aware application: "Create a secret", "Verify the secret"

Pod Recovery

The application pods should recover automatically when they are terminated gracefully or killed.

Kubernetes automatically restarts pods, while vault-manager will automatically unseal vault servers.

For example, delete the active vault server pod and watch the vault manager unseal it:

active="$( kubectl get pods -n vault \
    -o jsonpath='{.items[?(@.metadata.labels.vault-active=="true")].metadata.name}' )"
manager="$( kubectl get pods -n vault \
    -o jsonpath='{.items[?(@.metadata.labels.app\.kubernetes\.io/name=="vault-manager")].metadata.name}' )"
kubectl delete pods -n vault "$active"
kubectl logs -f -n vault "$manager"

Also delete the vault manager pod to observe it's recovery, and reassert that it can recover vault server pods that are deleted.

Vault Backup and Restore

TBD - the backup and restore feature of vault is new and not yet documented

Vault Rekey

The rekey feature of vault manager regenerates the unseal shards of the vault. Test the feature by creating a request, observing the vault manager rekey the vault and finally reasserting that vault server pods can be unsealed.

Create a rekey request and watch vault manager complete the request:

manager="$( kubectl get pods -n vault \
    -o jsonpath='{.items[?(@.metadata.labels.app\.kubernetes\.io/name=="vault-manager")].metadata.name}' )"

uuidgen | kubectl create secret generic -n vault cluster-rekey-request --from-file=strdata=/dev/stdin
kubectl logs -f -n vault "$manager"

Reassert the vault server pods will be unsealed by performing the "Pod Recovery" procedure.