Use StarlingX as DevOps Infra

The scalability of StarlingX is awesome. It can be functional with only 3 nodes and scale to 100+ nodes in the future. This is perfect for devops use case. A new project might only have a few servers at the beginning and need to scale it in the future.

This wiki assumes that the user already deployed a StarlingX and wants to practice devops for their projects. It will share some BKMs and an example StarlingX App for such use case.

Install StarlingX
Refer to: StarlingX R2.0 Installation

CephFs Setup
DevOps App will use Cephfs for persistent volumes. It needs to setup CephFS on starlingx before install App.

ceph auth get-or-create client.stx-devops mon 'allow r' mds 'allow r, allow rw path=/stx-devops' osd 'allow rw' ceph osd pool create fs_data 2048 ceph osd pool create fs_metadata 256 ceph fs new cephfs fs_metadata fs_data /usr/bin/ceph-mds --cluster ceph --id controller-0 --hot-standby 0 /usr/bin/ceph-mds --cluster ceph --id controller-1 --hot-standby 0 sudo mkdir -p /mnt/mycephfs sudo mount -t ceph controller-0:6789:/ /mnt/mycephfs sudo mkdir -p -m 777 /mnt/mycephfs/stx-devops/jenkins-master sudo mkdir -p -m 777 /mnt/mycephfs/stx-devops/pub sudo mkdir -p -m 777 /mnt/mycephfs/stx-devops/registry sudo mkdir -p -m 777 /mnt/mycephfs/stx-devops/docker-io-mirror
 * Create Ceph user and key
 * Create Cephfs pools
 * Start mds manually
 * 1) on controller-0
 * 1) on controller-1
 * init folders for stx-devops

Extend ephemeral-storage
ephemeral-storage is much faster than cephfs and can be used as local disk for CI/CD. By default, it is about 9G on every node.

Create extend-cgts-vg.sh script as below: source /etc/platform/openrc set -x
 * 1) !/bin/bash -e

HOST_NAME=$1 PV_NAME=cgts-vg EXT_DISK=$2

EXT_DISK_UUID=$(system host-disk-list $HOST_NAME --nowrap | grep ${EXT_DISK} | awk '{print $2}') EXT_PARTITION_SIZE=$(system host-disk-list $HOST_NAME --nowrap | grep ${EXT_DISK} | awk '{print $12}' | awk '{print int($1-1)}') EXT_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol $HOST_NAME ${EXT_DISK_UUID} ${EXT_PARTITION_SIZE}) EXT_PARTITION_UUID=$(echo ${EXT_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')

echo ">>> Wait for partition $EXT_PARTITION_UUID to be ready." while true; do   if system host-disk-partition-list $HOST_NAME --nowrap | grep $EXT_PARTITION_UUID | grep -E "Ready|unlock"; then break fi   sleep 1 done

Lock host and run extend-cgts-vg.sh to extend cgts-vg: extend-cgts-vg.sh 

Unlock the host and run below command: system host-fs-modify $HOST_NAME kubelet=

Restart kubelet on target node to update node info: ssh systemctl restart kubelet

Install StarlingX DevOps App
Download it from stx-devops App v1.0.0 system application-upload stx-devops-1.0.0.tgz

system helm-override-update stx-devops stx-devops stx-devops \ --set images.tags.registry="/registry:2.7.1" \ --set images.tags.nginx="/nginx:1.16.0" \ --set images.tags.jenkins="/jenkins/jenkins:lts" \ --set images.tags.jenkins_slave="/jenkins/jnlp-slave:3.35-5-alpine" \ --set images.tags.docker_build="/starlingxabc/docker-build" \ --set ingress.base_url=" " \ --set ceph.user="client.stx-devops" \ --set ceph.key=" " \ --set proxy.enabled=true \ --set proxy.http_proxy="http:// : " \ --set proxy.https_proxy="https:// : "

system application-apply stx-devops

Build docker image
Open "http:///jenkins" with browser and run "docker-build" as below:

When the build is completed, the docker image will be uploaded to registry "ingress_url".

An image tarball will also be saved to "ingress_url/pub/docker-build" as below:

Design diagram
This App uses Cephfs as the persistent volume for jenkins, docker registry and share folder. Jenkins master runs inside a pod as a service. Jenkins slave will be created when build is triggered. It runs inside a pod, which has slave agent and build containers. When build is finished, the result will be pushed to local registry and saved to share folder. All services will be exported by ingress.