Jump to: navigation, search


< StarlingX‎ | Containers
Revision as of 19:20, 7 January 2019 by Frank.miller (talk | contribs) (Introduction)

Installing and configuring StarlingX with containers


These instructions are for an All-in-one simplex system in VirtualBox. Other configurations are in development. Installing on bare metal is also possible, however the the process would have to be adapted for the specific hardware configuration.

Note: These instructions are valid for a load built on December 22, 2018 or later.

Building the Software

Follow the standard build process in the StarlingX Developer Guide. A prebuilt iso can be used, but additional packages are needed after the initial installation. A developer environment is required to build those packages.

Setup the VirtualBox VM

Create a virtual machine for the system with the following options:

     * Type: Linux
     * Version: Other Linux (64-bit)
     * Memory size: 16384 MB
     * Storage: 
        * Recommend to use VDI and dynamically allocated disks
        * Two disks are required
          * 240GB disk for a root disk 
          * 50GB for an OSD
     * System->Processors: 
        * 4 cpu
     * Network:
        * OAM network:
           OAM interface must have external connectivity, for now we will use a NatNetwork
           * Adapter 1: NAT Network; Name: NatNetwork Follow the instructions at #VirtualBox Nat Networking 
        * Data Network
           * Adapter 2: Internal Network, Name: intnet-data1; Advanced: Paravirtualized Network (virtio-net), Promiscuous Mode: Allow All
           * Adapter 3: Internal Network, Name: intnet-data2; Advanced: Paravirtualized Network (virtio-net), Promiscuous Mode: Allow All

Install StarlingX

Boot the VM from the ISO media. Select the following options for installation:

  • All-in-one Controller
  • Graphical Console
  • Standard Security Profile

Initial Configuration

Run config_controller

sudo config_controller --kubernetes

Use default settings during config_controller, except for the following

System mode: simplex External OAM address:

The system configuration should look like this:

System Configuration
Time Zone: UTC
System mode: simplex

PXEBoot Network Configuration
Separate PXEBoot network not configured
PXEBoot Controller floating hostname: pxecontroller

Management Network Configuration
Management interface name: lo
Management interface: lo
Management interface MTU: 1500
Management interface link capacity Mbps: 10000
Management subnet:
Controller floating address:
Controller 0 address:
Controller 1 address:
NFS Management Address 1:
NFS Management Address 2:
Controller floating hostname: controller
Controller hostname prefix: controller-
OAM Controller floating hostname: oamcontroller
Dynamic IP address allocation is selected
Management multicast subnet:

Infrastructure Network Configuration
Infrastructure interface not configured

External OAM Network Configuration
External OAM interface name: enp0s3
External OAM interface: enp0s3
External OAM interface MTU: 1500
External OAM subnet:
External OAM gateway address:
External OAM address:

Provisioning the platform

  • Set DNS server (so we can set the ntp servers)
source /etc/platform/openrc
system dns-modify nameservers= action=apply
  • Set the ntp server
system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
  • Create partitions on the root disk (40G for the cgts-vg, 34G for nova) and wait for them to be ready. The first partition extends the existing cgts volume group. This step is optional as there should be sufficient space by default. The second partition for nova-local is manidtory.
system host-disk-list controller-0
system host-disk-partition-add -t lvm_phys_vol controller-0 $(system host-disk-list controller-0 | awk '/sda/{print $2}') 40
while true; do system host-disk-partition-list controller-0 --nowrap | grep 40\.0 | grep Ready; if [ $? -eq 0 ]; then break; fi; sleep 1; done
system host-disk-partition-add -t lvm_phys_vol controller-0 $(system host-disk-list controller-0 | awk '/sda/{print $2}') 34
while true; do system host-disk-partition-list controller-0 --nowrap | grep 34\.0 | grep Ready; if [ $? -eq 0 ]; then break; fi; sleep 1; done
system host-disk-partition-list controller-0
  • Create the volume group for nova.
system host-lvg-add controller-0 nova-local
  • Create physical volumes from the partitions.
system host-pv-add controller-0 cgts-vg $(system host-disk-partition-list controller-0 | awk '/sda5/{print $2}')
system host-pv-add controller-0 nova-local $(system host-disk-partition-list controller-0 | awk '/sda6/{print $2}')
system host-pv-list controller-0
  • Add the ceph storage backend (note: you may have to wait a few minutes before this command will succeed)
system storage-backend-add ceph --confirmed
  • Wait for 'applying-manifests' task to complete
while [ $(system storage-backend-list | awk '/ceph-store/{print $8}') != 'configured' ]; do echo 'Waiting for ceph to be configured'; sleep 5; done
  • Add an OSD (/dev/sdb)
system host-disk-list controller-0
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
system host-stor-list controller-0
  • Set Ceph pool replication
ceph osd pool ls | xargs -i ceph osd pool set {} size 1
  • Configure data interfaces
export COMPUTE=controller-0
source /etc/platform/openrc
system host-port-list ${COMPUTE} --nowrap > ${SPL}
system host-if-list -a ${COMPUTE} --nowrap > ${SPIL}
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
DATA1PORTNAME=$(cat  $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
system host-if-modify -m 1500 -n data0 -p ${PHYSNET0} -c data ${COMPUTE} ${DATA0IFUUID}
system host-if-modify -m 1500 -n data1 -p ${PHYSNET1} -c data ${COMPUTE} ${DATA1IFUUID}
  • Unlock the controller
system host-unlock controller-0
  • After the host unlocks, test that the ceph cluster is operational
ceph -s
    cluster 6cb8fd30-622a-4a15-a039-b9e945628133
     health HEALTH_OK
     monmap e1: 1 mons at {controller-0=}
            election epoch 4, quorum 0 controller-0
     osdmap e32: 1 osds: 1 up, 1 in
            flags sortbitwise,require_jewel_osds
      pgmap v35: 1728 pgs, 6 pools, 0 bytes data, 0 objects
            39180 kB used, 50112 MB / 50150 MB avail
                1728 active+clean

Prepare the host for running the containerized services

  • On the controller node, apply all the node labels for each controller and compute functions
source /etc/platform/openrc
system host-label-assign controller-0 openstack-control-plane=enabled
system host-label-assign controller-0  openstack-compute-node=enabled
system host-label-assign controller-0  openvswitch=enabled
kubectl get nodes --show-labels
  • Add the DNS for the cluster
DNS_EP=$(kubectl describe svc -n kube-system kube-dns | awk /IP:/'{print $2}')
system dns-modify nameservers="$DNS_EP,"
  • Perform the following platform tweak.
sudo sh -c "echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables"

Using sysinv to bring up/down the containerized services

  • Generate the stx-openstack application tarball. In a development environment, run the following command to construct the application tarballs. The tarballs can be found under $MY_WORKSPACE/containers/build-helm/stx. Currently it produces 2 application tarballs, one with tests enabled and one without. Transfer the selected tarball to your lab/virtual box.
  • Stage application for deployment: Use sysinv to upload the application tarball.
system application-upload stx-openstack helm-charts-manifest-no-tests.tgz
system application-list
  • Bring Up Services: Use sysinv to apply the application. You can monitor the progress either by watching system application-list (watch -n 1.0 system application-list) or tailing Armada execution log (sudo docker exec armada_service tailf stx-openstack-apply.log).

NOTE This step requires access to the StarlingX docker images. There is outstanding work to add the StarlingX docker images to a public repository. This work is tracked under Story 2003907

system application-apply stx-openstack
system application-list

With the application applied the containerized openstack services are now running. You must now set Ceph pool replication for the new pools created when the application was applied:

ceph osd pool ls | xargs -i ceph osd pool set {} size 1

Skip to #Verify the cluster endpoints to continue the setup.

The following commands are for reference.

  • Bring Down Services: Use sysinv to uninstall the application.
system application-remove stx-openstack
system application-list
  • Delete Services: Use sysinv to delete the application definition.
system application-delete stx-openstack
system application-list
  • Bring Down Services: Clean up and stragglers (volumes and pods)
# Watch and wait for the pods to terminate
kubectl get pods -n openstack -o wide -w

# Armada Workaround: delete does not clean up the old test pods. Sooo... Delete them.
kubectl get pods -n openstack | awk '/osh-.*-test/{print $1}' | xargs -i kubectl delete pods -n openstack --force --grace-period=0 {}

# Cleanup all PVCs
kubectl get pvc --all-namespaces; kubectl get pv --all-namespaces
kubectl delete pvc --all --namespace openstack; kubectl delete pv --all --namespace openstack
kubectl get pvc --all-namespaces; kubectl get pv --all-namespaces

# Useful to cleanup the mariadb grastate data.
kubectl get configmaps -n openstack | awk '/osh-/{print $1}' | xargs -i kubectl delete configmaps -n openstack {}

# Remove all the contents of the ceph pools. I have seen orphaned contents here that take up space.
for p in cinder-volumes images kube-rbd; do rbd -p $p ls | xargs -i rbd -p $p snap unprotect {}@snap; done
for p in cinder-volumes images kube-rbd; do rbd -p $p ls | xargs -i rbd -p $p snap purge {}; done
for p in cinder-volumes images kube-rbd; do rbd -p $p ls | xargs -i rbd -p $p rm {}; done

Verify the cluster endpoints

# Note: Do this from a new shell as a root user (do not source /etc/platform/openrc in that shell).
        The 'password' should be set to the admin password which configured during config_controller.

mkdir -p /etc/openstack
tee /etc/openstack/clouds.yaml << EOF
    region_name: RegionOne
    identity_api_version: 3
      username: 'admin'
      password: 'Li69nux*'
      project_name: 'admin'
      project_domain_name: 'default'
      user_domain_name: 'default'
      auth_url: 'http://keystone.openstack.svc.cluster.local/v3'

export OS_CLOUD=openstack_helm
openstack endpoint list

Nova related items

  • Execute the nova cell DB workaround
# Remove the && from the end of the line if you are executing one command at a time
openstack hypervisor list
kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password=\$MYSQL_ROOT_PASSWORD --user=root nova -e 'select host,mapped from compute_nodes'" &&
kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password=\$MYSQL_ROOT_PASSWORD --user=root nova -e 'update compute_nodes set mapped=0'" &&
kubectl exec -it -n openstack $(kubectl get pods  -n openstack | grep nova-conductor | awk '{print $1}') -- nova-manage cell_v2 discover_hosts --verbose &&
openstack hypervisor list

Provider/tenant networking setup

  • Create the providernets
neutron providernet-create ${PHYSNET0} --type vlan
neutron providernet-create ${PHYSNET1} --type vlan
  • Create host and bind interfaces
#Query sysinv db directly instead of switching credentials
neutron host-create controller-0 --id $(sudo -u postgres psql -qt -d sysinv -c "select uuid from i_host where hostname='controller-0';") --availability up
neutron host-bind-interface --interface $(sudo -u postgres psql -qt -d sysinv -c "select uuid from ethernet_interfaces join interfaces on ethernet_interfaces.id=interfaces.id where providernetworks='physnet0';") --providernets physnet0 --mtu 1500 controller-0
neutron host-bind-interface --interface $(sudo -u postgres psql -qt -d sysinv -c "select uuid from ethernet_interfaces join interfaces on ethernet_interfaces.id=interfaces.id where providernetworks='physnet1';") --providernets physnet1 --mtu 1500 controller-0
#Alternatively, can source /etc/platform/openrc and then query using sysinv api.
  • Setup tenant networking (adapt based on lab config)
ADMINID=`openstack project list | grep admin | awk '{print $2}'`
neutron providernet-range-create ${PHYSNET0} --name ${PHYSNET0}-a --range 400-499
neutron providernet-range-create ${PHYSNET0} --name ${PHYSNET0}-b --range 10-10 --shared
neutron providernet-range-create ${PHYSNET1} --name ${PHYSNET1}-a --range 500-599
neutron net-create --tenant-id ${ADMINID} --provider:network_type=vlan --provider:physical_network=${PHYSNET0} --provider:segmentation_id=10 --router:external ${EXTERNALNET}
neutron net-create --tenant-id ${ADMINID} --provider:network_type=vlan --provider:physical_network=${PHYSNET0} --provider:segmentation_id=400 ${PUBLICNET}
neutron net-create --tenant-id ${ADMINID} --provider:network_type=vlan --provider:physical_network=${PHYSNET1} --provider:segmentation_id=500 ${PRIVATENET}
neutron net-create --tenant-id ${ADMINID} ${INTERNALNET}
PUBLICNETID=`neutron net-list | grep ${PUBLICNET} | awk '{print $2}'`
PRIVATENETID=`neutron net-list | grep ${PRIVATENET} | awk '{print $2}'`
INTERNALNETID=`neutron net-list | grep ${INTERNALNET} | awk '{print $2}'`
EXTERNALNETID=`neutron net-list | grep ${EXTERNALNET} | awk '{print $2}'`
neutron subnet-create --tenant-id ${ADMINID} --name ${PUBLICSUBNET} ${PUBLICNET}
neutron subnet-create --tenant-id ${ADMINID} --name ${PRIVATESUBNET} ${PRIVATENET}
neutron subnet-create --tenant-id ${ADMINID} --name ${INTERNALSUBNET} --no-gateway  ${INTERNALNET}
neutron subnet-create --tenant-id ${ADMINID} --name ${EXTERNALSUBNET} --gateway --disable-dhcp ${EXTERNALNET}
neutron router-create ${PUBLICROUTER}
neutron router-create ${PRIVATEROUTER}
PRIVATEROUTERID=`neutron router-list | grep ${PRIVATEROUTER} | awk '{print $2}'`
PUBLICROUTERID=`neutron router-list | grep ${PUBLICROUTER} | awk '{print $2}'`
neutron router-gateway-set --disable-snat ${PUBLICROUTERID} ${EXTERNALNETID}
neutron router-gateway-set --disable-snat ${PRIVATEROUTERID} ${EXTERNALNETID}
neutron router-interface-add ${PUBLICROUTER} ${PUBLICSUBNET}
neutron router-interface-add ${PRIVATEROUTER} ${PRIVATESUBNET}

Horizon access

# After successful armada manifest apply the following should be seen

kubectl get services -n openstack | grep horizon
horizon                       ClusterIP    <none>        80/TCP,443/TCP                 13h
horizon-int                   NodePort   <none>        80:31000/TCP                   13h

The platform horizon UI is available at http://<external OAM IP>

 $ curl -L -so - | egrep '(PlugIn|<title>)'
    <title>Login - StarlingX</title>
    global.horizonPlugInModules = ['horizon.dashboard.project', 'horizon.dashboard.container-infra', 'horizon.dashboard.dc_admin', 'horizon.dashboard.identity', 'horizon.app.murano'];

The containerized horizon UI is available at http://<external OAM IP>:31000

$ curl -L -so - | egrep '(PlugIn|<title>)'
    <title>Login - StarlingX</title>
    global.horizonPlugInModules = ['horizon.dashboard.project', 'horizon.dashboard.identity'];

After controller node reboot

Until we get them integrated into node startup, there are a number of the above items that need to be re-done after every controller node reboot:

sudo sh -c "echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables"
  • If the keystone-api pod is stuck in a CrashLoopBackOff, delete the pod and it will be re-created.
# List the pods to get the name of the keystone-api pod
kubectl -n openstack get pods
# Delete the keystone-api pod
kubectl -n openstack delete pod <name of keystone-api pod>
  • If you are seeing DNS failures for cluster addresses, restart dnsmasq on the controller after puppet has completed its initialization.
sudo sm-restart service dnsmasq

VirtualBox Nat Networking

First add a NAT Network in VirtualBox:

 * Select File -> Preferences menu
 * Choose Network, "Nat Networks" tab should be selected
   * Click on plus icon to add a network, which will add a network named NatNetwork
   * Edit the NatNetwork (gear or screwdriver icon)
     * Network CIDR: (to match OAM network specified in config_controller)
     * Disable "Supports DHCP"
     * Enable "Supports IPv6"
     * Select "Port Forwarding" and add any rules you desire. Some examples:
Name Protocol Host IP Host Port Guest IP Guest Port
controller-ssh TCP 22 22
controller-http TCP 80 80
controller-https TCP 443 443