Jump to: navigation, search

Difference between revisions of "StarlingX/Containers/Installation"

(Tenant Networking setup)
(41 intermediate revisions by 12 users not shown)
Line 1: Line 1:
 +
{{Warning|header='''Warning - Deprecated'''|body='''This wiki page is out of date and now deprecated. For the current and upcoming versions, see [https://docs.starlingx.io/deploy_install_guides/index.html StarlingX Installation and Deployment guides]'''}}
 +
 
= Installing StarlingX with containers: One node configuration =
 
= Installing StarlingX with containers: One node configuration =
 +
 +
The information on this wiki page is in the process of transitioning to "Deploy/Install" guides that are being created as part of the StarlingX documentation.
 +
Consequently, do not make edits to the content in this wiki page.  If you have changes that need to be made to the installation process described on this page of
 +
the wiki, contact StarlingX Documentation Team.
  
 
== History ==
 
== History ==
Line 7: Line 13:
 
* '''January 29, 2019:''' Removed obsolete neutron host/interface configuration and updated DNS instructions.
 
* '''January 29, 2019:''' Removed obsolete neutron host/interface configuration and updated DNS instructions.
 
* '''February 12, 2019:''' Add network segment range instructions for Stein: ' openstack network segment range create' supercedes 'neutron providernet-range-create’.
 
* '''February 12, 2019:''' Add network segment range instructions for Stein: ' openstack network segment range create' supercedes 'neutron providernet-range-create’.
 
+
* '''April 29, 2019:''' Add instructions for controller bootstrap using Ansible playbook.
 
== Introduction ==
 
== Introduction ==
  
Line 57: Line 63:
 
| controller-ssh || TCP || || 22 || 10.10.10.3 || 22
 
| controller-ssh || TCP || || 22 || 10.10.10.3 || 22
 
|-
 
|-
| controller-http || TCP || || 80 || 10.10.10.3 || 8080
+
| controller-http-openstack || TCP || || 31000 || 10.10.10.3 || 31000
 
|-
 
|-
| controller-https || TCP ||  || 443 || 10.10.10.3 || 8443  
+
| controller-https-openstack || TCP ||  || 443 || 10.10.10.3 || 443
 +
|-
 +
| controller-http-platform || TCP || || 8080 || 10.10.10.3 || 8080
 +
|-
 +
| controller-https-platform || TCP ||  || 8443 || 10.10.10.3 || 8443  
 
|}
 
|}
  
Line 80: Line 90:
 
Enter a new password for the wrsroot account and confirm it.
 
Enter a new password for the wrsroot account and confirm it.
  
=== Run config_controller ===
+
=== Bootstrap the controller ===
 +
 
 +
The following instructions are intended for local bootstrap of initial controller in virtual box. For ease of use in development and controlled test environments, passwords can be provided via command line/unencrypted text file (user override file) and wrsroot password is used for ssh authentication. In production, sensitive info should be stored in Ansible vault secret file and SSH keys instead of password should be used for authentication.
 +
 
 +
==== Location of controller bootstrap playbook ====
 +
All StarlingX playbooks are located under the directory /usr/share/ansible/stx-ansible/playbooks. Consequently, the controller bootstrap playbook is located at:
 +
/usr/share/ansible/stx-ansible/playbooks/bootstrap/.
 +
 
 +
==== Default bootstrap playbook settings ====
 +
The default inventory file, which resides in Ansible configuration directory (i.e. /etc/ansible/hosts), contains one single host - the localhost. You can override this file using a custom hosts file and the "-i option". Doing so makes the file available for remote play through the Ansible playbook.
 +
 
 +
The /usr/share/ansible/stx-ansible/playbooks/bootstrap/host_vars/default.yml file specifies the default configuration parameters. You can overwrite these parameters in two ways:
 +
 
 +
- Using either the --extra-vars or -e options at the command line.
  
<code>sudo config_controller </code>
+
- Using an override file.
  
Use default settings during config_controller, except for the following:
+
Using the override file is the preferred option when multiple parameters exist that need to be overwritten.
* System mode: '''simplex'''
 
* External OAM address: '''10.10.10.3'''
 
* If you do not have direct access to the google DNS nameserver(s) 8.8.8.8 , 8.8.4.4 you will need to configure that when prompted. Press Enter to choose the default, or type a new entry.
 
* If you do not have direct access to the public docker registry (https://hub.docker.com/u/starlingx) and instead use a proxy for internet access, you will need to add proxy information when prompted. (Storyboard 2004710 was merged on Jan 30, 2019. )
 
  
The system configuration should look like this:
+
By default Ansible looks for and imports user override files in the wrsroot home directory ($HOME). If you want to place these files in a different location, you must specify the location by using the -e option (e.g. -e "override_files_dir=<custom-override-dir>").
<pre>
+
 
System Configuration
+
The override file must conform to the following naming convention: <inventory_hostname>.yml
--------------------
+
An example filename is localhost.yml.
Time Zone: UTC
 
System mode: simplex
 
  
PXEBoot Network Configuration
+
==== Password types ====
-----------------------------
+
For local bootstrap, two types of passwords exist:
Separate PXEBoot network not configured
 
PXEBoot Controller floating hostname: pxecontroller
 
  
Management Network Configuration
+
- *ansible_become_pass*: a Sudo password to run tasks that require escalated privileges. Most bootstrap tasks must be run as root. Since the playbook is run by the wrsroot user, this is the wrsroot password.
--------------------------------
 
Management interface name: lo
 
Management interface: lo
 
Management interface MTU: 1500
 
Management subnet: 192.168.204.0/28
 
Controller floating address: 192.168.204.2
 
Controller 0 address: 192.168.204.3
 
Controller 1 address: 192.168.204.4
 
NFS Management Address 1: 192.168.204.5
 
NFS Management Address 2: 192.168.204.6
 
Controller floating hostname: controller
 
Controller hostname prefix: controller-
 
OAM Controller floating hostname: oamcontroller
 
Dynamic IP address allocation is selected
 
Management multicast subnet: 239.1.1.0/28
 
  
Infrastructure Network Configuration
+
- *admin_password*: A password used in when system commands, such as a Horizon login, are executed.
------------------------------------
 
Infrastructure interface not configured
 
  
Kubernetes Cluster Network Configuration
+
For remote bootstrap, if an automatic SSH login that uses an SSH key has not been set up between the Ansible control node and the target controller, another password is required:
----------------------------------------
 
Cluster pod network subnet: 172.16.0.0/16
 
Cluster service network subnet: 10.96.0.0/12
 
Cluster host interface name: lo
 
Cluster host interface: lo
 
Cluster host interface MTU: 1500
 
Cluster host subnet: 192.168.206.0/24
 
  
External OAM Network Configuration
+
- *ansible_ssh_pass*: The password used to log into the target host(s).
----------------------------------
 
External OAM interface name: enp0s3
 
External OAM interface: enp0s3
 
External OAM interface MTU: 1500
 
External OAM subnet: 10.10.10.0/24
 
External OAM gateway address: 10.10.10.1
 
External OAM address: 10.10.10.3
 
  
DNS Configuration
+
For all the passwords mentioned in this section, the defaults are set to "St8rlingX*".
-----------------
 
Nameserver 1: 8.8.8.8
 
Nameserver 2: 8.8.4.4
 
  
 +
==== Running the bootstrap playbook ====
 +
To run the playbook, you need to first set up external connectivity.
 +
<pre>
 +
ip address add 10.10.10.3/24 dev enp0s3
 +
ip link set up dev enp0s3
 +
route add default gw 10.10.10.1 enp0s3
 +
ping 8.8.8.8
 
</pre>
 
</pre>
 +
 +
*Run the local playbook with all defaults or with override file /home/wrsroot/localhost.yml
 +
<pre>ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml</pre>
 +
 +
Sample /home/wrsroot/localhost.yml override file. Add more (or remove) parameters or change values to your custom values as needed:
 +
<pre>
 +
external_oam_subnet: 10.10.10.0/24
 +
external_oam_gateway_address: 10.10.10.1
 +
external_oam_floating_address: 10.10.10.3
 +
management_subnet: 192.168.204.0/24
 +
dns_servers:
 +
  - 8.8.4.4
 +
admin_password: St8rlingX*
 +
ansible_become_pass: St8rlingX*
 +
</pre>
 +
 +
*Run the local playbook with custom wrsroot and admin passwords specified in the command line
 +
<pre>ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml -e "ansible_become_pass=<custom-wrsroot-password> admin_password=<custom-admin-password>" </pre>
 +
 +
Note: As Ansible does not yet support specifying playbooks search path(s). It is necessary to specify the path to the bootstrap playbook in the command line if you're not already in that directory. A command alias called bootstrap-controller will be provided in the near future.
  
 
=== Provisioning the platform ===
 
=== Provisioning the platform ===
  
==== Set the ntp server ====
+
==== Configure the OAM interface ====
 +
<pre>
 +
source /etc/platform/openrc
 +
OAM_IF=enp0s3
 +
system host-if-modify controller-0 $OAM_IF -c platform
 +
system interface-network-assign controller-0 $OAM_IF oam
 +
</pre>
 +
 
 +
==== (Hardware lab only) Set the ntp server ====
 +
Skip this step in virtualbox as it can cause Ceph's clock skew alarms. Moreover, clock of vbox instances is synchronized with the host clock so there is no need to configure NTP here.
  
 
<pre>
 
<pre>
Line 160: Line 177:
  
 
* As of March 29th 2019, OVS running in a container is deployed by default.
 
* As of March 29th 2019, OVS running in a container is deployed by default.
To deploy OVS-DPDK (supported only on baremetal hardware), please run the following command:
+
To deploy OVS-DPDK (supported only on baremetal hardware), please run the following commands:
  
 
<pre>
 
<pre>
 
system modify --vswitch_type ovs-dpdk
 
system modify --vswitch_type ovs-dpdk
 +
system host-cpu-modify -f vswitch -p0 1 controller-0
 
# To set the vswitch type back to the default (i.e. OVS running in a container), run:
 
# To set the vswitch type back to the default (i.e. OVS running in a container), run:
 
# system modify --vswitch_type none
 
# system modify --vswitch_type none
Line 169: Line 187:
 
* NOTE: For virtual environments, only OVS running in a container is supported.
 
* NOTE: For virtual environments, only OVS running in a container is supported.
 
* NOTE: The vswitch type cannot be modified after controller-0 is unlocked.
 
* NOTE: The vswitch type cannot be modified after controller-0 is unlocked.
 +
* IMPORTANT: When deploying OVS-DPDK, VMs must be configured to use a flavor with property: hw:mem_page_size=large
  
 
==== Configure data interfaces ====
 
==== Configure data interfaces ====
Line 197: Line 216:
  
 
# the host-if-modify '-p' flag is deprecated in favor of  the '-d' flag for assignment of datanetworks.
 
# the host-if-modify '-p' flag is deprecated in favor of  the '-d' flag for assignment of datanetworks.
system host-if-modify -m 1500 -n data0 -d ${PHYSNET0} -c data ${COMPUTE} ${DATA0IFUUID}
+
system host-if-modify -m 1500 -n data0 -c data ${COMPUTE} ${DATA0IFUUID}
system host-if-modify -m 1500 -n data1 -d ${PHYSNET1} -c data ${COMPUTE} ${DATA1IFUUID}
+
system host-if-modify -m 1500 -n data1 -c data ${COMPUTE} ${DATA1IFUUID}
 +
system interface-datanetwork-assign ${COMPUTE} ${DATA0IFUUID} ${PHYSNET0}
 +
system interface-datanetwork-assign ${COMPUTE} ${DATA1IFUUID} ${PHYSNET1}
 
</pre>
 
</pre>
  
Line 217: Line 238:
  
 
* Create partitions on the root disk  and wait for them to be ready
 
* Create partitions on the root disk  and wait for them to be ready
** 34G for nova-local (mandatory).
+
** 24G for nova-local (mandatory).
** 6G for the cgts-vg (optional). This extends the existing cgts volume group. There should be sufficient space by default)
 
 
 
 
<pre>
 
<pre>
 
export COMPUTE=controller-0
 
export COMPUTE=controller-0
Line 230: Line 249:
  
 
echo ">>>> Configuring nova-local"
 
echo ">>>> Configuring nova-local"
NOVA_SIZE=34
+
NOVA_SIZE=24
 
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${COMPUTE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
 
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${COMPUTE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
 
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
 
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
Line 239: Line 258:
 
echo ">>> Wait for partition $NOVA_PARTITION_UUID to be ready."
 
echo ">>> Wait for partition $NOVA_PARTITION_UUID to be ready."
 
while true; do system host-disk-partition-list $COMPUTE --nowrap | grep $NOVA_PARTITION_UUID | grep Ready; if [ $? -eq 0 ]; then break; fi; sleep 1; done
 
while true; do system host-disk-partition-list $COMPUTE --nowrap | grep $NOVA_PARTITION_UUID | grep Ready; if [ $? -eq 0 ]; then break; fi; sleep 1; done
 +
</pre>
  
 +
==== Extend cgts-vg (config_controller method only) ====
 +
* 6G for the cgts-vg (optional). This extends the existing cgts volume group. There should be sufficient space by default)
 +
<pre>
 
echo ">>>> Extending cgts-vg"
 
echo ">>>> Extending cgts-vg"
 
PARTITION_SIZE=6
 
PARTITION_SIZE=6
Line 260: Line 283:
  
 
<pre>
 
<pre>
source /etc/platform/openrc
 
echo ">>> Enable primary Ceph backend"
 
system storage-backend-add ceph --confirmed
 
 
echo ">>> Wait for primary ceph backend to be configured"
 
echo ">>> This step really takes a long time"
 
while [ $(system storage-backend-list | awk '/ceph-store/{print $8}') != 'configured' ]; do echo 'Waiting for ceph..'; sleep 5; done
 
 
echo ">>> Ceph health"
 
ceph -s
 
 
 
echo ">>> Add OSDs to primary tier"
 
echo ">>> Add OSDs to primary tier"
  
Line 276: Line 288:
 
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
 
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
 
system host-stor-list controller-0
 
system host-stor-list controller-0
 
echo ">>> ceph osd tree"
 
ceph osd tree
 
</pre>
 
 
==== Set Ceph pool replication (AIO-SX only) ====
 
<pre>
 
ceph osd pool ls | xargs -i ceph osd pool set {} size 1
 
 
</pre>
 
</pre>
  
Line 295: Line 299:
  
 
<pre>
 
<pre>
 +
# Verify that Ceph is in good health
 
ceph -s
 
ceph -s
 
     cluster 6cb8fd30-622a-4a15-a039-b9e945628133
 
     cluster 6cb8fd30-622a-4a15-a039-b9e945628133
Line 311: Line 316:
 
=== Generate the stx-openstack application tarball ===
 
=== Generate the stx-openstack application tarball ===
  
The [http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green_build/outputs/helm-charts/ stx-openstack application tarballs] are generated with each build on the CENGN mirror.
+
The [http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_docker_image_build/outputs/helm-charts/ stx-openstack application tarballs] are generated with each build on the CENGN mirror.
  
 
Alternatively, in a development environment, run the following command to construct the application tarballs.  
 
Alternatively, in a development environment, run the following command to construct the application tarballs.  
Line 317: Line 322:
 
$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh
 
$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh
 
</pre>
 
</pre>
The resulting tarballs can be found under $MY_WORKSPACE/std/build-helm/stx.  
+
* The resulting tarballs can be found under $MY_WORKSPACE/std/build-helm/stx.
 
+
* By default, the latest stable starlingx docker images are used in armada manifest. You can build the application tarball with different image versions by specifying the image record files/urls which contain the images you would like to use via option --image-record (The [http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/ starlingx image build records] can be found on the CENGN mirror)
If the build-helm-charts.sh command is unable to find the charts, run  "build-pkgs" to build the chart rpms and re-run the build-helm-charts.sh command.
+
* To construct a new name of stx-openstack tarball, specify a label with --label option. The name of the stx-openstack application tarball is stx-openstack-<stx-openstack-helm rpm version>(-<label>).tgz
 +
* If the build-helm-charts.sh command is unable to find the charts, run  "build-pkgs" to build the chart rpms and re-run the build-helm-charts.sh command.
  
 
=== Stage application for deployment ===
 
=== Stage application for deployment ===
Transfer the helm-charts-manifest.tgz application tarball onto your active controller.
+
Transfer the stx-openstack application tarball onto your active controller.
  
 
Use sysinv to upload the application tarball.  
 
Use sysinv to upload the application tarball.  
Line 328: Line 334:
 
<pre>
 
<pre>
 
source /etc/platform/openrc
 
source /etc/platform/openrc
system application-upload stx-openstack helm-charts-manifest.tgz
+
system application-upload stx-openstack-1.0-11.tgz
 
system application-list
 
system application-list
 
</pre>
 
</pre>
 +
* stx-openstack application tarball has a metadata.yaml file which contains the app name and version. The app name and version will be extracted from the metadata.yaml when uploading. For the application tarballs that do not have app name and version included in metadata.yaml, they need to be specified via --app-name and --app-version.
  
 
=== Bring Up Services ===
 
=== Bring Up Services ===
Line 339: Line 346:
 
  watch -n 5 system application-list
 
  watch -n 5 system application-list
  
or tailing Armada execution log
+
After the "retrieving images" phase, the Armada execution logs can be tailed:<br />
  sudo docker exec armada_service tailf stx-openstack-apply.log
+
* On the host with
 +
tail -f /var/log/armada/stx-openstack-apply.log
 +
 
 +
* Inside the container with:
 +
  sudo docker exec armada_service tail -f /logs/stx-openstack-apply.log
  
 
=== Update Ceph pool replication (AIO-SX only) ===  
 
=== Update Ceph pool replication (AIO-SX only) ===  
Line 381: Line 392:
  
 
== Provider/tenant networking setup ==
 
== Provider/tenant networking setup ==
=== Providernetworking setup (pre-Stein) ===
 
Note: this section is invalid for the f/stein branch; please refer to Providernetworking setup (Stein) instead.
 
 
* Create the providernets
 
<pre>
 
PHYSNET0='physnet0'
 
PHYSNET1='physnet1'
 
neutron providernet-create ${PHYSNET0} --type vlan
 
neutron providernet-create ${PHYSNET1} --type vlan
 
 
neutron providernet-range-create ${PHYSNET0} --name ${PHYSNET0}-a --range 400-499
 
neutron providernet-range-create ${PHYSNET0} --name ${PHYSNET0}-b --range 10-10 --shared
 
neutron providernet-range-create ${PHYSNET1} --name ${PHYSNET1}-a --range 500-599
 
</pre>
 
 
=== Providernetworking setup: Network Segment Ranges (Stein) ===
 
 
Note: This section is for the f/stein branch; this will be activated upon cutover to Stein (Pending).
 
  
 
* Create the network segment ranges
 
* Create the network segment ranges
 
  
 
Note: The 'physical-network' name must match name of the datanetwork configured in sysinv via the 'system datanetwork-add' command above.
 
Note: The 'physical-network' name must match name of the datanetwork configured in sysinv via the 'system datanetwork-add' command above.
Line 524: Line 516:
 
     global.horizonPlugInModules = ['horizon.dashboard.project', 'horizon.dashboard.identity'];
 
     global.horizonPlugInModules = ['horizon.dashboard.project', 'horizon.dashboard.identity'];
 
   
 
   
 +
</pre>
 +
 +
== Instance Console Access ==
 +
<pre>
 +
Add an alias to the novncproxy default local domain in /etc/hosts file on your machine or in other words, from the Horizon Graphical interfaces is launched.
 +
 +
<your Controller IP address> novncproxy.openstack.svc.cluster.local
 +
 +
'''On your Controller'''
 +
Disable firewall on port 80,31000 and 443
 +
sudo iptables -A INPUT -p tcp -m multiport --dports 80 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
 +
sudo iptables -A INPUT -p tcp -m multiport --dports 31000 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
 +
sudo iptables -A INPUT -p tcp -m multiport --dports 443 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
 +
 
</pre>
 
</pre>
  

Revision as of 15:09, 15 July 2019

Warning icon.svg Warning - Deprecated

This wiki page is out of date and now deprecated. For the current and upcoming versions, see StarlingX Installation and Deployment guides

Installing StarlingX with containers: One node configuration

The information on this wiki page is in the process of transitioning to "Deploy/Install" guides that are being created as part of the StarlingX documentation. Consequently, do not make edits to the content in this wiki page. If you have changes that need to be made to the installation process described on this page of the wiki, contact StarlingX Documentation Team.

History

  • January 18, 2019: Removed Nova Cell DB Workaround - no longer required on loads built January 15th or later.
  • January 25, 2019: Configure datanetworks in sysinv, prior to referencing it in the 'system host-if-modify/host-if-add command'. Needed on loads Jan 25, 2019 or later.
  • January 29, 2019: Removed obsolete neutron host/interface configuration and updated DNS instructions.
  • February 12, 2019: Add network segment range instructions for Stein: ' openstack network segment range create' supercedes 'neutron providernet-range-create’.
  • April 29, 2019: Add instructions for controller bootstrap using Ansible playbook.

Introduction

These instructions are for an All-in-one simplex system (AIO-SX) in VirtualBox. Other configurations are in development. Installing on bare metal is also possible, however the the process would have to be adapted for the specific hardware configuration.

Note: These instructions are valid for a load built on January 25, 2019 or later.

Building the Software

Follow the standard build process in the StarlingX Developer Guide.

Alternatively a prebuilt iso can be used, all required packages are provided by the StarlingX CENGN mirror

Setup the VirtualBox VM

Create a virtual machine for the system with the following options:

     * Type: Linux
     * Version: Other Linux (64-bit)
     * Memory size: 16384 MB
     * Storage: 
        * Recommend to use VDI and dynamically allocated disks
        * At least two disks are required
          * 240GB disk for a root disk 
          * 50GB for an OSD
     * System->Processors: 
        * 4 cpu
     * Network:
        * OAM network:
           OAM interface must have external connectivity, for now we will use a NatNetwork
           * Adapter 1: NAT Network; Name: NatNetwork Follow the instructions at #VirtualBox Nat Networking 
        * Data Network
           * Adapter 2: Internal Network, Name: intnet-data1; Advanced: Paravirtualized Network (virtio-net), Promiscuous Mode: Allow All
           * Adapter 3: Internal Network, Name: intnet-data2; Advanced: Paravirtualized Network (virtio-net), Promiscuous Mode: Allow All

VirtualBox Nat Networking

First add a NAT Network in VirtualBox:

 * Select File -> Preferences menu
 * Choose Network, "Nat Networks" tab should be selected
   * Click on plus icon to add a network, which will add a network named NatNetwork
   * Edit the NatNetwork (gear or screwdriver icon)
     * Network CIDR: 10.10.10.0/24 (to match OAM network specified in config_controller)
     * Disable "Supports DHCP"
     * Enable "Supports IPv6"
     * Select "Port Forwarding" and add any rules you desire. Some examples:
Name Protocol Host IP Host Port Guest IP Guest Port
controller-ssh TCP 22 10.10.10.3 22
controller-http-openstack TCP 31000 10.10.10.3 31000
controller-https-openstack TCP 443 10.10.10.3 443
controller-http-platform TCP 8080 10.10.10.3 8080
controller-https-platform TCP 8443 10.10.10.3 8443

Setup Controller-0

Install StarlingX

Boot the VM from the ISO media. Select the following options for installation:

  • All-in-one Controller
  • Graphical Console
  • Standard Security Profile

Once booted, log into Controller-0 as user wrsroot, with password wrsroot. The first time you log in as wrsroot, you are required to change your password. Enter the current password (wrsroot):

 
Changing password for wrsroot.
(current) UNIX Password: wrsroot

Enter a new password for the wrsroot account and confirm it.

Bootstrap the controller

The following instructions are intended for local bootstrap of initial controller in virtual box. For ease of use in development and controlled test environments, passwords can be provided via command line/unencrypted text file (user override file) and wrsroot password is used for ssh authentication. In production, sensitive info should be stored in Ansible vault secret file and SSH keys instead of password should be used for authentication.

Location of controller bootstrap playbook

All StarlingX playbooks are located under the directory /usr/share/ansible/stx-ansible/playbooks. Consequently, the controller bootstrap playbook is located at: /usr/share/ansible/stx-ansible/playbooks/bootstrap/.

Default bootstrap playbook settings

The default inventory file, which resides in Ansible configuration directory (i.e. /etc/ansible/hosts), contains one single host - the localhost. You can override this file using a custom hosts file and the "-i option". Doing so makes the file available for remote play through the Ansible playbook.

The /usr/share/ansible/stx-ansible/playbooks/bootstrap/host_vars/default.yml file specifies the default configuration parameters. You can overwrite these parameters in two ways:

- Using either the --extra-vars or -e options at the command line.

- Using an override file.

Using the override file is the preferred option when multiple parameters exist that need to be overwritten.

By default Ansible looks for and imports user override files in the wrsroot home directory ($HOME). If you want to place these files in a different location, you must specify the location by using the -e option (e.g. -e "override_files_dir=<custom-override-dir>").

The override file must conform to the following naming convention: <inventory_hostname>.yml An example filename is localhost.yml.

Password types

For local bootstrap, two types of passwords exist:

- *ansible_become_pass*: a Sudo password to run tasks that require escalated privileges. Most bootstrap tasks must be run as root. Since the playbook is run by the wrsroot user, this is the wrsroot password.

- *admin_password*: A password used in when system commands, such as a Horizon login, are executed.

For remote bootstrap, if an automatic SSH login that uses an SSH key has not been set up between the Ansible control node and the target controller, another password is required:

- *ansible_ssh_pass*: The password used to log into the target host(s).

For all the passwords mentioned in this section, the defaults are set to "St8rlingX*".

Running the bootstrap playbook

To run the playbook, you need to first set up external connectivity.

ip address add 10.10.10.3/24 dev enp0s3
ip link set up dev enp0s3
route add default gw 10.10.10.1 enp0s3
ping 8.8.8.8
  • Run the local playbook with all defaults or with override file /home/wrsroot/localhost.yml
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml

Sample /home/wrsroot/localhost.yml override file. Add more (or remove) parameters or change values to your custom values as needed:

external_oam_subnet: 10.10.10.0/24
external_oam_gateway_address: 10.10.10.1
external_oam_floating_address: 10.10.10.3
management_subnet: 192.168.204.0/24
dns_servers:
  - 8.8.4.4
admin_password: St8rlingX*
ansible_become_pass: St8rlingX*
  • Run the local playbook with custom wrsroot and admin passwords specified in the command line
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml -e "ansible_become_pass=<custom-wrsroot-password> admin_password=<custom-admin-password>" 

Note: As Ansible does not yet support specifying playbooks search path(s). It is necessary to specify the path to the bootstrap playbook in the command line if you're not already in that directory. A command alias called bootstrap-controller will be provided in the near future.

Provisioning the platform

Configure the OAM interface

source /etc/platform/openrc
OAM_IF=enp0s3
system host-if-modify controller-0 $OAM_IF -c platform
system interface-network-assign controller-0 $OAM_IF oam

(Hardware lab only) Set the ntp server

Skip this step in virtualbox as it can cause Ceph's clock skew alarms. Moreover, clock of vbox instances is synchronized with the host clock so there is no need to configure NTP here.

source /etc/platform/openrc
system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org

Configure the vswitch type (optional)

  • As of March 29th 2019, OVS running in a container is deployed by default.

To deploy OVS-DPDK (supported only on baremetal hardware), please run the following commands:

system modify --vswitch_type ovs-dpdk
system host-cpu-modify -f vswitch -p0 1 controller-0
# To set the vswitch type back to the default (i.e. OVS running in a container), run:
# system modify --vswitch_type none
  • NOTE: For virtual environments, only OVS running in a container is supported.
  • NOTE: The vswitch type cannot be modified after controller-0 is unlocked.
  • IMPORTANT: When deploying OVS-DPDK, VMs must be configured to use a flavor with property: hw:mem_page_size=large

Configure data interfaces

DATA0IF=eth1000
DATA1IF=eth1001
export COMPUTE=controller-0
PHYSNET0='physnet0'
PHYSNET1='physnet1'
SPL=/tmp/tmp-system-port-list
SPIL=/tmp/tmp-system-host-if-list
source /etc/platform/openrc
system host-port-list ${COMPUTE} --nowrap > ${SPL}
system host-if-list -a ${COMPUTE} --nowrap > ${SPIL}
DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
DATA1PORTNAME=$(cat  $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')

# configure the datanetworks in sysinv, prior to referencing it in the 'system host-if-modify command'
system datanetwork-add ${PHYSNET0} vlan
system datanetwork-add ${PHYSNET1} vlan

# the host-if-modify '-p' flag is deprecated in favor of  the '-d' flag for assignment of datanetworks.
system host-if-modify -m 1500 -n data0 -c data ${COMPUTE} ${DATA0IFUUID}
system host-if-modify -m 1500 -n data1 -c data ${COMPUTE} ${DATA1IFUUID}
system interface-datanetwork-assign ${COMPUTE} ${DATA0IFUUID} ${PHYSNET0}
system interface-datanetwork-assign ${COMPUTE} ${DATA1IFUUID} ${PHYSNET1}

Prepare the host for running the containerized services

  • On the controller node, apply all the node labels for each controller and compute functions
source /etc/platform/openrc

system host-label-assign controller-0 openstack-control-plane=enabled
system host-label-assign controller-0 openstack-compute-node=enabled
system host-label-assign controller-0 openvswitch=enabled
system host-label-assign controller-0 sriov=enabled

Setup partitions for Controller-0

  • Create partitions on the root disk and wait for them to be ready
    • 24G for nova-local (mandatory).
export COMPUTE=controller-0
source /etc/platform/openrc

echo ">>> Getting root disk info"
ROOT_DISK=$(system host-show ${COMPUTE} | grep rootfs | awk '{print $4}')
ROOT_DISK_UUID=$(system host-disk-list ${COMPUTE} --nowrap | grep ${ROOT_DISK} | awk '{print $2}')
echo "Root disk: $ROOT_DISK, UUID: $ROOT_DISK_UUID"

echo ">>>> Configuring nova-local"
NOVA_SIZE=24
NOVA_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${COMPUTE} ${ROOT_DISK_UUID} ${NOVA_SIZE})
NOVA_PARTITION_UUID=$(echo ${NOVA_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')
system host-lvg-add ${COMPUTE} nova-local
system host-pv-add ${COMPUTE} nova-local ${NOVA_PARTITION_UUID}
sleep 2

echo ">>> Wait for partition $NOVA_PARTITION_UUID to be ready."
while true; do system host-disk-partition-list $COMPUTE --nowrap | grep $NOVA_PARTITION_UUID | grep Ready; if [ $? -eq 0 ]; then break; fi; sleep 1; done

Extend cgts-vg (config_controller method only)

  • 6G for the cgts-vg (optional). This extends the existing cgts volume group. There should be sufficient space by default)
echo ">>>> Extending cgts-vg"
PARTITION_SIZE=6
CGTS_PARTITION=$(system host-disk-partition-add -t lvm_phys_vol ${COMPUTE} ${ROOT_DISK_UUID} ${PARTITION_SIZE})
CGTS_PARTITION_UUID=$(echo ${CGTS_PARTITION} | grep -ow "| uuid | [a-z0-9\-]* |" | awk '{print $4}')

echo ">>> Wait for partition $CGTS_PARTITION_UUID to be ready"
while true; do system host-disk-partition-list $COMPUTE --nowrap | grep $CGTS_PARTITION_UUID | grep Ready; if [ $? -eq 0 ]; then break; fi; sleep 1; done

system host-pv-add ${COMPUTE} cgts-vg ${CGTS_PARTITION_UUID}
sleep 2

echo ">>> Waiting for cgts-vg to be ready"
while true; do system host-pv-list ${COMPUTE} | grep cgts-vg | grep adding; if [ $? -ne 0 ]; then break; fi; sleep 1; done

system host-pv-list ${COMPUTE} 

Configure Ceph for Controller-0

echo ">>> Add OSDs to primary tier"

system host-disk-list controller-0
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
system host-stor-list controller-0

Unlock the controller

system host-unlock controller-0
  • After the host unlocks, test that the ceph cluster is operational
# Verify that Ceph is in good health
ceph -s
    cluster 6cb8fd30-622a-4a15-a039-b9e945628133
     health HEALTH_OK
     monmap e1: 1 mons at {controller-0=127.168.204.3:6789/0}
            election epoch 4, quorum 0 controller-0
     osdmap e32: 1 osds: 1 up, 1 in
            flags sortbitwise,require_jewel_osds
      pgmap v35: 1728 pgs, 6 pools, 0 bytes data, 0 objects
            39180 kB used, 50112 MB / 50150 MB avail
                1728 active+clean

Using sysinv to bring up/down the containerized services

Generate the stx-openstack application tarball

The stx-openstack application tarballs are generated with each build on the CENGN mirror.

Alternatively, in a development environment, run the following command to construct the application tarballs.

$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh
  • The resulting tarballs can be found under $MY_WORKSPACE/std/build-helm/stx.
  • By default, the latest stable starlingx docker images are used in armada manifest. You can build the application tarball with different image versions by specifying the image record files/urls which contain the images you would like to use via option --image-record (The starlingx image build records can be found on the CENGN mirror)
  • To construct a new name of stx-openstack tarball, specify a label with --label option. The name of the stx-openstack application tarball is stx-openstack-<stx-openstack-helm rpm version>(-<label>).tgz
  • If the build-helm-charts.sh command is unable to find the charts, run "build-pkgs" to build the chart rpms and re-run the build-helm-charts.sh command.

Stage application for deployment

Transfer the stx-openstack application tarball onto your active controller.

Use sysinv to upload the application tarball.

source /etc/platform/openrc
system application-upload stx-openstack-1.0-11.tgz
system application-list
  • stx-openstack application tarball has a metadata.yaml file which contains the app name and version. The app name and version will be extracted from the metadata.yaml when uploading. For the application tarballs that do not have app name and version included in metadata.yaml, they need to be specified via --app-name and --app-version.

Bring Up Services

Use sysinv to apply the application.

system application-apply stx-openstack

You can monitor the progress by watching system application-list

watch -n 5 system application-list

After the "retrieving images" phase, the Armada execution logs can be tailed:

  • On the host with
tail -f /var/log/armada/stx-openstack-apply.log
  • Inside the container with:
sudo docker exec armada_service tail -f /logs/stx-openstack-apply.log

Update Ceph pool replication (AIO-SX only)

With the application applied the containerized openstack services are now running.

In an AIO SX environment, you must now set Ceph pool replication for the new pools created when the application was applied:

ceph osd pool ls | xargs -i ceph osd pool set {} size 1

Verify the cluster endpoints

Note: Do this from a new shell as a root user (do not source /etc/platform/openrc in that shell).

The 'password' should be set to the admin password which configured during config_controller.


mkdir -p /etc/openstack
tee /etc/openstack/clouds.yaml << EOF
clouds:
  openstack_helm:
    region_name: RegionOne
    identity_api_version: 3
    endpoint_type: internalURL
    auth:
      username: 'admin'
      password: 'Li69nux*'
      project_name: 'admin'
      project_domain_name: 'default'
      user_domain_name: 'default'
      auth_url: 'http://keystone.openstack.svc.cluster.local/v3'
EOF

export OS_CLOUD=openstack_helm
openstack endpoint list

The remaining networking steps are done using this root user.

Provider/tenant networking setup

  • Create the network segment ranges

Note: The 'physical-network' name must match name of the datanetwork configured in sysinv via the 'system datanetwork-add' command above.

ADMINID=`openstack project list | grep admin | awk '{print $2}'`
PHYSNET0='physnet0'
PHYSNET1='physnet1'

openstack network segment range create ${PHYSNET0}-a --network-type vlan --physical-network ${PHYSNET0}  --minimum 400 --maximum 499 --private --project ${ADMINID}

openstack network segment range create  ${PHYSNET0}-b --network-type vlan  --physical-network ${PHYSNET0}  --minimum 10 --maximum 10 --shared

openstack network segment range create ${PHYSNET1}-a --network-type vlan  --physical-network  ${PHYSNET1} --minimum 500 --maximum 599  --private --project ${ADMINID}

Tenant Networking setup

  • Setup tenant networking (adapt based on lab config)
ADMINID=`openstack project list | grep admin | awk '{print $2}'`
PHYSNET0='physnet0'
PHYSNET1='physnet1'
PUBLICNET='public-net0'
PRIVATENET='private-net0'
INTERNALNET='internal-net0'
EXTERNALNET='external-net0'
PUBLICSUBNET='public-subnet0'
PRIVATESUBNET='private-subnet0'
INTERNALSUBNET='internal-subnet0'
EXTERNALSUBNET='external-subnet0'
PUBLICROUTER='public-router0'
PRIVATEROUTER='private-router0'

openstack network create --project ${ADMINID} --provider-network-type=vlan --provider-physical-network=${PHYSNET0} --provider-segment=10 --share --external ${EXTERNALNET}
openstack network create --project ${ADMINID} --provider-network-type=vlan --provider-physical-network=${PHYSNET0} --provider-segment=400 ${PUBLICNET}
openstack network create --project ${ADMINID} --provider-network-type=vlan --provider-physical-network=${PHYSNET1} --provider-segment=500 ${PRIVATENET}
openstack network create --project ${ADMINID} ${INTERNALNET}
PUBLICNETID=`openstack network list | grep ${PUBLICNET} | awk '{print $2}'`
PRIVATENETID=`openstack network list | grep ${PRIVATENET} | awk '{print $2}'`
INTERNALNETID=`openstack network list | grep ${INTERNALNET} | awk '{print $2}'`
EXTERNALNETID=`openstack network list | grep ${EXTERNALNET} | awk '{print $2}'`
openstack subnet create --project ${ADMINID} ${PUBLICSUBNET} --network ${PUBLICNET} --subnet-range 192.168.101.0/24
openstack subnet create --project ${ADMINID} ${PRIVATESUBNET} --network ${PRIVATENET} --subnet-range 192.168.201.0/24
openstack subnet create --project ${ADMINID} ${INTERNALSUBNET} --gateway none --network ${INTERNALNET} --subnet-range 10.1.1.0/24
openstack subnet create --project ${ADMINID} ${EXTERNALSUBNET} --gateway 192.168.1.1 --no-dhcp --network ${EXTERNALNET} --subnet-range 192.168.51.0/24 --ip-version 4
openstack router create ${PUBLICROUTER}
openstack router create ${PRIVATEROUTER}
PRIVATEROUTERID=`openstack router list | grep ${PRIVATEROUTER} | awk '{print $2}'`
PUBLICROUTERID=`openstack router list | grep ${PUBLICROUTER} | awk '{print $2}'`
openstack router set ${PUBLICROUTER} --external-gateway ${EXTERNALNETID} --disable-snat
openstack router set ${PRIVATEROUTER} --external-gateway ${EXTERNALNETID} --disable-snat
openstack router add subnet ${PUBLICROUTER} ${PUBLICSUBNET}
openstack router add subnet ${PRIVATEROUTER} ${PRIVATESUBNET}

Additional Setup Instructions

The following commands are for reference.


  • Bring Down Services: Use sysinv to uninstall the application.
system application-remove stx-openstack
system application-list
  • Delete Services: Use sysinv to delete the application definition.
system application-delete stx-openstack
system application-list
  • Bring Down Services: Clean up and stragglers (volumes and pods)
# Watch and wait for the pods to terminate
kubectl get pods -n openstack -o wide -w

# Armada Workaround: delete does not clean up the old test pods. Sooo... Delete them.
kubectl get pods -n openstack | awk '/osh-.*-test/{print $1}' | xargs -i kubectl delete pods -n openstack --force --grace-period=0 {}

# Cleanup all PVCs
kubectl get pvc --all-namespaces; kubectl get pv --all-namespaces
kubectl delete pvc --all --namespace openstack; kubectl delete pv --all --namespace openstack
kubectl get pvc --all-namespaces; kubectl get pv --all-namespaces

# Useful to cleanup the mariadb grastate data.
kubectl get configmaps -n openstack | awk '/osh-/{print $1}' | xargs -i kubectl delete configmaps -n openstack {}

# Remove all the contents of the ceph pools. I have seen orphaned contents here that take up space.
for p in cinder-volumes images kube-rbd; do rbd -p $p ls | xargs -i rbd -p $p snap unprotect {}@snap; done
for p in cinder-volumes images kube-rbd; do rbd -p $p ls | xargs -i rbd -p $p snap purge {}; done
for p in cinder-volumes images kube-rbd; do rbd -p $p ls | xargs -i rbd -p $p rm {}; done

Horizon access

# After successful armada manifest apply the following should be seen

kubectl get services -n openstack | grep horizon
horizon                       ClusterIP   10.104.34.245    <none>        80/TCP,443/TCP                 13h
horizon-int                   NodePort    10.101.103.238   <none>        80:31000/TCP                   13h

The platform horizon UI is available at http://<external OAM IP>

 $ curl -L http://10.10.10.3:8080 -so - | egrep '(PlugIn|<title>)'
    <title>Login - StarlingX</title>
    global.horizonPlugInModules = ['horizon.dashboard.project', 'horizon.dashboard.container-infra', 'horizon.dashboard.dc_admin', 'horizon.dashboard.identity', 'horizon.app.murano'];

The containerized horizon UI is available at http://<external OAM IP>:31000

$ curl -L http://10.10.10.3:31000 -so - | egrep '(PlugIn|<title>)'
    <title>Login - StarlingX</title>
    global.horizonPlugInModules = ['horizon.dashboard.project', 'horizon.dashboard.identity'];
 

Instance Console Access

Add an alias to the novncproxy default local domain in /etc/hosts file on your machine or in other words, from the Horizon Graphical interfaces is launched.

<your Controller IP address> novncproxy.openstack.svc.cluster.local

'''On your Controller'''
Disable firewall on port 80,31000 and 443
sudo iptables -A INPUT -p tcp -m multiport --dports 80 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
sudo iptables -A INPUT -p tcp -m multiport --dports 31000 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT
sudo iptables -A INPUT -p tcp -m multiport --dports 443 -m comment --comment "500 horizon ingress dashboard" -j ACCEPT

Known Issues and Troubleshooting

None