Difference between revisions of "StarlingX/Packet SIG"
Greg-waines (talk | contribs) (→Installing the Central Cloud ........................................................................) |
Greg-waines (talk | contribs) (→Packet SIG) |
||
Line 30: | Line 30: | ||
The initial server of a StarlingX cloud, i.e. controller-0, must be installed via a PXE Boot Server. For packet.com, IPXE Booting will actually be used ... however the setup of the ipxe boot server is the same.<br /> | The initial server of a StarlingX cloud, i.e. controller-0, must be installed via a PXE Boot Server. For packet.com, IPXE Booting will actually be used ... however the setup of the ipxe boot server is the same.<br /> | ||
<br /> | <br /> | ||
− | Get a small packet.com server (e.g. c1.small.x86) for the IPXE Boot Server. It should use packet.com L3 Networking and | + | Get a small packet.com server (e.g. c1.small.x86) for the IPXE Boot Server. It should use packet.com L3 Networking and run ubuntu 16.04 on it .<br /> |
<br /> | <br /> | ||
Setup the PXE Boot Server as follows: | Setup the PXE Boot Server as follows: | ||
Line 52: | Line 52: | ||
cd | cd | ||
/media/iso/pxeboot_setup.sh -u http://<IPADDRESS-OF-IPXE-BOOT-SERVER>/BIOS-Client -t /export/pxeboot | /media/iso/pxeboot_setup.sh -u http://<IPADDRESS-OF-IPXE-BOOT-SERVER>/BIOS-Client -t /export/pxeboot | ||
+ | </pre> | ||
+ | <br /> | ||
+ | |||
+ | ==== Get L2 VLANs and L3 Public IPs .......................................................................... ==== | ||
+ | |||
+ | Get the following L2 VLANs for the project: | ||
+ | * StarlingX MGMT Network (CC) | ||
+ | * StarlingX OAM Network (CC) | ||
+ | * StarlingX SC1 MGMT Network | ||
+ | * StarlingX SC1 OAM Network | ||
+ | * StarlingX SC1 DATA Network | ||
+ | * StarlingX SC2 MGMT Network | ||
+ | * StarlingX SC2 OAM Network | ||
+ | * StarlingX SC2 DATA Network | ||
+ | <br /> | ||
+ | Get the following L3 Public IPs for the project: | ||
+ | * CC OAM Subnet - Public IPv4/29 subnet | ||
+ | * SC1 DATA Subnet - Public IPv4/29 subnet | ||
+ | * SC2 DATA Subnet - Public IPv4/29 subnet | ||
+ | |||
+ | ==== Setting up the Router-Nat ................................................................ ==== | ||
+ | |||
+ | All of the packet.com servers running StarlingX software will use L2-only packet.com networking. A hybrid L3-Networking/L2-Networking server is required in order to route from the StarlingX L2 networks into the packet.com L3 networks and ultimately the public internet.<br /> | ||
+ | <br /> | ||
+ | Get another packet.com server, that supports hybrid L3/L2 networking (e.g. c2.medium.x86) and run ubuntu 16.04 on it. | ||
+ | * Add all the VLANs onto its L2 port | ||
+ | * Add all Public IPv4 /29 subnets to this server | ||
+ | <br /> | ||
+ | Setup the router-nat server as follows: | ||
+ | <pre> | ||
+ | apt-get update | ||
+ | apt-get install vlan | ||
+ | sudo su -c 'echo "8021q" >> /etc/modules' | ||
+ | /sbin/reboot | ||
+ | </pre> | ||
+ | vi /etc/network/interfaces | ||
+ | <pre> | ||
+ | auto lo | ||
+ | iface lo inet loopback | ||
+ | |||
+ | # L3 Interface to INTERNET | ||
+ | auto bond0 | ||
+ | iface bond0 inet static | ||
+ | address 147.75.39.198 | ||
+ | netmask 255.255.255.252 | ||
+ | gateway 147.75.39.197 | ||
+ | bond-downdelay 200 | ||
+ | bond-miimon 100 | ||
+ | bond-mode 4 | ||
+ | bond-updelay 200 | ||
+ | bond-xmit_hash_policy layer3+4 | ||
+ | bond-lacp-rate 1 | ||
+ | bond-slaves enp1s0f0 | ||
+ | dns-nameservers 147.75.207.207 147.75.207.208 | ||
+ | iface bond0 inet6 static | ||
+ | address 2604:1380:1:b800::1 | ||
+ | netmask 127 | ||
+ | gateway 2604:1380:1:b800:: | ||
+ | |||
+ | auto bond0:0 | ||
+ | iface bond0:0 inet static | ||
+ | address 10.99.254.1 | ||
+ | netmask 255.255.255.254 | ||
+ | post-up route add -net 10.0.0.0/8 gw 10.99.254.0 | ||
+ | post-down route del -net 10.0.0.0/8 gw 10.99.254.0 | ||
+ | |||
+ | auto enp1s0f0 | ||
+ | iface enp1s0f0 inet manual | ||
+ | bond-master bond0 | ||
+ | |||
+ | # CC OAM Subnet | ||
+ | auto enp1s0f1 | ||
+ | iface enp1s0f1 inet static | ||
+ | address 147.75.105.201 | ||
+ | netmask 255.255.255.248 | ||
+ | network 147.75.105.200 | ||
+ | broadcast 147.75.105.207 | ||
+ | |||
+ | # SC1 OAM Subnet | ||
+ | auto enp1s0f1.1092 | ||
+ | iface enp1s0f1.1092 inet static | ||
+ | address 11.11.11.1 | ||
+ | netmask 255.255.255.0 | ||
+ | network 11.11.11.0 | ||
+ | broadcast 11.11.11.255 | ||
+ | vlan-raw-device enp1s0f1 | ||
+ | |||
+ | # SC2 OAM Subnet | ||
+ | auto enp1s0f1.1162 | ||
+ | iface enp1s0f1.1162 inet static | ||
+ | address 12.12.12.1 | ||
+ | netmask 255.255.255.0 | ||
+ | network 12.12.12.0 | ||
+ | broadcast 12.12.12.255 | ||
+ | vlan-raw-device enp1s0f1 | ||
+ | |||
+ | # CC MGMT Subnet | ||
+ | auto enp1s0f1.1172 | ||
+ | iface enp1s0f1.1172 inet static | ||
+ | address 192.168.204.1 | ||
+ | netmask 255.255.255.0 | ||
+ | network 192.168.204.0 | ||
+ | broadcast 192.168.204.255 | ||
+ | mtu 1400 | ||
+ | vlan-raw-device enp1s0f1 | ||
+ | |||
+ | # SC1 MGMT Subnet | ||
+ | auto enp1s0f1.1117 | ||
+ | iface enp1s0f1.1117 inet static | ||
+ | address 192.168.214.1 | ||
+ | netmask 255.255.255.0 | ||
+ | network 192.168.214.0 | ||
+ | broadcast 192.168.214.255 | ||
+ | mtu 1400 | ||
+ | vlan-raw-device enp1s0f1 | ||
+ | |||
+ | # SC2 MGMT Subnet | ||
+ | auto enp1s0f1.1177 | ||
+ | iface enp1s0f1.1177 inet static | ||
+ | address 192.168.224.1 | ||
+ | netmask 255.255.255.0 | ||
+ | network 192.168.224.0 | ||
+ | broadcast 192.168.224.255 | ||
+ | mtu 1400 | ||
+ | vlan-raw-device enp1s0f1 | ||
+ | |||
+ | # SC1 DATA Subnet | ||
+ | auto enp1s0f1.1145 | ||
+ | iface enp1s0f1.1145 inet static | ||
+ | address 147.75.105.193 | ||
+ | netmask 255.255.255.248 | ||
+ | network 147.75.105.192 | ||
+ | broadcast 147.75.105.199 | ||
+ | vlan-raw-device enp1s0f1 | ||
+ | |||
+ | # SC2 DATA Subnet | ||
+ | auto enp1s0f1.1179 | ||
+ | iface enp1s0f1.1179 inet static | ||
+ | address 139.178.66.41 | ||
+ | netmask 255.255.255.248 | ||
+ | network 147.75.105.40 | ||
+ | broadcast 147.75.105.47 | ||
+ | vlan-raw-device enp1s0f1 | ||
</pre> | </pre> | ||
<br /> | <br /> |
Revision as of 18:30, 5 July 2019
Contents
- 1 Packet SIG
- 1.1 StarlingX Distributed Cloud on Packet.Com
- 1.1.1 Packet.com Servers deployed
- 1.1.2 Networking
- 1.1.3 Setting up the iPXE Boot Server ................................................................
- 1.1.4 Get L2 VLANs and L3 Public IPs ..........................................................................
- 1.1.5 Setting up the Router-Nat ................................................................
- 1.1.6 Installing the Central Cloud ........................................................................
- 1.1.7 Installing the Sub Clouds ............................................................................
- 1.1.8 Problem Tracking
- 1.1 StarlingX Distributed Cloud on Packet.Com
Packet SIG
Packet.com is a baremetal public cloud, and they have donated some resources to the StarlingX project. The resources are available under the STX-PROJECT-01 project on Packet.com.
StarlingX Distributed Cloud on Packet.Com
As a demonstration of the OpenStack Edge Computing Group's Distributed Control Plane MVP Architecture, StarlingX Distributed Cloud has been deployed on Packet.com.
STX R1 : http://mirror.starlingx.cengn.ca/mirror/starlingx/release/2018.10/centos/2018.10.0/outputs/iso/
Horizon for Central Cloud: http://147.75.105.202
SSH to Central Cloud: ssh wrsroot@147.75.105.202
Packet.com Servers deployed
Networking
Setting up the iPXE Boot Server ................................................................
The initial server of a StarlingX cloud, i.e. controller-0, must be installed via a PXE Boot Server. For packet.com, IPXE Booting will actually be used ... however the setup of the ipxe boot server is the same.
Get a small packet.com server (e.g. c1.small.x86) for the IPXE Boot Server. It should use packet.com L3 Networking and run ubuntu 16.04 on it .
Setup the PXE Boot Server as follows:
apt-get update apt-install apache2 -y wget http://mirror.starlingx.cengn.ca/mirror/starlingx/release/2018.10/centos/2018.10.0/outputs/iso/bootimage.iso mkdir -p /media/iso mount -o loop ./bootimage.iso /media/iso mount -o remount,exec,dev /media/iso mkdir -p /export/pxeboot cd /var/www/html ln -s /export/pxeboot BIOS-Client # for some reason have to remove pxeboot directory before running pxeboot_setup.sh cd /export rmdir pxeboot cd /media/iso/pxeboot_setup.sh -u http://<IPADDRESS-OF-IPXE-BOOT-SERVER>/BIOS-Client -t /export/pxeboot
Get L2 VLANs and L3 Public IPs ..........................................................................
Get the following L2 VLANs for the project:
- StarlingX MGMT Network (CC)
- StarlingX OAM Network (CC)
- StarlingX SC1 MGMT Network
- StarlingX SC1 OAM Network
- StarlingX SC1 DATA Network
- StarlingX SC2 MGMT Network
- StarlingX SC2 OAM Network
- StarlingX SC2 DATA Network
Get the following L3 Public IPs for the project:
- CC OAM Subnet - Public IPv4/29 subnet
- SC1 DATA Subnet - Public IPv4/29 subnet
- SC2 DATA Subnet - Public IPv4/29 subnet
Setting up the Router-Nat ................................................................
All of the packet.com servers running StarlingX software will use L2-only packet.com networking. A hybrid L3-Networking/L2-Networking server is required in order to route from the StarlingX L2 networks into the packet.com L3 networks and ultimately the public internet.
Get another packet.com server, that supports hybrid L3/L2 networking (e.g. c2.medium.x86) and run ubuntu 16.04 on it.
- Add all the VLANs onto its L2 port
- Add all Public IPv4 /29 subnets to this server
Setup the router-nat server as follows:
apt-get update apt-get install vlan sudo su -c 'echo "8021q" >> /etc/modules' /sbin/reboot
vi /etc/network/interfaces
auto lo iface lo inet loopback # L3 Interface to INTERNET auto bond0 iface bond0 inet static address 147.75.39.198 netmask 255.255.255.252 gateway 147.75.39.197 bond-downdelay 200 bond-miimon 100 bond-mode 4 bond-updelay 200 bond-xmit_hash_policy layer3+4 bond-lacp-rate 1 bond-slaves enp1s0f0 dns-nameservers 147.75.207.207 147.75.207.208 iface bond0 inet6 static address 2604:1380:1:b800::1 netmask 127 gateway 2604:1380:1:b800:: auto bond0:0 iface bond0:0 inet static address 10.99.254.1 netmask 255.255.255.254 post-up route add -net 10.0.0.0/8 gw 10.99.254.0 post-down route del -net 10.0.0.0/8 gw 10.99.254.0 auto enp1s0f0 iface enp1s0f0 inet manual bond-master bond0 # CC OAM Subnet auto enp1s0f1 iface enp1s0f1 inet static address 147.75.105.201 netmask 255.255.255.248 network 147.75.105.200 broadcast 147.75.105.207 # SC1 OAM Subnet auto enp1s0f1.1092 iface enp1s0f1.1092 inet static address 11.11.11.1 netmask 255.255.255.0 network 11.11.11.0 broadcast 11.11.11.255 vlan-raw-device enp1s0f1 # SC2 OAM Subnet auto enp1s0f1.1162 iface enp1s0f1.1162 inet static address 12.12.12.1 netmask 255.255.255.0 network 12.12.12.0 broadcast 12.12.12.255 vlan-raw-device enp1s0f1 # CC MGMT Subnet auto enp1s0f1.1172 iface enp1s0f1.1172 inet static address 192.168.204.1 netmask 255.255.255.0 network 192.168.204.0 broadcast 192.168.204.255 mtu 1400 vlan-raw-device enp1s0f1 # SC1 MGMT Subnet auto enp1s0f1.1117 iface enp1s0f1.1117 inet static address 192.168.214.1 netmask 255.255.255.0 network 192.168.214.0 broadcast 192.168.214.255 mtu 1400 vlan-raw-device enp1s0f1 # SC2 MGMT Subnet auto enp1s0f1.1177 iface enp1s0f1.1177 inet static address 192.168.224.1 netmask 255.255.255.0 network 192.168.224.0 broadcast 192.168.224.255 mtu 1400 vlan-raw-device enp1s0f1 # SC1 DATA Subnet auto enp1s0f1.1145 iface enp1s0f1.1145 inet static address 147.75.105.193 netmask 255.255.255.248 network 147.75.105.192 broadcast 147.75.105.199 vlan-raw-device enp1s0f1 # SC2 DATA Subnet auto enp1s0f1.1179 iface enp1s0f1.1179 inet static address 139.178.66.41 netmask 255.255.255.248 network 147.75.105.40 broadcast 147.75.105.47 vlan-raw-device enp1s0f1
Installing the Central Cloud ........................................................................
SW Install of initial server (cc-controller-0):
In order to IPXE Install the StarlingX Load onto a packet.com server, it must have L3 connectivity.
Setup a new packet.com server for controller-0 (e.g. m1.xlarge.x86), initially with L3 connectivity and initially with an ubuntu 16.04 load.
After the server boots, 'REINSTALL' the packet.com server using 'Custom IPXE' and specifying the following ipxe.conf file:
#!ipxe set base-url http://147.75.38.129/BIOS-Client kernel ${base-url}/vmlinuz console=ttyS1,115200n8 root=live:${base-url}/LiveOS/squashfs.img ip=dhcp ks=${base-url}/pxeboot_controller.cfg boot_device=sda rootfs_device=sda inst.gpt inst.text inst.repo=${base-url} security_profile=standard user_namespace.enable=1 network ksdevice=bootif BOOTIF=01-${netX/mac} initrd ${base-url}/initrd.img imgstat boot
NOTE: I setup a GitHub repo with ipxe.conf files for a StarlingX Standard configuration and a StarlingX All-In-One configuration: https://github.com/gwaines/ipxe-configs
Specify https://raw.githubusercontent.com/gwaines/ipxe-configs/master/ipxe-starlingx-standard-sda.conf as the URL for the ipxe.conf file for the Central Cloud.
The server will reboot a few times, run the StarlingX installer to install the StarlingX load on /dev/sda and then reboot a final time running the StarlingX load.
Change to L2-only Networking:
Bootstrap StarlingX Software:
Configure and Unlock Controller-0
SW Install of second controller (cc-controller-1):
Configure and Unlock Controller-0
Installing the Sub Clouds ............................................................................
SW Install of initial server (cc-controller-0):
Bootstrap StarlingX Software:
Configure and Unlock Controller-0
Problem Tracking
Problems Found with Packet.com:
- Multi-cast Packets appear to be dropped by Packet.com switches
- Results in StartingX Maintenance reporting a false alarm that it has lost connectivity with other nodes in the StarlingX Cloud. E.g. In Central Cloud, cc-controller-0 claims that it has a heartbeat audit failure with cc-controller-1.
- WORKAROUND: Change maintenance's heartbeat failure behaviour to simply raise an alarm ... rather than declare the node as failed and reset the node.
system service-parameter-list system service-parameter-modify platform maintenance heartbeat_failure_action=alarm system service-parameter-apply platform
- On some interfaces, packets between 1400 and 1500 bytes were dropped by packet.com switch, even though MTU of Interface was 1500 bytes.
- WORKAROUND: set all MTUs to 1400 bytes
Problems Found with StarlingX R1.0:
- config_subcloud would fail if prior to running the command on controller-0 of the subcloud, the mgmt interface was already configured and there were any unreachable DNS servers configured in /etc/resolv.conf
- WORKAROUND: ifdown <mgmt-if-dev> and vi /etc/resolv.conf and remove all DNS Server entries
- HORIZON subcloud 'managed' would fail with some error about JSON
- WORKAROUND: Use CLI to set subcloud to managed