OpenStack - User contributions [en]

StarlingX/Containers/Applications/app-metallb

2025-12-30T18:39:10Z

Mohammad.issa:

= StarlingX MetalLB App installation guide =
app-metallb is a StarlingX containerized application that deploys MetalLB to provide Kubernetes LoadBalancer services for clusters running on bare metal or non-cloud environments.

MetalLB integrates with the StarlingX networking model to advertise service IP addresses using either BGP or Layer 2 (ARP/NDP) mechanisms, enabling external connectivity to Kubernetes services without relying on a cloud provider load balancer.

NOTE: This app will be available starting '''stx.13.0'''

== Source ==
* [https://opendev.org/starlingx/app-metallb Code Repository]
* [https://review.opendev.org/q/project:starlingx/app-metallb Gerrit Reviews]
* [https://metallb.io/ Official Documentation]

== Tarball Package ==
* Get the .tgz file located at
/usr/local/share/applications/helm/
* Or build it yourself using the Debian Build environment
build-pkgs -c -p metallb-helm,python3-k8sapp-metallb,stx-metallb-helm

== Deployment Architecture ==
MetalLB is deployed into the metallb-system namespace and consists of:

* metallb-controller Deployment
* metallb-speaker DaemonSet
* Optional FRR containers when BGP mode is enabled<br />

In BGP mode, the speaker establishes BGP sessions with configured peers and advertises service IPs as host routes.

== Usage ==
* Apply the app
[root@controller-0 sysadmin(keystone_admin)]# system application-apply metallb

* Verify MetalLB controller and speaker pods are running
[root@controller-0 sysadmin(keystone_admin)]# kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
metallb-controller-bbfd4845b-gknmx 1/1 Running 0 75s
metallb-speaker-tfwmz 1/1 Running 0 75s

=== Simple L2 mode example ===
* Apply the configuration for the ''IPAddressPool'' and "L2Advertisement".
[root@controller-0 sysadmin(keystone_admin)]# cat metallb-l2.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: l2-pool
namespace: metallb-system
spec:
addresses: ["192.168.206.200-192.168.206.210"]
avoidBuggyIPs: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: metallb-adv
namespace: metallb-system
spec:
ipAddressPools:
- l2-pool

* Apply a simple nginx service that a host outside the cluster can access
[root@controller-0 sysadmin(keystone_admin)]# cat nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
replicas: 1
selector:
matchLabels:
app: nginx-test
template:
metadata:
labels:
app: nginx-test
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: LoadBalancer
selector:
app: nginx-test
ports:
- port: 80
targetPort: 80

* Notice that the MetalLB controller assigns the nginx service an external IP while the speaker is announcing that IP.
[root@controller-0 sysadmin(keystone_admin)]# kubectl get svc nginx-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service LoadBalancer 10.111.148.132 192.168.206.200 80:30966/TCP 6s

[root@controller-0 sysadmin(keystone_admin)]# kubectl describe svc nginx-service
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 23s metallb-controller Assigned IP ["192.168.206.200"]
Normal nodeAssigned 12s metallb-speaker announcing from node "controller-0" with protocol "layer2"

StarlingX/Containers/Applications/app-metallb

2025-12-30T18:37:14Z

Mohammad.issa:

= StarlingX MetalLB App installation guide =
app-metallb is a StarlingX containerized application that deploys MetalLB to provide Kubernetes LoadBalancer services for clusters running on bare metal or non-cloud environments.

MetalLB integrates with the StarlingX networking model to advertise service IP addresses using either BGP or Layer 2 (ARP/NDP) mechanisms, enabling external connectivity to Kubernetes services without relying on a cloud provider load balancer.

== Source ==
* [https://opendev.org/starlingx/app-metallb Code Repository]
* [https://review.opendev.org/q/project:starlingx/app-metallb Gerrit Reviews]
* [https://metallb.io/ Official Documentation]

== Tarball Package ==
* Get the .tgz file located at
/usr/local/share/applications/helm/
* Or build it yourself using the Debian Build environment
build-pkgs -c -p metallb-helm,python3-k8sapp-metallb,stx-metallb-helm

== Deployment Architecture ==
MetalLB is deployed into the metallb-system namespace and consists of:

* metallb-controller Deployment
* metallb-speaker DaemonSet
* Optional FRR containers when BGP mode is enabled<br />

In BGP mode, the speaker establishes BGP sessions with configured peers and advertises service IPs as host routes.

== Usage ==
* Apply the app
[root@controller-0 sysadmin(keystone_admin)]# system application-apply metallb

* Verify MetalLB controller and speaker pods are running
[root@controller-0 sysadmin(keystone_admin)]# kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
metallb-controller-bbfd4845b-gknmx 1/1 Running 0 75s
metallb-speaker-tfwmz 1/1 Running 0 75s

=== Simple L2 mode example ===
* Apply the configuration for the ''IPAddressPool'' and "L2Advertisement".
[root@controller-0 sysadmin(keystone_admin)]# cat metallb-l2.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: l2-pool
namespace: metallb-system
spec:
addresses: ["192.168.206.200-192.168.206.210"]
avoidBuggyIPs: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: metallb-adv
namespace: metallb-system
spec:
ipAddressPools:
- l2-pool

* Apply a simple nginx service that a host outside the cluster can access
[root@controller-0 sysadmin(keystone_admin)]# cat nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
replicas: 1
selector:
matchLabels:
app: nginx-test
template:
metadata:
labels:
app: nginx-test
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: LoadBalancer
selector:
app: nginx-test
ports:
- port: 80
targetPort: 80

* Notice that the MetalLB controller assigns the nginx service an external IP while the speaker is announcing that IP.
[root@controller-0 sysadmin(keystone_admin)]# kubectl get svc nginx-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service LoadBalancer 10.111.148.132 192.168.206.200 80:30966/TCP 6s

[root@controller-0 sysadmin(keystone_admin)]# kubectl describe svc nginx-service
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 23s metallb-controller Assigned IP ["192.168.206.200"]
Normal nodeAssigned 12s metallb-speaker announcing from node "controller-0" with protocol "layer2"

StarlingX/Containers/Applications/app-metallb

2025-12-30T18:36:39Z

Mohammad.issa:

= StarlingX MetalLB App installation guide =
app-metallb is a StarlingX containerized application that deploys MetalLB to provide Kubernetes LoadBalancer services for clusters running on bare metal or non-cloud environments.

MetalLB integrates with the StarlingX networking model to advertise service IP addresses using either BGP or Layer 2 (ARP/NDP) mechanisms, enabling external connectivity to Kubernetes services without relying on a cloud provider load balancer.

== Source ==
* [https://opendev.org/starlingx/app-metallb Code Repository]
* [https://review.opendev.org/q/project:starlingx/app-metallb Gerrit Reviews]
* [https://metallb.io/ Official Documentation]

== Tarball Package ==
* Get the .tgz file located at
/usr/local/share/applications/helm/
* Or build it yourself using the Debian Build environment
build-pkgs -c -p metallb-helm,python3-k8sapp-metallb,stx-metallb-helm

== Deployment Architecture ==
MetalLB is deployed into the metallb-system namespace and consists of:

* metallb-controller Deployment
* metallb-speaker DaemonSet
* Optional FRR containers when BGP mode is enabled<br />

In BGP mode, the speaker establishes BGP sessions with configured peers and advertises service IPs as host routes.

== Usage ==
* Apply the app
[root@controller-0 sysadmin(keystone_admin)]# system application-apply metallb

* Verify MetalLB controller and speaker pods are running
[root@controller-0 sysadmin(keystone_admin)]# kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
metallb-controller-bbfd4845b-gknmx 1/1 Running 0 75s
metallb-speaker-tfwmz 1/1 Running 0 75s

=== Simple L2 mode example ===
* Apply the configuration for the ''IPAddressPool'' and "L2Advertisement".
[root@controller-0 sysadmin(keystone_admin)]# cat metallb-l2.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: l2-pool
namespace: metallb-system
spec:
addresses: ["192.168.206.200-192.168.206.210"]
avoidBuggyIPs: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: metallb-adv
namespace: metallb-system
spec:
ipAddressPools:
- l2-pool

* Apply a simple nginx service that a host outside the cluster can access
[root@controller-0 sysadmin(keystone_admin)]# cat nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
replicas: 1
selector:
matchLabels:
app: nginx-test
template:
metadata:
labels:
app: nginx-test
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: LoadBalancer
selector:
app: nginx-test
ports:
- port: 80
targetPort: 80

* Notice that the MetalLB controller assigns the nginx service an external IP while the speaker is announcing that IP.
[root@controller-0 sysadmin(keystone_admin)]# kubectl get svc nginx-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service LoadBalancer 10.111.148.132 192.168.206.200 80:30966/TCP 6s

[root@controller-0 sysadmin(keystone_admin)]# kubectl describe svc nginx-service
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 23s metallb-controller Assigned IP ["192.168.206.200"]
Normal nodeAssigned 12s metallb-speaker announcing from node "controller-0" with protocol "layer2"

StarlingX/Containers/Applications/app-rook-ceph

2025-12-30T17:54:45Z

Mohammad.issa:

= StarlingX Rook Ceph App installation guide =
Rook Ceph a orchestrator providing a containerized solution for Ceph Storage with a specialized Kubernetes Operator to automate the management of the cluster. It is an alternative solution to the bare metal Ceph Storage.

== Source ==
* [https://opendev.org/starlingx/app-rook-ceph Code Repository]
* [https://review.opendev.org/q/project:starlingx/app-rook-ceph Gerrit Reviews]
* [https://rook.io/docs/rook/latest-release/Getting-Started/intro/ Official Documentation]

== Tarball Package ==
* Get the .tgz file located at
/usr/local/share/applications/helm/
* Or build it yourself using the Debian Build environment
build-pkgs -c -p rook-ceph-helm,rook-ceph-provisioner-helm,python3-k8sapp-rook-ceph,rook-ceph-floating-monitor-helm,stx-rook-ceph-helm

== Deployment Models ==
The deployment model is the topology strategy that defines the storage backend capabilities of the deployment. The deployment model dictates how the storage solution will look like, defining rules for the placement of the storage cluster elements.

=== Available deployment models ===
Each deployment model works with different deployment strategies and rules to fit different needs. Choose one of the following models according to the demands of your cluster:

==== Controller Model (default) ====
* The OSDs must be added only on hosts with controller personality.
* The replication factor can be configured up to a size of 3.

==== Dedicated Model ====
* The OSDs must be added only on hosts with worker personality;
* The replication factor can be configured up to a size of 3.

==== Open Model ====
* The OSD placement does not have any limitation;
* The replication factor does not have any limitation.

'''IMPORTANT:''' The Open deployment model allows more flexible configurations. Users should ensure they understand the implications of their settings, as responsibility for correct configuration rests with them.

=== Changing the deployment model ===
The deployment models can be changed as long as the system follows the previously estabilished rules

* To change to another deployment model, execute the following command
~(keystone_admin)$ system storage-backend-modify ceph-rook-store -d <desired_deployment_model>

=== Replication factor ===

The replication factor is the number of copies that each piece of data has, spread across the cluster to provide redundancy.

* You can change the replication of an existing Rook Ceph storage backend with the following command

~(keystone_admin)$ system storage-backend-modify ceph-rook-store replication=<size>

Possible Replication Factors on Deployment Models for platforms.

==== AIO-Simplex Controller Model ====
* Default: 1
* Max: 3

==== AIO-Simplex Open Model ====
* Default: 1
* Max: ANY

==== AIO-Duplex Controller Model ====
* Default: 2
* Max: 2

==== AIO-Duplex Open Model ====
* Default: 2
* Max: ANY

==== AIO-Duplex+ or Standard Controller Model ====
* Default: 2
* Max: 3

==== AIO-Duplex+ or Standard Dedicated Mode: ====
* Default: 2
* Max: 3

==== AIO-Duplex+ or Standard Open Model ====
* Default: 2
* Max: ANY

=== Minimum Replication Factor ===

The minimum replication factor is the least number of copies that each piece of data has, spread across the cluster to provide redundancy.

You can assign any number smaller than the replication factor to this parameter. The default value is (replication - 1).

'''Note''': When the replication factor changes, the minimum replication will be readjusted automatically.

* You can change the minimum replication of an existing Rook Ceph storage backend with the command
~(keystone_admin)$ system storage-backend-modify ceph-rook-store min_replication=<size>

=== Monitor, host-fs and controllerfs ===
Ceph monitors are the central nervous system of the Ceph cluster, ensuring that all components are aware of each other and that data is stored and accessed reliably. To properly set the environment for Rook Ceph monitors, some filesystems are needed: host-fs for fixed monitor and controllerfs for the floating monitor.

'''NOTE:''' All changes in "host-fs" and "controllerfs" need a reapply on the application to properly propagate
the modifications in the Rook ceph cluster.

==== Functions ====

The functions parameter contains the Ceph cluster function of a given host. A
"host-fs" can have monitor and OSD functions, a "controllerfs" can only
have the monitor function.

To modify the function of a "host-fs" the complete list of functions desired
must be informed.

'''Examples'''
*host-fs
#(only monitor)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=monitor

#(only osd)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd

#(no function)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=

#(both functions)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd,monitor
*controllerfs
#(only monitor)
~(keystone_admin)$ system controllerfs-modify ceph-float --functions=monitor

#(no function)
~(keystone_admin)$ system controllerfs-modify ceph-float --functions=

==== Monitor Count ====

Monitors (mons) are allocated on all the hosts that have a "host-fs ceph"
with the monitor capability on it.

===== Possible Monitor Count on Deployment Models for Platforms =====

======AIO-Simplex======
* Min: 1
* Max: 1

======AIO-Duplex======
* Min: 1
* Recommended: 3 (using floating monitor)
* Max: 3 (using floating monitor)

======AIO-Duplex+ or Standard======
* Min: 1
* Recommended: 3
* Max: 5

==== Fixed Monitors ====
Fixed monitor is the normal monitor that is associated with a given host. Each fixed monitor requires a "host-fs ceph" properly set and configured on the host.

===== Adding a monitor =====

To add a monitor to a Rook Ceph cluster, the host-fs ceph must be created or have the function 'monitor' added to its capabilities

*When the host has no OSD registered on the platform, you should add a "host-fs ceph" in every node intended to house a monitor. Creating a host-fs this way automatically sets the monitor function. To create a "host-fs ceph", run the following command
~(keystone_admin)$ system host-fs-add <hostname> ceph=<size>

* When there are OSDs registered on a host you should add the monitor function to the existing "host-fs"
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd,monitor

*After the adding the 'monitor' function, the application must be reapplied
~(keystone_admin)$ system application-apply rook-ceph

===== Removing a monitor =====
To remove a monitor, the function 'monitor' must be removed from the capabilities list of the "host-fs ceph"

* When the host has no OSD registered on the platform, remove the 'monitor' function of the "host-fs ceph" with the following command

~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=

*When there are OSDs registered on the same host, only the 'monitor' function should be removed from "host-fs ceph" capabilities list. Use the following command

~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd

*After the removal of the 'monitor' function, the application must be reapplied

~(keystone_admin)$ system application-apply rook-ceph

==== Floating Monitor (only in AIO-Duplex) ====
A Floating monitor is possible and recommended on AIO-Duplex platforms. The monitor
roams and is always allocated on the active controller, providing redundancy and improving stability.

===== Adding the floating monitor =====

'''Note:''' You should lock the inactive controller before add "controllerfs ceph-float" to the platform.

To add the floating monitor follow the procedure

*Lock the inactive controller
~(keystone_admin)$ system host-lock controller-1 (with controller-0 as the active controller)

*With the inactive controller locked, add the controllerfs
~(keystone_admin)$ system controllerfs-add ceph-float=<size>

*Unlock the inactive controller
~(keystone_admin)$ system host-unlock controller-1 (with controller-0 as the active controller)

*With inactive controller unlocked and available, reapply the Rook Ceph Application
~(keystone_admin)$ system application-apply rook-ceph

===== Removing the floating monitor =====

*To remove the floating monitor, the function 'monitor' must be removed from the capabilities list of the "controllerfs ceph-float". Use the following command

~(keystone_admin)$ system controllerfs-modify ceph-float --functions=

* After the removal of the 'monitor' function, the application must be reapplied

~(keystone_admin)$ system application-apply rook-ceph

===== Migrating between AIO-Duplex and AIO-Duplex+ =====

====== Migrating to AIO-Duplex+ ======
To migrate from AIO-Duplex to AIO-Duplex+ the floating monitor must be removed before the migration and new fixed monitor should be added in a worker after the migration is done.

====== Migrating to AIO-Duplex ======
To migrate from AIO-Duplex+ to AIO-Duplex, the fixed monitor should be removed from the cluster before the migration and a floating monitor should be added after the migration is done.

=== Services ===
Services are the storage types (or classes) that provides storage to each pod
with some mount or storage space allocation.

==== Available Services ====

There are four possible services compatible with Rook Ceph. You can combine
them following the rules

* '''block (default)'''
** Not possible to be deployed together with ecblock.
** Will enable the block service in rook, will use cephRBD.

*'''ecblock'''
** Not possible to be deployed together with block.
** Will enable the ecblock service in rook, will use cephRBD.

* '''filesystem (default)'''
** Will enable the ceph filesystem and use cephFS.

* '''object'''
** Will enable the ceph object store (RGW).

'''IMPORTANT:''' A Service cannot be removed or replaced. Services only can be added

==== Adding New Services ====
To add a new service to the storage-backend, first choose a possible service compatible with the aforementioned rules.

* Get the list of the current services of the storage-backend with the following command
~(keystone_admin)$ system storage-backend-show ceph-rook-store

* Then, add the desired service to the list via the command
~(keystone_admin)$ system storage-backend-modify ceph-rook-store --services=<previous_list>,<new_service>

* Reapply Rook Ceph application
~(keystone_admin)$ system application-apply rook-ceph

*Example: For a storage-backend with the service list "block,filesystem", only 'object' can be added as service
~(keystone_admin)$ system storage-backend-modify ceph-rook-store --services=block,filesystem,object

==== Services Parameterization for the Open Model ====
In the 'open' deployment model, no specific configurations are enforced. Users are responsible
for customizing settings based on their specific needs. To update configurations, a Helm override is required.

When applying a helm-override update, list-type values are completely replaced, not incrementally updated.
For example, modifying cephFileSystems (or cephBlockPools, cephECBlockPools, cephObjectStores) via
Helm override will overwrite the entire entry.

Here is two examples of how to change a parameter, using failureDomain as
an example, for '''Cephfs''' and '''RBD'''

*'''Cephfs'''
# Get the current crush rule information
ceph osd pool get kube-cephfs-data crush_rule

# Get the current default values
helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n '/^cephFileSystems:/,/^[[:alnum:]_-]*:/p;' | sed '$d' > cephfs_overrides.yaml

# Update the failure domain
sed -i 's/failureDomain: osd/failureDomain: host/g' cephfs_overrides.yaml

# Get the current user override values ("combined overrides" is what will be deployed)
system helm-override-show rook-ceph rook-ceph-cluster rook-ceph

# Set the new overrides
system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --reuse-values --values cephfs_overrides.yaml

# Get the updated user override values
system helm-override-show rook-ceph rook-ceph-cluster rook-ceph

# Apply the application
system application-apply rook-ceph

# Confirm the current crush rule information
ceph osd pool get kube-cephfs-data crush_rule

* '''RBD'''
# Get the current crush rule information
ceph osd pool get kube-cephfs-data crush_rule

# Retrieve the current values and extract the cephBlockPools section
helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n '/^cephBlockPools:/,/^[[:alnum:]_-]*:/p;' | sed '$d' > rbd_overrides.yaml

# Modify the failureDomain parameter from osd to host in the rbd_overrides.yaml file
sed -i 's/failureDomain: osd/failureDomain: host/g' rbd_overrides.yaml

# Set the update configuration
system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --reuse-values --values rbd_overrides.yaml

# Apply the application
system application-apply rook-ceph

# Confirm the current crush rule information
ceph osd pool get kube-cephfs-data crush_rule
==== Disable Helm Chart ====
'''IMPORTANT:''' Do not disable any of the Rook Ceph Helm charts using "system helm-chart-attribute-modify"
as this may result in a broken installation.

==Installation==
===Preparing the Environment===

====Configuring deployment model and services====
To configure the deployment model, first certify that there is no no ceph-store storage backend configured on the system
~(keystone_admin)$ system storage-backend-list

Create a storage backend for Rook Ceph, choosing your deployment model (controller, dedicated, open), and, optionally, the desired services (block or ecblock, filesystem, object)
~(keystone_admin)$ system storage-backend-add ceph-rook --deployment controller --confirmed

Create a host-fs ceph for each host that will house a Rook Ceph monitor (preferably an '''ODD''' number of hosts)
~(keystone_admin)$ system host-fs-add <hostname> ceph=<size>

Configuring OSDs

*Check the uuid of the disks of the desired host that will house the OSDs
~(keystone_admin)$ system host-disk-list <hostname>
Note: The OSD placement should follow the chosen deployment model placement rules.
*Add the desired disks to the system as OSDs (Preferably an EVEN number of OSDs)
~(keystone_admin)$ system host-stor-add <hostname> osd <disk_uuid>

===Installing the application===
After configuring environment acordingly to the chosen deployment model correctly, Rook Ceph should install automatically.

== Verifying the Installation ==
* Verify that rook is running
[sysadmin@controller-0 ~(keystone_admin)]$ NS=rook-ceph
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n ${NS} -o wide -w

NAME READY STATUS RESTARTS AGE IP NODE
csi-cephfsplugin-provisioner-86d7d5d9b7-98gn2 6/6 Running 0 11m 172.16.192.101 controller-0
csi-rbdplugin-nd8xk 3/3 Running 0 11m 192.168.206.2 controller-0
csi-rbdplugin-provisioner-c9cd4ffd7-mmfp9 6/6 Running 0 11m 172.16.192.100 controller-0
rook-ceph-mds-kube-cephfs-a-7b544db87d-4ctrf 1/1 Running 0 11m 192.168.206.2 controller-0
rook-ceph-mds-kube-cephfs-b-5fcbb478fd-qjw4g 1/1 Running 0 11m 192.168.206.2 controller-0
rook-ceph-mgr-a-69fbb9d7b4-tcbt4 1/1 Running 0 11m 192.168.206.2 controller-0
rook-ceph-mon-a-55bbd66678-9cvt6 1/1 Running 0 12m 192.168.206.2 controller-0
rook-ceph-operator-5ddc68f467-kqbwt 1/1 Running 0 12m 172.16.192.95 controller-0
rook-ceph-osd-0-bfbf89745-679kp 1/1 Running 0 11m 192.168.206.2 controller-0
rook-ceph-osd-1-69799854f5-qvgvg 1/1 Running 0 11m 192.168.206.2 controller-0
rook-ceph-osd-prepare-controller-0-jmxmz 0/1 Completed 0 11m 192.168.206.2 controller-0
rook-ceph-provision-g8qfg 0/1 Completed 0 5m20s 172.16.192.104 controller-0
rook-ceph-tools-644f4dbc4b-8qc2m 1/1 Running 0 12m 192.168.206.2 controller-0
stx-ceph-manager-66764cb49d-hxt4d 1/1 Running 0 11m 172.16.192.98 controller-0
stx-ceph-osd-audit-28484346-r5nfq 0/1 Completed 0 8s 192.168.206.2 controller-0

* Verify that the cluster is operational
[sysadmin@controller-0 ~(keystone_admin)]$ ceph -s

* Verify the ceph version
[sysadmin@controller-0 ~(keystone_admin)]$ceph version
ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)

* Make sure that the disk is reported used
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
+-----------+---------+---------+-------+------------+-----+--------------------------------------------+
| device_no | device_ | device_ | size_ | available_ | rpm | device_path |
| de | num | type | gib | gib | | |
+-----------+---------+---------+-------+------------+-----+--------------------------------------------+
| /dev/sda | 2048 | SSD | 520.0 | 0.0 | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
| /dev/sdb | 2064 | SSD | 520.0 | 0.0 | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
| /dev/sdc | 2080 | SSD | 520.0 | 519.996 | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
+-----------+---------+---------+-------+------------+-----+--------------------------------------------+

* Make sure that the volume group is reported
[sysadmin@controller-0 ~(keystone_admin)]$ system host-lvg-list controller-0
+-------------------+-------------+--------+------------------+------------------+-------------+-------------+
| LVG Name | State | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
+-------------------+-------------+--------+------------------+------------------+-------------+-------------+
| cgts-vg | provisioned | wz--n- | 488.406 | 261.593 | 1 | 15 |
| ceph-2e5d9569-... | provisioned | wz--n- | 519.996 | 0.0 | 1 | 1 |
+-------------------+-------------+--------+------------------+------------------+-------------+-------------+

* Make sure that the physical volume is reported
[sysadmin@controller-0 ~(keystone_admin)]$ system host-pv-list controller-0 --nowrap
+-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
| lvm_pv_name | disk_or_part | disk_or_part_device_path | pv_state | pv_type | lvm_vg_name |
| | _device_node | | | | |
+-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
| /dev/sdb | /dev/sdb | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 | provisnd | disk | ceph-2e5d... |
| /dev/sda5 | /dev/sda5 | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0-part5 | provisnd | partition | cgts-vg |
+-------------+--------------+--------------------------------------------------+----------+-----------+--------------+

== OSD Management ==

===Add OSDs===
* Obtain the UUIDs of the target disk(s).

~(keystone_admin)$ system host-disk-list compute-0

*Add the target disk(s) as OSDs.

~(keystone_admin)$ system host-stor-add compute-0 ed4c51a7-372f-4460-915d-43049cf23dd5
~(keystone_admin)$ system host-stor-add compute-0 ab35233a-9952-4437-87ed-80cc4d6817d7

* At this point, the OSDs will be in the configuring-with-app state.

~(keystone_admin)$ system host-stor-list compute-0

*Apply rook-ceph application and wait for the application to become applied.

~(keystone_admin)$ system application-apply rook-ceph

The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.

Use the system application-show rook-ceph command to monitor progress.

* List the OSDs and verify that they are now in the configured state.

~(keystone_admin)$ system host-stor-list compute-0

===Remove OSDs===
*Obtain the UUIDs of the OSDs to be removed.

~(keystone_admin)$ system host-stor-list compute-0

*Mark the OSDs for removal.
~(keystone_admin)$ system host-stor-delete 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397

*At this point, the OSDs will be in the deleting-with-app state.
~(keystone_admin)$ system host-stor-list compute-0

* Apply rook-ceph application and wait for the application to become applied.

~(keystone_admin)$ system application-apply rook-ceph

The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.

Use the system application-show rook-ceph command to monitor progress.

Wait until the OSDs are removed.

*The OSDs will be removed asynchronously after rook-ceph reaches the applied state. Then, once the OSDs entry disappears from system host-stor-list, the OSDs are successfully removed.
~(keystone_admin)$ system host-stor-list 3

===Replace OSDs===

Repeat the following steps for each target host, one at a time so that it does not impact Ceph availability.

Follow the steps in the Remove OSDs section to remove the target OSDs from the host.

'''Note:''' If after several minutes the OSD entry is reverted to the configured state, then the OSD removal is considered unsafe and is aborted to avoid data loss.

You can proceed using system host-stor-delete --force <uuid>. This will skip data integrity checks and will forcefully remove the OSD. However, this may cause loss of data. Thus, use his only when you are sure that there is enough replicas of data. Otherwise, before using the –force option, always attempt the standard deletion first.

* Lock the host

~(keystone_admin)$ system host-lock <host>

* After the host is locked, power off the host

~(keystone_admin)$ system host-power-off <host>

Replace the disk(s) while the host is powered off

Power on the host and wait for the host to become online.

*Unlock the host

~(keystone_admin)$ system host-unlock <host>

Wait for the host to become unlocked/available.

Follow the steps in Add OSDs to add the new replaced disk(s) as OSDs.

==Additional Features and Procedures==

===Add new OSDs on a running cluster===
To add new OSDs to the cluster, just add the new OSD to the platform and reapply the application
~(keystone_admin)$ system host-stor-add <host> <disk_uuid>
~(keystone_admin)$ system application-apply rook-ceph

===Add new monitor on a running cluster===
To add a new monitor to the cluster, just add the host-fs to the desired host and reapply the application
~(keystone_admin)$ system host-fs-add <host> ceph=<size>
~(keystone_admin)$ system application-apply rook-ceph

===Enabling Ceph Dashboard===
To enable Ceph Dashboard a helm override must be provided before the application apply. You should provide a password coded in base64.

*Create the override file
$ openssl base64 -e <<< "my_dashboard_passwd"
bXlfZGFzaGJvYXJkX3Bhc3N3ZAo=

$ cat << EOF >> dashboard-override.yaml
cephClusterSpec:
dashboard:
enabled: true
password: "bXlfZGFzaGJvYXJkX3Bhc3N3ZAo="
EOF
*Apply the override file
~(keystone_admin)$ system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --values dashboard-override.yaml

===Check Rook Ceph pods===
You can check the pods of the storage cluster running the following command
kubectl get pod -n rook-ceph

===Ceph Health Status Filter===
Some Ceph health statuses can be filtered to avoid generating alarms. The detection of a particular Health error or warning can be disabled.
'''IMPORTANT:''' Disabling the detection of any health error or warning can prevent the system from generating alarms, detecting issues and log generating. This feature must be used at user's discretion. The recommended is to use this feature temporarily during some analysis or procedure and then revert back to the default empty values.
There is two filters: '''health_filters_for_ignore''' for filtering at any time (always active) and '''health_filters_for_upgrade''' that applies the filter only during an upgrade of the Rook Ceph.
*To apply the always-on filter (health_filters_for_ignore), use the following procedure:
Check for the name of any Ceph health issues that might wanted to be filtered out
~(keystone_admin)$ ceph health detail
Consult the list of the Ceph health issues currently ignored:
~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed -n '/ceph-manager-config.yaml:/,/^[^ ]/p' | sed -n 's/^[ ]*health_filters_for_ignore:[ ]*//p'
Edit the config map adding the name of the all Ceph health issues comma separated and delimited by [] to the list "health_filters_for_ignore"
#Examples of useful Health Status to ignore: OSD_DOWN,PG_AVAILABILITY,BLUESTORE_SLOW_OP_ALERT,DB_DEVICE_STALLED_READ_ALERT
~(keystone_admin)$ health_filters='[<ceph_health_status_1>,<ceph_health_status_2>]'
~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed "s/^$\s*health_filters_for_ignore:\s*$.*/\1$health_filters/" | kubectl apply -f -
Restart the stx-ceph-manager pod
~(keystone_admin)$ kubectl rollout restart -n rook-ceph deployment stx-ceph-manager

*To use the upgrade only filter (health_filters_for_upgrade), follow the same procedure but change commands for consult and edit for the following versions:
Consult command for only upgrade
~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed -n '/ceph-manager-config.yaml:/,/^[^ ]/p' | sed -n 's/^[ ]*health_filters_for_upgrade:[ ]*//p'

Edit command for only upgrade
~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed "s/^$\s*health_filters_for_upgrade:\s*$.*/\1$health_filters/" | kubectl apply -f -

==Uninstall Rook Ceph==
To completely remove Rook Ceph, you must remove the application and clear all the environment configurations to prevent an automatic reinstall.

'''Note:''' Before removing the Rook Ceph application(s), unmount all the persistent volumes associated with the Rook Ceph cluster. This is achieved by removing all the applications dependent on StorageClasses provided by the Rook Ceph application. After removing the application(s), delete all PVCs and PVs that were previously provided to the application(s).

If dependent application removal is not desired, then minimal cleanup is needed. All pods with PVCs must be deleted, scaled down to zero, or edited removing the PVC volume mounts. Otherwise, Rook Ceph application removal will fail, volumes may hang, and nodes may require a restart. In this case, a --force option is needed to remove the Rook Ceph application and tear down the Ceph cluster.

===Uninstalling===
To completely remove Rook Ceph, you must remove the application and clear all the
environment configurations to prevent an automatic reinstall.

'''NOTE:''' Before removing the Rook Ceph application(s), unmount all the persistent
volumes associated with the Rook Ceph cluster. This is achieved by removing
all the applications dependent on StorageClasses provided by the Rook Ceph
application. After removing the application(s), delete all |PVCs| and |PVs|
that were previously provided to the application(s).

If dependent application removal is not desired, then minimal cleanup is
needed. All pods with |PVCs| must be deleted, scaled down to zero, or
edited removing the |PVC| volume mounts. Otherwise, Rook Ceph application
removal will fail, volumes may hang, and nodes may require a restart. In
this case, a ``--force`` option is needed to remove the Rook Ceph
application and tear down the Ceph cluster.

1) Remove the application by running the script

source /etc/platform/openrc
system application-remove rook-ceph --force
retry_count=1
retries=200
while [ $retry_count -le $retries ]; do
rookstatus=$(system application-list | grep rook-ceph | awk '{print $10}')
echo $rookstatus
<nowiki>if [[ "$rookstatus" == "uploaded" ]]</nowiki>; then
system application-delete rook-ceph
break
fi
echo "Retry #" $retry_count
let retry_count++
done

2) Remove OSDs
*2.1) List all OSDs to get the uuid of each OSD
~(keystone_admin)$ system host-stor-list <hostname>
*2.2) Remove each OSD using the uuid of all OSDs
~(keystone_admin)$ system host-stor-delete <uuid>

3) (AIO-Duplex Only) Remove controllerfs
*3.1) To remove a controllerfs, the standby controller must be locked and the controllerfs needs to be in Ready state.
~(keystone_admin)$ system host-lock <hostname>
*3.2) Remove the controllerfs
~(keystone_admin)$ system controllerfs-delete ceph-float
*3.3) Unlock the standby controller
~(keystone_admin)$ system host-unlock <hostname>

4) Remove the host-fs
~(keystone_admin)$ system host-fs-delete <hostname> ceph
5) Remove the storage backend ceph-rook
~(keystone_admin)$ system storage-backend-delete ceph-rook-store --force

StarlingX/Containers/Applications/app-metallb

2025-12-30T17:51:05Z

Mohammad.issa: Created page with "= StarlingX MetalLB App installation guide ="

= StarlingX MetalLB App installation guide =

StarlingX/Containers/Applications

2025-12-30T17:49:58Z

Mohammad.issa:

= Containerized Applications =
This page serves a launch pad for all StarlingX Containerized application bundles
== Requirements ==
* '''''STX.APP.01''''' - StarlingX shall provide application compatibility for all k8s releases supported in the planned StarlingX Release
** Currently supported versions can be seen here: https://opendev.org/starlingx/integ/src/branch/master/kubernetes
* '''''STX.APP.02''''' - StarlingX shall use the latest helm chart(s) and container image release(s) in an effort to stay current from a feature perspective and to include the latest CVE fixes available upstream.
* '''''STX.APP.03''''' - StarlingX applications shall support all application framework lifecycle operations exercised with the '''system application-xxx''' API/CLI.
* '''''STX.APP.04''''' - StarlingX applications shall support all application framework lifecycle operations exercised with the '''system system helm-override-xxx''' API/CLI.
* '''''STX.APP.05''''' - StarlingX applications shall support all application framework lifecycle operations exercised with the '''system helm-chart-attribute-modify''' API/CLI.
* '''''STX.APP.06''''' - StarlingX applications shall be updatable without operator intervention from one application version to another.
* '''''STX.APP.07''''' - StarlingX applications shall support automatic updates when a new version is delivered to the platform via a platform patch.
** This is controlled via the [https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/common/app_metadata.py#L40 application metatdata] packaged with each version of the application
* '''''STX.APP.08''''' - StarlingX applications shall support orchestrated updates over a platform release upgrade, if a new version of the application is provided in the new release.
** Currently this is automated via a platform [https://opendev.org/starlingx/config/src/branch/master/controllerconfig/controllerconfig/upgrade-scripts/65-k8s-app-upgrade.sh upgrade-activation script] ([https://opendev.org/starlingx/docs/src/branch/master/doc/source/sample_apps corresponding repo])
* '''''STX.APP.09''''' - StarlingX applications shall support orchestrated downgrades (i.e., updates to a previous version) over a platform release rollback of an upgrade, if the platform upgrade had resulted in an application upversion.
** This will be automated via a platform upgrade-activate-rollback script.
* '''''STX.APP.10''''' - StarlingX applications shall support orchestrated updates over a k8s upgrade if a new version of the application is available on the platform and when a specific version k8s is enabled.
** This is controlled via the [https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/common/app_metadata.py#L46 application metatdata] packaged with each version of the application
* '''''STX.APP.11''''' - StarlingX applications should be documented in the StarlingX Documentation explaining the basic purpose, functionality, limitations, and tested use cases for each version of the application available.
** Based on the current limited documentation, application documentation should be added [https://docs.starlingx.io/sample_apps/index-sample-apps-15fce86ad4bf.html here] ([https://opendev.org/starlingx/docs/src/branch/master/doc/source/sample_apps code repo])
* '''''STX.APP.12''''' - StarlingX applications should maintain a [[StarlingX/Containers/Applications#Active_Applications|StarlingX Wiki page]] documenting source code location, build instructions, basic functionality, limitations, and testing instructions. The content of the Wiki page should track to the latest content delivered to the master branch.
* '''''STX.APP.13''''' - StarlingX applications should strive to use unmodified upstream helm charts and integrate into StarlingX with only static/dynamic helm overrides.
* '''''STX.APP.14''''' - StarlingX applications that need helm chart modifications to properly integrate into StarlingX should upstream those helm chart modifications so that in the future these customizations can be dropped from the product.
* '''''STX.APP.15''''' - StarlingX applications should be tested in simplex and multi-node environments to ensure the pods/replicas are deployed as desired
* '''''STX.APP.16''''' - StarlingX applications should be tested in multi-personality environments to ensure the pods/replicas are deployed as as desired on k8s master+worker(AIO) and master(STD controller) + worker(worker) nodes
* '''''STX.APP.17''''' - StarlingX applications should be tested over all MTC operations ensure the pods/replicas are operational/scaled/recovered as as desired on all nodes. MTC operations include: install/reinstall/lock/unlock/swact/power-off/reboot
* '''''STX.APP.18''''' - To support proper CPU affining StarlingX application pods/namespaces should be labeled with '''app.starlingx.io/component=platform''' or '''app.starlingx.io/component=application'''
** We will standardize helm charts as specified [https://wiki.openstack.org/wiki/StarlingX/Containers/Applications/AppIntegration#Parameterizing_the_Component_Label here].
* '''''STX.APP.19''''' - StarlingX applications should be compliant with our use of the Pod Security Admission Controller
* '''''STX.APP.20''''' - StarlingX follows the [https://governance.openstack.org/tc/reference/opens.html Four Opens] and must provide software this is appropriately licensed. Applications that rely on and integrate upstream technologies (helm charts/images), must have a compatible license for inclusion in StarlingX. This implies that as applications are maintained and updated, licenses must be re-checked for compliance.
** Known list of issues that have impacted applications
*** [https://redis.com/blog/redis-adopts-dual-source-available-licensing/ Redis Adopts Dual Source-Available Licensing | Redis]

== Active Applications ==
These applications are currently under development and actively maintained
* [[StarlingX/Containers/Applications/app-dell-storage | app-dell-storage]]
* [[StarlingX/Containers/Applications/app-harbor | app-harbor]]
* [[StarlingX/Containers/Applications/app-intel-device-plugins | app-intel-device-plugins]]
* [[StarlingX/Containers/Applications/app-intel-ethernet-operator | app-intel-ethernet-operator]]
* [[StarlingX/Containers/Applications/app-istio | app-istio]]
* [[StarlingX/Containers/Applications/app-kubernetes-power-manager | app-kubernetes-power-manager]]
* [[StarlingX/Containers/Applications/app-kubevirt | app-kubevirt]]
* [[StarlingX/Containers/Applications/app-metallb | app-metallb]]
* [[StarlingX/Containers/Applications/app-node-feature-discovery | app-node-feature-discovery]]
* [[StarlingX/Containers/Applications/app-node-interface-metrics-exporter | app-node-interface-metrics-exporter]]
* [[StarlingX/Containers/Applications/app-oran-o2 | app-oran-o2]]
* [[StarlingX/Containers/Applications/app-power-metrics | app-power-metrics]]
* [[StarlingX/Containers/Applications/app-rook-ceph | app-rook-ceph]]
* [[StarlingX/Containers/Applications/app-security-profiles-operator | app-security-profiles-operator]]
* [[StarlingX/Containers/Applications/app-sriov-fec-operator | app-sriov-fec-operator]]
* [[StarlingX/Containers/Applications/app-sts-silicom | app-sts-silicom]]
* [[StarlingX/Containers/Applications/app-audit | audit-armada-app]]
* [[StarlingX/Containers/Applications/app-cert-manager | cert-manager-armada-app]]
* [[StarlingX/Containers/Applications/app-metrics-server | metrics-server-armada-app]]
* [[StarlingX/Containers/Applications/app-nginx-ingress-controller | nginx-ingress-controller-armada-app]]
* [[StarlingX/Containers/Applications/app-oidc-auth | oidc-auth-armada-app]]
* [[StarlingX/Containers/Applications/app-openstack | openstack-armada-app]]
* [[StarlingX/Containers/Applications/app-ceph | platform-armada-app]]
* [[StarlingX/Containers/Applications/app-portieris | portieris-armada-app]]
* [[StarlingX/Containers/Applications/app-ptp-notification | ptp-notification-armada-app]]
* [[StarlingX/Containers/Applications/rook-ceph-migration | rook-ceph (migration)]]
* [[StarlingX/Containers/Applications/app-snmp | snmp-armada-app]]
* [[StarlingX/Containers/Applications/app-vault | vault-armada-app]]

== Inactive Applications ==
These applications are currently NOT maintained and their functionality/usability is unknown
* monitor-armada-app
* SDO-rv-service
== Developer Resources ==
* [[StarlingX/Containers/Applications/HowToAddNewFluxCDAppInSTX | Create a new application]]
* [[StarlingX/Containers/Applications/AppIntegration | Application Integration with the Platform]]

== Page Archive ==
* [[StarlingX/Containers/Applications/Archive/HowToAddNewArmadaAppInSTX | How To Add an Armada App]]
* [[StarlingX/Containers/Application/Archive/ArmadaAppCodeStructure | Armada App Code Structure]]
* [[StarlingX/Containers/Application/Archive/ConvertingArmadaAppsToFluxCD | Armada to FluxCD Conversion ]]