Difference between revisions of "Oath FFU Juno To Ocata"
James Penick (talk | contribs) |
Rybridges16 (talk | contribs) (→Upgrade) |
||
| Line 20: | Line 20: | ||
=== Upgrade === | === Upgrade === | ||
| − | # Take cluster snapshot of | + | # Take cluster snapshot of VM status |
| − | # | + | # Snooze or silence any alerting utilities so that you don't get spammed while your cluster is down |
| − | # | + | # Remove the cluster you are about to start upgrading from the dropdown in Horizon. You don't want users hitting the cluster that you are upgrading from Horizon |
| − | # Stop | + | # Stop the API and MQ services. |
| − | # Stop nova compute on all | + | # Stop nova-compute on all hypervisors |
| − | # | + | # Make a full backup of the database before migration just in case anything happens or your upgrade fails for some reason and you want to revert. (We never needed to restore our backup thankfully, but it was nice to have for piece of mind) |
| − | # Run DB migration | + | # Back up your configurations from your old deployment. This is nice to have. We ran into some cases where we missed something and having old working configs as reference was nice. |
| − | # Re-image API and MQ nodes to | + | # Run DB migration scripts |
| − | + | # Re-image your API and MQ nodes if needed. Everyone's operating system requirements are different. We upgraded ours from RHEL6 to RHEL7 during this process. However, this was not required. We could have just as easily left our API and MQ nodes on RHEL6. If you don't need / want to upgrade your operating system as part of this, re-imaging your control plane with the same OS is still a good idea just to make sure all of the old cruft is removed. Your CI/CD pipelines should be able to deploy OpenStack from scratch. | |
| − | + | * We had 3 API nodes per cluster. During the upgrade process, we shut down 2 of the API nodes and left them on Juno. We attempted our upgrade / reimage on the third API node. This was done so that in case anything went wrong with the upgrade, we could bring back the 2 Juno API nodes easily to get our service back up while we figure out what went wrong. | |
| − | + | * We did not re-image our DB hosts | |
| − | # Deploy to | + | # Deploy the code for the release that you are upgrading to. We went from Juno to Ocata. So we deployed the Ocata code directly. We did not deploy each release of the code one by one. We just deployed straight to Ocata. |
| − | + | # Cleanup the DB backups if everything is working | |
| − | + | # Repeat the API node upgrade process for the API nodes that are still on your old revision. | |
| − | # | ||
| − | # | ||
| − | |||
Revision as of 11:19, 26 February 2018
Preparing to upgrade
- Create CMR.
- Finalize CMR steps
- Testing chef bootstrap & converge for api, mq, db, HVs
- Verify runbooks for SRE/PE
- Add new VIPs to LB
- Validate control plane pipeline
- HV preparation
- Upgrade all compute nodes to RHEL 7
- DB preparation
- archive deleted rows
- validate backups
- Verify network ACLs are correct
- Update the Horizon Banner with CMR information
- Add VIP settings in Chef recipe
- Internal announcements on intranet, email, etc
- Build new jumphosts that use openstack-client, rather than the novaclient
Upgrade
- Take cluster snapshot of VM status
- Snooze or silence any alerting utilities so that you don't get spammed while your cluster is down
- Remove the cluster you are about to start upgrading from the dropdown in Horizon. You don't want users hitting the cluster that you are upgrading from Horizon
- Stop the API and MQ services.
- Stop nova-compute on all hypervisors
- Make a full backup of the database before migration just in case anything happens or your upgrade fails for some reason and you want to revert. (We never needed to restore our backup thankfully, but it was nice to have for piece of mind)
- Back up your configurations from your old deployment. This is nice to have. We ran into some cases where we missed something and having old working configs as reference was nice.
- Run DB migration scripts
- Re-image your API and MQ nodes if needed. Everyone's operating system requirements are different. We upgraded ours from RHEL6 to RHEL7 during this process. However, this was not required. We could have just as easily left our API and MQ nodes on RHEL6. If you don't need / want to upgrade your operating system as part of this, re-imaging your control plane with the same OS is still a good idea just to make sure all of the old cruft is removed. Your CI/CD pipelines should be able to deploy OpenStack from scratch.
- We had 3 API nodes per cluster. During the upgrade process, we shut down 2 of the API nodes and left them on Juno. We attempted our upgrade / reimage on the third API node. This was done so that in case anything went wrong with the upgrade, we could bring back the 2 Juno API nodes easily to get our service back up while we figure out what went wrong.
- We did not re-image our DB hosts
- Deploy the code for the release that you are upgrading to. We went from Juno to Ocata. So we deployed the Ocata code directly. We did not deploy each release of the code one by one. We just deployed straight to Ocata.
- Cleanup the DB backups if everything is working
- Repeat the API node upgrade process for the API nodes that are still on your old revision.
