Difference between revisions of "Zero downtime upgrade"
| Line 8: | Line 8: | ||
Assume that Heat is deployed in the Hight-Availability mode and there are two nodes for heat-api, heat-engine and other heat components. During the upgrade, following are the possible upgrade artifacts other than python modules: | Assume that Heat is deployed in the Hight-Availability mode and there are two nodes for heat-api, heat-engine and other heat components. During the upgrade, following are the possible upgrade artifacts other than python modules: | ||
| − | # API REQUEST RESPONSE version change | + | # '''API REQUEST RESPONSE version change''' : This is completely maintained by using the API version like v2.0, v3. |
| − | # Message version change | + | # '''Message version change''' : currently API and backend components are maintaining this using RPC version. and oslo.message package provides some sort of backward compatibility on the minor versions of the messages using RPC_VERSION in rpc clients and services. |
| − | # Db version change. | + | # '''Db version change''' : For each major release of openstack, a particular version of the db model is getting delivered. And db_sync tools is provided to upgrade/downgrade the db model versions. But there is no coordination between application ORM and the db model version, which is causing the system to go down during the upgrade/downgrade while the system is in HA as well. More details on how to handle the Db zero down time upgrade is elaborated in this document. |
| − | |||
| − | More details on how to handle the Db zero down time upgrade is elaborated in this document. | ||
== Zero down time Db upgrade == | == Zero down time Db upgrade == | ||
Heat is provided with db migration tool and each Heat major release is provided with given db version. Followings are the db version for the past Heat releases: | Heat is provided with db migration tool and each Heat major release is provided with given db version. Followings are the db version for the past Heat releases: | ||
| − | + | * Juno: 46 | |
And to support zero down time upgrade, following assumptions need to be in place: | And to support zero down time upgrade, following assumptions need to be in place: | ||
| Line 23: | Line 21: | ||
* Downward compatibility should be provided with last 2-3 versions from the current version(inclusive) | * Downward compatibility should be provided with last 2-3 versions from the current version(inclusive) | ||
* DB migration tool take cares of upgrading/downgrading of db data with out loss. | * DB migration tool take cares of upgrading/downgrading of db data with out loss. | ||
| + | * Data is not lost while the db is upgraded from lower version to higher version. | ||
Once these assumptions are in place, we should be able to start the migration as follows | Once these assumptions are in place, we should be able to start the migration as follows | ||
| Line 29: | Line 28: | ||
When these engines are upgraded to juno, followings would be the steps to be followed: | When these engines are upgraded to juno, followings would be the steps to be followed: | ||
| − | + | # Mark the engine-A and engine-B are in maintenance mode | |
| + | # Upgrade engine-A and engine-B to juno | ||
| + | # run db_sync | ||
| + | # Start engine-A and engine-B | ||
| + | |||
| + | This will make sure that all current request will be executed without aborting it | ||
Revision as of 04:04, 21 October 2014
DRAFT COPY
On the enterprise public cloud, while upgrading openstack to a given latest version, the downtime should be zero (0). This wiki page provides all related information to this requirement.
- Etherpad Entry: https://etherpad.openstack.org/p/Zero_downtime_upgrade
Introduction
Assume that Heat is deployed in the Hight-Availability mode and there are two nodes for heat-api, heat-engine and other heat components. During the upgrade, following are the possible upgrade artifacts other than python modules:
- API REQUEST RESPONSE version change : This is completely maintained by using the API version like v2.0, v3.
- Message version change : currently API and backend components are maintaining this using RPC version. and oslo.message package provides some sort of backward compatibility on the minor versions of the messages using RPC_VERSION in rpc clients and services.
- Db version change : For each major release of openstack, a particular version of the db model is getting delivered. And db_sync tools is provided to upgrade/downgrade the db model versions. But there is no coordination between application ORM and the db model version, which is causing the system to go down during the upgrade/downgrade while the system is in HA as well. More details on how to handle the Db zero down time upgrade is elaborated in this document.
Zero down time Db upgrade
Heat is provided with db migration tool and each Heat major release is provided with given db version. Followings are the db version for the past Heat releases:
- Juno: 46
And to support zero down time upgrade, following assumptions need to be in place:
- Downward compatibility should be provided with last 2-3 versions from the current version(inclusive)
- DB migration tool take cares of upgrading/downgrading of db data with out loss.
- Data is not lost while the db is upgraded from lower version to higher version.
Once these assumptions are in place, we should be able to start the migration as follows
Assume the scenario where heat is deployed with heat-engine-A and heat-engine-B and ice-house version of heat is used. When these engines are upgraded to juno, followings would be the steps to be followed:
- Mark the engine-A and engine-B are in maintenance mode
- Upgrade engine-A and engine-B to juno
- run db_sync
- Start engine-A and engine-B
This will make sure that all current request will be executed without aborting it
