NovaOrchestration
Long Running Transactions in Nova
What is a long running transaction? A long running transaction is a business activity that may have to wait for a long period of time between steps. For example, if you are a new employee with a company the HR department might have to perform the following operations before your first day of work:
- allocate you a parking place
- get you an access card
- add you to the HR system
- get you set up with health care and benefits
- find you a desk, chair, etc
- determine whom you will report to
- order you a PC
- etc, etc.
There could be a long period of time from the start to the end of each task. Each task in itself might be composed of many other long-running transactions. Also, many of these tasks might be able to be performed in parallel, which means we have to be able to fork many sub-transactions and join them together at some point before continuing on.
During the time between the process starting and ending a multitude of IT events may have occurred. Servers may have died, new servers added, databases changed, power failures, etc. We can't assume that a conventional "process" is robust enough to handle these long running operations.
Nova has a number of long running transactions that it needs to manage. Most importantly the provisioning of instances. Consider a request to provision 1000 instances. We have to do the following steps:
- Talk to all of the zones and come up with a build plan for the request.
- Delegate the "provision" operation to each host in each zone to create the instance.
- Wait for the provisioning to occur on all of the hosts.
- If a request fails, retry the request on another host (from the build plan)
- Periodically create a new build plan with fresher data.
- If all of this takes too long we may need to cancel the transaction, notifying the requester.
From the time this operation starts to the time it fails (or completes), the scheduler that started the request could have died and restarted a dozen times. We need to be able to watch this transaction as an outside observer and "orchestrate" the transaction over time.
