MoveIpAllocation
- Launchpad Entry: NovaSpec:move-ip-allocation
- Created: 10/31/10
- Contributors: Vish Ishaya
Summary
Ip allocation currently occurs at the api layer. This is not the best place for ip allocation to occur because it forces the api to know to much about the topology of the network. This spec is most likely a precursor to VirtualNetworks.
Release Note
Ip allocation was moved down the stack. This means that api requests to launch an instance no longer return the ip of the instance. Once the instance has been allocated an ip, the ip will be returned in any api call that gets instance data.
Rationale
Nova supports various network topologies. In large deployments, the ip address may be specific to the host machine the instance is running on, or the cluster that the host machine is a part of. The api does not have this information and should not need to have it. Additionally, there is no need for the api command to run an instance to block waiting for an ip address to be allocated.
User stories
This is more of a provider feature than a user feature. It allows providers to have more flexibility in designing the network for their infrastructure
Assumptions
Design
Current Implementation
Currently, there are three strategies for networking, implemented by different managers: FlatManager -- ip addresses are grabbed from a network and injected into the image on launch. All instances are attached to the same manually configured bridge. FlatDHCPManager -- ip addresses are grabbed from a network, and a single bridge is created for all instances. A dhcp server is started to pass out addresses VlanManager -- each project gets its own vlan, bridge and network. A dhcpserver is started for each vlan, and all instances are bridged into that vlan.
The implementation of creating bridges, vlans, dhcpservers, and firewall rules is done by the driver linux_net. This layer of abstraction is so that we can at some point support configuring hardware switches etc. using the same managers.
My goal with the Manager refactor was to move all of the code relating to each component into the Manager classes. For example, all code relating to networks/ips would be done by NetworkManager. That said, code relating to businesss objects needs to be run by three separate workers, and not all of it has been encapsulated in the manager. For example, the initial creation of the record for instances/volumes is done by nova-api by hitting the db directly instead of using a method on the manager.
Here is some horrible ascii art of what each component does during run instance. The manager's methods are called directly for local setup, and they are called via rpc to nova-network which runs the manager wrapped by a service which exposes public methods to rpc. The *d items aren't specifically related to networking but help make it clear when things occur
+----------------+ +----------------+ +----------------+
| nova-api | | nova-network | | nova-compute |
-----------------+ +----------------+ +----------------+
(direct manager) (service) (direct manager)
*create instance in db
find correct network host
(get_network)
call (if no host) -----> sets up network
(set_network_host)
<----------------------- returns host
allocates fixed ip
(allocate_fixed_ip)
cast-------------------> sets up fixed ip
V (setup_fixed_ip)
cast-----------(goes through scheduler)---------> (ComputeManager.run_instance)
| sets up compute network
| (setup_compute_network)
| * launches instance
| sets up security groups
V (done by compute driver)
*return instance data
This all works and allows us to return an ip very quickly from an api call, but it does have some issues. Most importantly, in some network models, the api server may not have enough information to know how to allocate an ip. For example, in the flat mode, we may want to have a cluster of compute nodes associated with a particular network host, and get_network returns a network based on the compute host instead of the project as we do in vlan mode.
New Implementation
The allocation of the ip should move further down the stack like so:
+----------------+ +----------------+ +----------------+ | nova-api | | nova-network | | nova-compute | -----------------+ +----------------+ +----------------+ (direct manager) (service) (direct manager) *create instance in db check for available network resources (check_free_ips?) cast-----------(goes through scheduler)---------> (ComputeManager.run_instance) | find correct network host | (get_network) | sets up network <------- call (if no host) | (set_network_host) | -----------------------> returns host | sets up compute network | (setup_compute_network) | allocates fixed ip <---- call (because we're not in a rush) | (allocate_fixed_ip) | (setup_fixed_ip) | -----------------------> returns fixed_ip | * launches instance | sets up security groups V (done by compute driver) *return instance data (with no ip address)
Implementation
The implementation itself should be fairly straightforward. It means moving the calls to network to compute manager.
Proof of Concept
At least one implementation using a different networking model should be tried. Either FlatNetwork can be modified to allocate ips from a network associated with the host instead of a project, or a new networking model can be created to support clusters and one network per cluster.
Migration
This shouldn't require data migration, but it does need to be made clear to users that their ip address will not be available immediately on return from api calls. They will need to poll the api to find the ip address.
Unresolved issues
At some point, it would be useful to have some kind of webhook that can be specified so that end-users have an alternative to constantly polling the api system. Webhooks are useful for more than just ip allocation, so they should be addressed in a separate blueprint
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
