While developing high level descriptions of how users might move from "standard" (admittedly trivial) nova-network configurations to neutron, the gaps that would affect developers, deployers and administrators became more "glaring". These are high level points as technical details are covered in various places elsewhere and something short might be of more value at this point.
Usability of L3, Floating IPs and Access to External Networks
Use of floating IPs in nova-network result in the manipulation of firewall rules and the addition of the assigned floating IPs as aliases to the configured "public" interface on the network controller nodes. The consequence of the aliases on the public interface is that these addresses become immediately addressable on the controller node. This more severely affects developers or proof of concept users as the base system requirements with respect to network interfaces and configuration is straightforward. In an all-in-one configuration such as the fairly typical devstack deployment, access to external networks is straightforward and typically immediate. Without a deployment tool, the learning curve to deploying even a simple network configuration with L3 is a tough climb for the uninitiated.
At the moment, neutron's floating IP implementation requires some handiwork with bridging the br-ex interface, some additional iptables forwarding work, or configuration of a provider network. For the non-network oriented, especially those familiar with nova-network, this bit of complexity seems hard to justify. More generally, neutron's usability when doing "simple things' is a non-starter.
Conversations regarding multi-host can become someone circuitious. The Nova-network multi-host implementation makes compromises to obtain certain goals. What it provides is fault-isolation and de-facto load balancing of DHCP and floating IP related processing. By collocating floating IP related changes and DHCP services on the same host as the relevant server instance, issues such as loss of power, network connectivity or software failure on a multi-host node only affects the instances on that multi-host node. The system related load is also contained to that particular multi-host node. This also makes resource scaling easier to conceive as each addtional node adds processing power without additional load on shared services, at least with respect to virtualization and network services. Despite the compromises, the benefits of multi-host are strongly supported by some users. Failing to provide an alternative with the same benefits would be significant obstacle to deprecating nova-network with neutron as a replacement.
Some of the compromises:
- By default, each multi-host node has an addressable bridge interface with its own dnsmasq instance. This results in a consumption of one IP address from the private network address space per multi-host node. An option was added in the grizzly release to allow sharing of the IP address so this is not an issue. (note: as this is not currently set as the default, I wonder if there are trade-offs when enabling it)
- East/west routing of mutliple networks depends on the existence of a server instance for each network on a multi-host node. If a multi-node only has server instances connected to a single network, east-west routing will not work for that node. For consistenct cross-network routing, you need to resort to floating IPs
- Multi-host is a property of a network definition, not a configuration. This is a pervasive choice and makes the multi-host option rather inflexible. Changing a system to or from multihost requires recreating the network and redeploying the instances.
- The networking services on a multi-host node are only available to server instances on that node.
Being aware of these compromises is valuable in that the decision to provide a neutron based solution that has the same compromises without shortcomings with respect to the nova-network feature set could be a win. It is quite likely that some compromises may be mitigated by the flexibility provided by neutron's architecture. Eventually it should be possible to provide the aforementioned benefits without any of the nova-network compromises. That being said, this would likely come with additional complexity.
Some useful references: