|
|
| (2 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| | = Manila = | | = Manila = |
| | | | |
| − | This is a page about networking for Manila (OpenStack Shared Filesystem Service) | + | This is a page about networking for Manila (OpenStack Shared File Systems) |
| | | | |
| − | = Protocols = | + | == Overview of Choices == |
| | | | |
| − | Currently we support:
| + | There are many options for how to create your network configuration in OpenStack and with Manila. |
| − | * NFS
| + | Below are some examples of choices an administrator needs to make and what kind of impact thay have. |
| − | * CIFS
| |
| | | | |
| − | We intend to support more though. Possible future protocols include:
| + | === L2 vs L3 connectivity === |
| − | * GlusterFS | + | * L2 - share servers have network interface created on a tenant-owned subnet |
| − | * GPFS | + | ** Direct access to storage by VMs on the same subnet |
| − | * Ceph FS | + | ** Requires network segmentation setup (VLAN, VXLAN) |
| − | * Dropbox (or similar) | + | ** Requires a driver capable of creating share-servers for each tenant |
| | + | ** Can potentially cause broadcast domains to span racks or the whole datacenter |
| | + | * L3 - share servers only accessible across a router |
| | + | ** Works with any driver |
| | + | ** Works with controllers that don't support the SVM/share-server concept |
| | + | ** Limits broadcast domains |
| | + | ** Requires virtual or physical router |
| | + | ** For creation of share-servers, manila needs to take IPs from existing subnets, and in some cases create new subnets/routers |
| | | | |
| − | = Backends = | + | === Share servers === |
| | + | * Created per tenant |
| | + | ** Private network |
| | + | *** Most secure |
| | + | *** Works with generic driver, and some hardware-based drivers |
| | + | ** Public network |
| | + | *** Still provides high level of security |
| | + | *** Accomodates storage controllers that don't understand vlans/vxlans |
| | + | *** Acommodates lab configurations with physical switches not controlled by the developer |
| | + | * Common |
| | + | ** Security enforced by the storage controller rather than the network |
| | + | ** Manila doesn't need to manage any network resources |
| | | | |
| − | Currently we've implemented two backends:
| + | === Network management === |
| − | * Linux LVM | + | * Neutron |
| − | * NetApp (7-mode and C-mode) | + | ** Most capable, many features: flat, VLAN, VXLAN, GRE, etc |
| | + | ** Not widely deployed by end users |
| | + | ** This is where the manila project started |
| | + | * Nova-net |
| | + | ** Support both flat and VLAN based networking |
| | + | ** Widely used |
| | + | * Other Plugins |
| | + | ** Minimize dependencies on external components |
| | + | ** Simple plugin for development/test, not for production |
| | + | ** Possible for end user customized plugins |
| | | | |
| − | We're looking to other vendors to implement backends for their hardware/software.
| + | == Plans == |
| | | | |
| − | = Frontends =
| + | Our top level goal is to provide flexibility to administrators so they can use Manila in their environment no matter what other choices they've made. |
| | + | The tradeoff is how to do that without overwhelming them with complexity. |
| | | | |
| − | How the end user actually interacts with the filesystems provided for Manila is an interesting question. Consider the following approaches:
| + | For Icehouse and Juno, we stuck with a relatively narrow set of optionsː Neutron only, always create a share server for every tenant, and prefer L2 networking over L3. These choices were made because they provide maximum security and work well in a public cloud type environment. |
| | | | |
| − | === Network Plumbing ===
| + | The problem we face is that developers often can't recreate the complex network configuration that exist in public clouds in their development labs, and users who want to experiment with Manila face a high bar to setup a basic configuration. |
| | | | |
| − | The approach here is to interact with Neutron to ensure that the server(s) providing the storage are connected to the tenant's private network. This means that by the time an attach operation completes, a network link has been configured so that the client(s) specified in the attach operation can send packets to the server(s) returned hosting the storage.
| + | For Kilo we need to expand the set of choices to make developer's lives easier and to make a simple proof-of-concept style deployment less difficult to achieve. |
| − | | |
| − | In this mode, clients can directly interact with the storage server using the storage protocol that they intended (i.e. CIFS or NFS). The storage server must be responsible for enforcing security such that clients are not able to see resources that haven't been attached to them, and more importantly, tenants must not see data owned by other tenants. On the controller side this may be accomplished by virtualizing the storage server (The NetApp vserver approach), by using firewalls between the clients and server, or by using rules built into the storage server itself (rules in /etc/exports or /etc/smb.conf, etc).
| |
| − | | |
| − | Advantages:
| |
| − | * No performance bottlenecks introduced
| |
| − | * It feels natural to the end user, and applications qualified against CIFS/NFS in the physical work are more likely to work without issues in the cloud
| |
| − | * Users can take advantage of special CIFS/NFS features if they are available
| |
| − | * It pushes the hard security problems down into the drivers
| |
| − | * It does not depend on Nova
| |
| − | * It can work with bare metal clients
| |
| − | | |
| − | Disadvantages:
| |
| − | * It relies on Neutron, creating a (possibly unwanted) dependency
| |
| − | * It pushes the hard security problems down into the drivers
| |
| − | | |
| − | For security to work in this approach, each tenant will have to provide some details before they can connect to any shares. Each tenant must provide these details a minimum of one time, but possibly more than once (if the tenant has multiple security domains). The details will depend on the protocols being used. Some examples: | |
| − | * Active Directory domain and credentials (CIFS)
| |
| − | * LDAP and/or Kerberos domain and credentials (NFS)
| |
| − | | |
| − | === Hypervisor Mediated ===
| |
| − | | |
| − | This approach involves using some special technology to virtualize the filesystem through to the guest. The one we've looked at most closely is VirtFS.
| |
| − | | |
| − | An attach operation under this approach would simply involve attaching the storage to the compute node rather than the ultimate user (as far as the backend is concerned). An additional sequence of actions would have to take place in the Manila manager to complete the attachment all the way through to the guest, which would include callbacks to nova to configure the needed VirtFS steps.
| |
| − | | |
| − | Advantages:
| |
| − | * This approach most closely mirrors how guests interact with storage controllers in Cinder. The storage controller only needs to know about compute nodes and the hypervisor deals with the problem of presenting the storage to the guest.
| |
| − | * Security is enforced by the hypervisor. Backends need not support any security to protect tenant data from other tenants (or even enforce protection between instances of one tenant).
| |
| − | * No dependence or interaction with Neutron
| |
| − | | |
| − | Disadvantages:
| |
| − | * Requires a hypervisor (will not work w/ bare metal)
| |
| − | * Hypervisor support limited (only KVM and Xen)
| |
| − | * Guest OS support limited (Only Linux and BSD)
| |
| − | * User sees a Plan9 filesystem instead of a NFS or CIFS filesystem which may prevent access to features
| |
| − | * Depends on adding some new features to Nova
| |
| − | * It may be hard to preserve ACLs on the files if people care about that (for example UID/GID/mode bits to enforce filesystem security in multiuser environments)
| |
| − | * VirtFS may be immature and not well maintained (this is an opinion)
| |
| − | * Possible performance bottleneck in VirtFS itself, although benchmarks suggest that VirtFS can be even faster than running NFS over a virtual NIC
| |
| − | | |
| − | === Gateway Mediated ===
| |
| − | | |
| − | The theory of the gateway mediated approach is that the hypervisor mediated (VirtFS) approach is desirable, but we want to overcome the limitations of VirtFS itself. One way to do this is to take advantage of the hypervisor to add an additional virtual NIC through which the guest OS may access its shared filesystems.
| |
| − | | |
| − | The virtual NIC would implement half of a point to point tunnel, the other half being at the storage gateway. Under normal circumstances, the storage gateway would run on the compute node alongside nova. In this configuration, backends would still attach storage to the compute node just like the VirtFS case, but the gateway would simply tunnel filesystem access protocols straight through, such that the guest could use a regular NFS or CIFS client, connecting to special tunnel-specific IP addresses, and the gateway would do any firewalling and NAT necessary to connect the client with the real backend while preventing the client from seeing data it shouldn't.
| |
| − | | |
| − | Advantages:
| |
| − | * This approach also closely mirrors how guests interact with storage controllers in Cinder
| |
| − | * Security is enforced by the gateway, in the form of spoofing-prevention and firewalling of packets from the client. Backends could implement only IP-based security.
| |
| − | * No dependence or interaction with Neutron
| |
| − | * Works with any hypervisor (unlike VirtFS)
| |
| − | * Work with any guest (unlike VirtFS)
| |
| − | * Users can take advantage of special CIFS/NFS features if they are available (with some exceptions)
| |
| − | * It feels natural to the end user, and applications qualified against CIFS/NFS in the physical work are more likely to work without issues in the cloud (with some exceptions)
| |
| − | | |
| − | Disadvantages:
| |
| − | * Requires a hypervisor (will not work w/ bare metal)
| |
| − | * Depends on adding some new features to Nova
| |
| − | * The gateway can become a performance bottleneck
| |
| − | * The extra NIC may seem strange to end users, especially if it uses magic IP for the 2 ends of the tunnel
| |
| − | * The NAT aspect of the gateway could interfere with protocol features
| |
| − | * Security is limited to methods that can work with 1-way communication (UNIX/system security for NFS, workgroup security for CIFS)
| |
| − | | |
| − | === Flat Network ===
| |
| − | | |
| − | Included for completeness, this is what Manila implements today. There will always be scenarios where the administrator is either doing something way more complicated that what we can imagine, or the administrator is happy with a simple flat network and minimal security. In this case, connectivity between the ultimate users of the storage and the backends is left up to somebody else, and Manila is only responsible for provisioning and exporting storage to a client without regard to what's on the other end.
| |
| − | | |
| − | Advantages:
| |
| − | * It's very simple from an implementation standpoint (it's done)
| |
| − | * It allows compatibility with unanticipated future designs
| |
| − | * It may be the preferred approach for single tenant scenarios like private clouds
| |
| − | | |
| − | Disadvantages:
| |
| − | * Cannot support multi-tenant scenarios like public cloud
| |
| − | | |
| − | == Plugins ==
| |
| − | | |
| − | Given the advantages and disadvantages of the different approaches above, it seems clear that no one approach will be ideal for all circumstances. I believe a plugin system is needed so that the administrator can choose what he wants to use in his own environment.
| |
| − | | |
| − | = Security =
| |
| − | | |
| − | As I mentioned in the network plumbing section, using filesystem security requires some configuration work up-front.
| |
| − | | |
| − | In the of CIFS, the user needs to tell Manila what domain (active directory) the CIFS share should be part of, if any. The backend may need to create a new virtual storage controller and join it to the domain, which means the user needs to provide relevant information, such as DNS server IP, domain controller IP, domain administrator account name and password, etc. In theory this could be a one time operation, separate from the storage provisioning operation, but in the case where a tenant has 2 different domains and wants to provision into both of them, then individual provisioning requests need to either include all of the above details or a reference to a previously created object. In both cases, Manila will need to store the details so it can tell whether new requests can be placed on a controller that's already joined the domain, or whether a new controller needs to be set up.
| |
| − | | |
| − | NFS has similar requirements, if Kerberos is used. The user needs to provide Manila with enough details that the backend can join the Kerberos and LDAP domains, which means IP addresses of those services, or DNS and hostnames for those services, as well as credentials for an administrative account so the controller can join the domains.
| |
| − | | |
| − | = Comments =
| |
| − | == Caitlin Bestler ==
| |
| − | My analysis is similar, but structured a bit differently.
| |
| − | | |
| − | There are three fundamental objectives that must be achieved by any mediation scheme:
| |
| − | * Disambiguation of User IDs (as well as Roles/Groups/etc.), Security Domain X UID 500 must never be confused with Security Domain Y UID 500.
| |
| − | * Restrict mounting of shares to being within Security Domains. All access to Security Domain X shares must be approved by a Security Domain X specified authentication server (probably LDAP or AD).
| |
| − | * Disallow cross-security domain access, even with forged mount handles. A would-be pirate within Security Domain X must not be able to access Security Domain Y file servers by just faking an IP address. It is not enough to have a Security Domain X login, the VM or port you are using must be enabled to access Security Domain X.
| |
| − | | |
| − | The different solutions can be summarized as:
| |
| − | * Network mediation: Each security domain is placed on an isolated logical network (whether by tunneling or VLANs). The NAS servers are expected to provide a virtual interface for each security domain network.
| |
| − | * Disambiguation of User identifiers is the responsibility of the virtual server. Manila does not care how the server appliance does this.
| |
| − | * A virtual server associated with Security Domain X can only export file shares owned by Security Domain X.
| |
| − | * Neutron will only allow ports that are admitted to Security Domain X to use that logical network.
| |
| − | * The NAS Client is in the Guest OS (or user space).
| |
| − | * Hypervisor-mediated with virtFS
| |
| − | * Virtual File System operations are serialized over the virtualization bus, so the file system is really running in the Hypervisor or a designated domain.
| |
| − | * This file system acts as a proxy for accessing the true storage servers.
| |
| − | * It must map Security Domain specific user identifiers to universal identifiers.
| |
| − | * It may only allow same-domain mounts.
| |
| − | * This requires adding functionality to the Guest. Therefore it cannot be the *only* solution.
| |
| − | * Hypervisor-mediated: with proxy
| |
| − | * The Hypervisor runs NAS proxies (either itself or in designated domains)
| |
| − | * A NAS client runs in each Guest (OS or user).
| |
| − | * The proxy otherwise must do the same translations as with the Hyerpvisor-mediated-virtFS solution.
| |
| − | * At worst case the Guest must be configured to access a different file server, therefore this can work without modifying Guest code (only guest configuration).
| |
| − | * Transparent proxying can eliminate even that, but might not be worth the effort.
| |
There are many options for how to create your network configuration in OpenStack and with Manila.
Below are some examples of choices an administrator needs to make and what kind of impact thay have.
Our top level goal is to provide flexibility to administrators so they can use Manila in their environment no matter what other choices they've made.
The tradeoff is how to do that without overwhelming them with complexity.
For Icehouse and Juno, we stuck with a relatively narrow set of optionsː Neutron only, always create a share server for every tenant, and prefer L2 networking over L3. These choices were made because they provide maximum security and work well in a public cloud type environment.
The problem we face is that developers often can't recreate the complex network configuration that exist in public clouds in their development labs, and users who want to experiment with Manila face a high bar to setup a basic configuration.
For Kilo we need to expand the set of choices to make developer's lives easier and to make a simple proof-of-concept style deployment less difficult to achieve.
