Latest revision as of 20:05, 12 May 2015

Intro

We considered multiple approaches for addressing mount automation. The thinking has come a long way since we originally considered the problem 2+ years ago, but we still don't have a "silver bullet" approach that works in a wide variety of application and has no drawbacks.

Questions

We decided to evaluate the various approaches using the follow measurements:

• Is the approach limited to a subset of scenarios?
• Does the approach allow for instantaneous/synchronous mounting?
• Does the approach allow for persistent mounts that survive across reboots?
• Can the approach handle the case when the client is down at the time of attach/detach?
• What security considerations exist?
• What network considerations exist?
• Are there any external dependencies?
• Does the approach require Manila to store any state?
• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?
• How much effort is required to implement the approach?
• How resistant are users likely to be to the approach?
• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

The hardest thing about this feature, aside from the fact that there are no silver bullet solutions, is that the feature doesn't allow users to do anything that wasn't already possible. For the most part, this is about simplifying something which can be done already (with the exception of the DR use case, mentioned above).

Approaches Summary

Approaches considered:

• SSH/PsExec (very well understood approach we've considered from the beginning)
• Agent - write a custom agent to solve the problem (also well understood and old idea)
• Zeroconf-based advertisements - a new idea that came from the Paris summit
• AutoFS - leverage autofs to solve the problem on UNIX
• LDAP/AD - insert mount information into the directory service and let clients pull it from there
• Cloud-Init - piggyback on cloudinit (a shippable demo is available for this that will be shown at the Liberty summit)
• VirtFS - virtfs gives us the closest model to cinder, and attaches/detaches make significantly more sense in this context
• Do nothing, and prescribe recipes how to automate mounts with existing tools, such as Heat.

Approaches

SSH/PsExec

The simplest and most widely usable approach for automating mounts is to use management software that already exists in the OS. For UNIX, this is SSH. For Windows, it would be the various forms of Windows remote-management such as PsExec (PsExec is an SSH-like tool that works out of the box on nearly all Windows machines) or PowerShell.

The idea here is to give Manila the root password, root key, or Administrator creds for all of your clients, and to let Manila reach out and poke them to trigger them to mount or unmount shares at the appropriate times.

The value of this approach is that it has no dependencies on anything other than standard OS facilities and is therefore likely to be widely compatible.

• Is the approach limited to a subset of scenarios?

good No.

• Does the approach allow for instantaneous/synchronous mounting?

good Yes.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

bad No.

• What security considerations exist?

bad Tenants have to hand over to Manila administrator credentials for all of the guests for which they want Manila to automate mounts. This can create a huge trust barrier.

• What network considerations exist?

bad Manila will need network access directly to every client for which its automating mounts. This can be easy in some cases and near impossible in other cases.

• Are there any external dependencies?

caution Manila needs an SSH client (easy) and a PsExec client (hard). PsExec is a Windows binary, which could be ported to UNIX, but most likely at considerable cost.

• Does the approach require Manila to store any state?

caution Yes. Manila needs to store credentials for the clients, and also information about IP addresses and networks to find the clients.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. If Manila can push mount operations at will, then automatically updating mounts in the DR failover case becomes possible.

• How much effort is required to implement the approach?

caution For UNIX-only, medium. For Windows, significantly more if we want to undertake something with PsExec.

• How resistant are users likely to be to the approach?

good It's optional, but some users may choose not to use this feature based on the requirement to hand overs root creds to Manila.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila would needs to become aware of clients and their networks. This is currently out of scope for Manila.

Agent

Writing an agent gives us ultimate flexibility. Agents can listen for requests from Manila, they can pull updates from Manila periodically or at boot, and they can perform ANY change we like on the clients, at any time. Many problems becomes simple when you can assume the existence of an agent on the client.

There are obvious downsides to this approach, chiefly the trust issue that most users will have about installing a new piece of software on their guests, and the significant effort involved with creating a new piece of software and maintaining it. There are also security concerns, because we will be responsible if a bug in the agent software exposes a user to mischief and mayhem.

• Is the approach limited to a subset of scenarios?

good No.

• Does the approach allow for instantaneous/synchronous mounting?

good Yes.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

• What security considerations exist?

bad We get to design our own security. This gives us excellent flexibility, but we have to design something that strikes a good balance between protecting users and being usable, which is always hard. Regardless, we will face trust problems until we have a proven track record.

• What network considerations exist?

caution The network considerations are similar to the SSH approach, because Manila will need to reach out and poke the agent. Also the agent will need a reverse path back to the Manila server to do pull requests, which might be desirable at boot time.

• Are there any external dependencies?

caution The external dependency is the agent itself. We cannot force anyone to install it, so in cases where it doesn't exist, we will be unable to do anything.

• Does the approach require Manila to store any state?

caution Yes. Manila will need to track the agents and some security information in order to contact them and authenticate with them.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes.

• How much effort is required to implement the approach?

bad Very high.

• How resistant are users likely to be to the approach?

bad Very resistent. It's another thing to install and another thing to security audit. Another thing to worry about.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Not only does Manila need to become aware of clients and client networks, but now Manila needs to create a whole new project and maintain that software as well.

Zeroconf-based advertisements

Zeroconf is an interesting approach for advertising services on networks, which layers on top of DNS (or mDNS in peer to peer environments). Zeroconf has existing support for advertisement of CIFS share and NFS exports, but there is no evidence that modern operating systems react to these advertisements automatically by (for example) mounting them.

Zeroconf also only solves half the problem of mount automation. It can notify clients of the existence of something they can mount, but it doesn't tell them precisely WHERE to mount it. For some scenarios, a random mount point might be acceptable (VDI) but for others, it would be a dealbreaker (database as a service).

It's unlikely that we could address unmounting with this approach.

• Is the approach limited to a subset of scenarios?

bad Yes.

• Does the approach allow for instantaneous/synchronous mounting?

caution Maybe.

• Does the approach allow for persistent mounts that survive across reboots?

caution Maybe.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

• What security considerations exist?

good Possibly none.

• What network considerations exist?

bad Zeroconf relies heavily on multicast, which rarely crosses router boundaries (the admin has to configure multicast routing). In order to get the advertisements onto the right networks, Manila would need to perform some weird gymnastics, most likely involving service VMs with lots and lots of network ports.

• Are there any external dependencies?

bad Yes. Zeroconf is not universally available in guest images or enabled.

• Does the approach require Manila to store any state?

caution Yes. Manila would need to know which clients to advertise to and on which networks.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No. It's unlikely that an advertisement-based approach would allow for the fast unmount-move-mount semantics needed to cleanly failover.

• How much effort is required to implement the approach?

bad Unknown, but not small. There are probably existing libraries we can use to do zeroconf-based-advertising, but all of the service VM and networking stuff required to make it work will still be considerable effort.

• How resistant are users likely to be to the approach?

good Very little. The approach is not invasive at all, aside from the possibly scary service VM that attaches to thousands of networks to send advertisements.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila needs to know about clients and networks to be able to advertise to them. Also, we need to maintain the software image that goes on the advertising service VM. This would require creating a new GitHub repo and maintaining a new project.

AutoFS

Many UNIX users already use autofs to control large numbers of NFS mounts. The advantage to autofs is that you can make a single change to a file on a server, and suddenly thousands of clients can mount something new.

One downside is that autofs doesn't actually mount stuff immediately. It's an event-based system that relies on just-in-time mounting on the client side, so that if the client tries to access the mount point before it's mounted, it asks the server what is the correct thing to mount and mounts it. This creates real problems with changes to mounts or unmounts, because they basically become impossible without a reboot or some other approach described elsewhere in this document.

The other downside is that autofs is extremely flexible. Different users probably use it in very different ways and a single approach probably won't meet all users' needs without some customization by the user.

• Is the approach limited to a subset of scenarios?

bad Yes. It would be UNIX only, and only work if autofs was in use, and only if autofs was configured in a way that it was compatible with Manila.

• Does the approach allow for instantaneous/synchronous mounting?

bad Not really. The just-in-time approach is instantaneous, except that mount doesn't actually happen until its needed.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

• What security considerations exist?

caution Manila needs write access to the place where the autofs tables live. This is likely to vary from case to case, but it would be sensitive information and access to it would always be protected.

• What network considerations exist?

good None.

• Are there any external dependencies?

caution Yes, autofs would be needed.

• Does the approach require Manila to store any state?

good Possibly not. Manila could simply push updates to autofs and then forget about them, and the right thing would happen.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad Probably not. Autofs is not friendly to changes, because once the client has its mount it doesn't go back and pull changes from the server anymore.

• How much effort is required to implement the approach?

good Fairly low. It's just some code that tweaks and autofs table. The effort gets higher if we try to get clever and support lots of ways of using autofs.

• How resistant are users likely to be to the approach?

good Very low. Users like this idea, but they probably haven't thought through all the downsides.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No. This feature can be implemented on the side without significantly changing Manila itself.

LDAP/AD

Directory services create a centralized place for clients to get information. A list of mounts could be one kind of information stored in a directory service. For Windows, the AD server tells each user where his home directory is, for example. On UNIX, LDAP can be a similar repository, but coupled with autofs for mounting.

We haven't determined 100% whether these tools are flexible enough out of the box, but they provide a reasonable building block that could be combined with some scripts to facilitate mount automation. That being said, this approach has enough drawbacks that we didn't consider it worthwhile to dig deeper.

• Is the approach limited to a subset of scenarios?

bad Yes. Not all environments have LDAP/AD.

• Does the approach allow for instantaneous/synchronous mounting?

bad No.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

• What security considerations exist?

caution Manila needs write access to the LDAP/AD server. In theory we already have this with the security service that's part of the share network, but not all clients actually use that feature, so it would make setting up security services a prerequisute.

• What network considerations exist?

good None.

• Are there any external dependencies?

caution Yes, the LDAP/AD server. Also software to communicate with those servers from Manila.

• Does the approach require Manila to store any state?

good No.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No.

• How much effort is required to implement the approach?

caution Medium. Storing some records in a server is not super hard. The challenge would be getting client software to talk to those services.

• How resistant are users likely to be to the approach?

good Pretty low. It's not invasive and it piggybacks on stuff they already do.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No, not unless we need to write some new software to communicate with AD or LDAP.

Cloud-Init

Clout-init is widely available in OpenStack clouds (but nowhere near universally available). It runs at startup and pulls identity information from a directory service. We could store our mount information in that directory and make Cloud-Init perform the mounts for us.

This is similar to an agent based approach, except the agent is owned by someone else and only runs at startup time, so listening for requests wouldn't be possible.

As a slight variation, a script inserted by Cloud-Init could get mount information from the Nova metadata service, which would not store Manila data but would query Manila for the latest share mounts on demand.

• Is the approach limited to a subset of scenarios?

caution Yes. Cloud-Init is common, but by no means universal.

• Does the approach allow for instantaneous/synchronous mounting?

bad No.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

• What security considerations exist?

caution Manila needs access to modify the directory that Cloud-Init pulls from, or the service invoked by Cloud-Init (i.e. Nova metadata service) needs Manila credentials). However, this is a single trust relationship which should be easy to configure.

• What network considerations exist?

good None.

• Are there any external dependencies?

caution Dependency on Cloud-Init.

• Does the approach require Manila to store any state?

good No.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good No (unless clients were rebooted as part of the same DR failover).

• How much effort is required to implement the approach?

good Fairly low. We prototyped this (Nova metadata service + Cloud-Init) in a week.

• How resistant are users likely to be to the approach?

good Fairly low. Users that like Cloud-Init will find this to be a natural expansion.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

VirtFS

VirtFS is hypervisor controlled and extremely similar to the Cinder/block-based approach insofar as the guest simply sees a PCI PnP event and reacts to it. Manila would only need to communicate with Nova and we could safely ignore the guest.

The downsides to VirtFS are well known -- chiefly the lack of support in Windows guests, lack of support in certain hypervisors, and the lack of support for bare metal.

• Is the approach limited to a subset of scenarios?

bad Yes, only situations where VirtFS is used.

• Does the approach allow for instantaneous/synchronous mounting?

good Yes.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Undefined. Like Cinder, there is no concept of a volume attached to an instance that's not currently running.

• What security considerations exist?

good None.

• What network considerations exist?

good None.

• Are there any external dependencies?

bad VirtFS. And something to manage the virtfs/hypervisor setup -- Nova would be the obvious thing to do this.

• Does the approach require Manila to store any state?

good No.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. In fact this approach is by far the best approach for making DR seamless.

• How much effort is required to implement the approach?

good Low. In this scenario we really don't do much of anything in Manila because all of the VirtFS work would be in Nova anyways.

• How resistant are users likely to be to the approach?

caution Mixed. Some love virtfs, others hate it. For cases where it's possible, it would be a very attractive. For cases where it's not possible, who cares?

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

Do Nothing

The idea here is to recognize that mount automation is a non-feature. Nothing we add in Manila will enable anything you can't already do manually, and there is little point to making Manila try to simplify a task which can already be simplified more effectively using existing tools.

Consider a workflow that uses Heat to create a Manila share and some Nova instances to provide a service of some kind. Heat has all the of information it needs to perform the mount itself, and Heat is more than capable of automating these kinds of things. Customers may prefer to do this themselves, because they'll get extra flexibility and won't be locked into a narrow approach implemented inside Manila.

If we were to pursue this approach, we would focus on providing concrete examples to users for how to leverage tools like Heat to automate mounts, and we would add these examples to the documentation.

There are no obvious downsides to this approach, other than that it feels like we're sidestepping the problem.

• Is the approach limited to a subset of scenarios?

good No.

• Does the approach allow for instantaneous/synchronous mounting?

good Yes.

• Does the approach allow for persistent mounts that survive across reboots?

good Yes.

• Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

• What security considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need authentication creds, but such tools need those anyways, so they are in many ways best suited to solve the security problems.

• What network considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need network access to the client, but such tools need that anyways, so they are in many ways best suited to deal with network complexity.

• Are there any external dependencies?

good No real dependencies. You need to bring your own automation and the dependencies are whatever you bring.

• Does the approach require Manila to store any state?

good No.

• Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No, not unless the DR failover is also automated by the same process automating the mounts.

• How much effort is required to implement the approach?

good Very little. This is largely a documentation exercise.

• How resistant are users likely to be to the approach?

good Not resistant at all, since this choice gives them maximum flexibility. Some users may be disappointed that there is no magic button, but that was fantasy anyways.

• Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

Plugin Approach

Based on the mixed advantages and disadvantages of the above approaches, and based on the likelihood that other people will have other ideas for how to tackle this problem, we recommend tackling this problem by building a "plugin" framework in Manila that enables mount automation with a variety of approaches.

We will assume that all of the above approaches will eventually be implemented by various people in the community, and the goal of the core feature is to provide the basic primitives to allow all of the approaches to work. Which approaches get used in practice will depend on choices made by individual deployers or tenants.

The plugin approach will need to provide a mechanism for "attachments" between clients and shares to be tracked and for the software responsible for performing mounts and unmounts to query that mechanism and to be notified of changes.

Some questions to be answered by the design:

• What are the types of things that shares can be attached to? IPs? Networks? Users? Nova Instances?
• When does a share become "attached"? When access is granted? When a separate "attach" API is called?
• Who is the consumer of the attachment information/notifications? The administrator? The tenants? Both?

The plugin design should include enough detail that we can be confident that ALL of the proposed approaches could be accommodated, however we won't actually design all of the above approaches. We will start with designs for 1 or 2 highly popular approaches (to prove the approach and make it actually usable) and encourage the community to expand on the others.

Assumed highly popular approaches:

• Cloud-Init w/ Nova metadata
• Nova agent