• VNCConsoleCleanup

VNC Console Cleanup

* DRAFT * This document outlines a plan for consolidating nova's vnc console code and apis.  At the moment, there are two partial and inconsistent implementations for instance vnc console access, each of which is hypervisor specific.  For essex, the goal is to create a consistent, hypervisor-independent way for users to access vnc consoles. To enable a variety of use cases, we aim to support two types of clients:

• XVP-like vnc clients (explained below)
• html5 websockets (ie novnc)

In each case, the design must enable the wire protocol between client and proxy to be the same regardless of hypervisor.

The State of the Current System

For libvirt, vnc consoles are supported only through a websocket proxy.  It uses a simple expiring token based auth system.  The present proxy, based on eventlet, has poor browser support.

For xen, vnc consoles are supported through a console system that backends to XVP (Xenserver VNC Proxy).  The system manages access to consoles by manipulating the configuration of an xvp conf file.  Additionally, this system uses db storage to manage consoles, pools, and other concepts found in xvp.  The underlying api/manager/driver mechanic, however, is generic, but possibly a bit more than we need since the required functionality could also be added to the compute api/manager/driver (generally, the compute driver is required to retrieve all the info we need to set up a vnc connection).

The XVP Auth Protocol

XS uses a simple auth protocol to control access to its consoles, which I'll refer to as the XVP auth protocol. The XVP auth protocol is essentially a normal vnc connection preceded with a quick http handshake.  The handshake works like this:

• Client sends an HTTP CONNECT that contains a path like /consoles?[credentials string]
• The server validates the auth info, if valid, return HTTP/1.1 200 OK\r\n\r\n
• The server sets up a vnc proxy

For XS, the credentials string requires 2 components:

• The OpaqueRef of the instance whose console we require

• A valid session_id

Both of these pieces of information can be retrieved by accessing the XenAPI python library and stored on nova's backend to facilitate client connections. Note that any xen sessions that are set-up for vnc consoles will have to be cleaned up.

The Proposal Part I: Java client parity using XVP Auth

Since the XVP auth protocol is simple, it is easy to make a python server that can interpret an HTTP CONNECT to authenticate and create a vnc proxy.  For essex, the proposal is to use the XVP protocol as a standard auth mechanism for java client vnc authentication (for both libvirt and xs).

Client Compatibility

The client for this is assumed to be based off of Citrix's java vnc client, though a variant of TightVNC should be similarly easy to implement.

Supporting Services

vncproxy

The role of the vncproxy is threefold:

• to verify client auth tokens

• to transparently handle the on-wire differences between XenServer and KVM

• to bridge between the public and management networks networks.
  • Each hypervisor is attached to some management network, and isn’t attached to the public internet. Therefore, _all_ VNC traffic has to go through a proxy, because it’s bridging between the two networks.

Each protocol implementation will use its own proxy, and for Essex the following two are planned:

• nova-xvpvncproxy - proxy for the java client
• nova-wsvncproxy - proxy for websockets

There can be many proxy services running simultaneously as a cluster. To enable this, the functions of token validation and management will be separated into its own service(nova-consoleauth).

nova-consoleauth

This is a nova manager that stores token and connection information for console sessions. Like other managers, it communicates to interested services over the nova message bus. It's internal api consists of the following calls:

• get_connection_details(token) - called by proxies to validate client tokens. A null return value would imply a non-existent or expired token. This would return a connection_details dict with these keys:
  • instance_id
  • console_type - xvp-vnc, ws-vnc (ajax?)
  • token - random token, not associated with internals
  • host - host where vnc server can be found
  • port - port of vnc server
  • access_url - public url to be shared with users/clients for console access
  • internal_access_url - internal auth params
  • updated_at - can be used by compute drivers to flush out old sessions
  • created_at - when the entry was created
• add_connection_details(instance_id, console_type, token, host, port, access_url, internal_access_url)
• get_connection_details_by_instance(instance_id, include_idle=True, include_expired=False) - used by compute to find consoles for an instance (in case one already exists)
• get_connection_details_by_host(host_id, only_idle=True) - used by compute hosts to find consoles that may need to be garbage collected
• touch_connection_details([tokenA, tokenB…]) - sets connection_details.updated_at = now
• delete_connection_details([tokenA, tokenB…]) - allows drivers to delete connections.

NOTE: connection_details are said to be 'expired' when now - created_at > TOKEN_LIFE.

NOTE: connection_details are said to be 'idle' when now - updated_at > IDLE_TIMEOUT.

vncproxy servers will will not honor expired tokens, which means that there is a narrow window of time for users to use access_urls. Once a proxy connection is established, vncproxy will call touch_connection_details for active connections once every X seconds or so. Specific drivers may determine how to handle idle connections. XS, for example, will need to actively close idle sessions before deletion.

Connection Process A: Retrieving a client connection url from the api

Story: Bob wants to access an instance vnc console, so he uses novaclient to authenticate with nova and retrieve a connection url.

Details:

• novaclient calls the OS api to request a console
• The OS API forwards the request to compute.api
• The compute.api calls out to the compute host to retrieve hypervisor specific connect_info.  The compute.api then generates a token and casts/calls the token and connect_info to nova-consoleauth
  • Note: The nova-consoleauth is a simple nova manager that proxies can query when they need to validate a client's token.  This functionality is presently in nova-vncproxy, and would be separated out into its own service.

  • Note: The specific information that nova-consoleauth would store includes: host, port, access_url (what is ultimately sent to clients), and an internal_access_path.  The internal_access_path can be used to store, for example, the connection string that xs requires to create a console connection (like "/console?ref=OpaqueRef%3A437d90db-79f0-82ac-4c8c-69941f67543e&session_id=OpaqueRef%3A25034f54-1e82-d07c-0d92-e315cbfd314a")

• Finally, the OS API returns to novaclient the url that a java client can use: {'url': 'http://nova_xvp_proxy:6081/console?token=xyz'}.  This url has a short life span (say 300 seconds) after which it is no longer usable.

NOTE: the OS api call to request a console will be a servers action: server.get_console(console_type), where console_type is one of ["xvp-vnc", "ws-vnc"]

Connection Process B: Citrix java client connects to nova

Story:  Bob then executes the Citrix java client using the url he just retrieved as a parameter. He now is able to log-in to his server.

Details:

• VNC Client sends an HTTP CONNECT to the url returned from the above process (http://nova_xvp_proxy:6081/console?token=xyz).  That url points to a simple nova-xvpvncproxy wsgi server

• nova-xvpvncproxy grabs the token from the connection string, and then calls out to the nova-consoleauth to retrieve the internal connection details 
• Assuming the token is valid, nova-xvpvncproxy returns HTTP/1.1 200 OK\r\n\r\n to the client
• nova-xvpvncproxy then uses the connect_info to create a connection to the actual vnc server, and then proxies that info to the client, who now has a live vnc session
  • Again, for xs, the connect_info would contain an internal_access_path consisting of the Opaque instance Ref and session_id
  • For libvirt/kvm, no additional auth is needed and the thus the tcp proxy can be configured a bit more easily

The Proposal Part II: novnc parity

The underlying mechanics of setting up sessions for novnc are very similar to above.  Some things of note:

• nova-vncproxy will be rewritten, and will be based on the web socket proxy that ships with novnc.  Presently, nova-vncproxy uses eventlet for the ws proxy, which has poor browser support.  novnc's server is much more heavily tested and should give a better user experience in general.
• The new ws proxy server will also use nova-consoleauth to validate tokens.

Other Stuff

Once this is in place, we should also get the Ajax console to use nova-consoleauth, and rework the associated api call so that it is similar in function to the other interactive consoles.

Console log would not be treated like an interactive console, and should have its own api.

Also note that atm there is very little useful debug traceback. The updated implementation should be done with debug-ability and audit-ability in mind.