ReleaseNotes/Kilo

= OpenStack 2015.1.0 (Kilo) Release Notes =

Erasure Code (beta)
Swift now supports an erasure-code (EC) storage policy type. This allows deployers to achieve very high durability with less raw capacity as used in replicated storage. However, EC requires more CPU and network resources, so it is not good for every use case. EC is great for storing large, infrequently accessed data in a single region.

Swift's implementation of erasure codes is meant to be transparent to end users. There is no API difference between replicated storage and EC storage.

To support erasure codes, Swift now depends on PyECLib and liberasurecode. liberasurecode is a pluggable library that allows for the actual EC algorithm to be implemented in a library of your choosing.

Full docs are at http://swift.openstack.org/overview_erasure_code.html

Composite tokens
Composite tokens allow other OpenStack services to store data in Swift on behalf of a client so that neither the client nor the service can update the data without both party's consent.

An example of this is that a user requests that Nova save a snapshot of a VM. Nova passes the request to Glance, Glance writes the image to a Swift container as a set of objects. In this case, the user cannot modify the snapshot without also having a valid token from the service. Nor can the service update the data without a valid token from the user. But the data is still stored in the user's account in Swift, which makes accounting simpler.

Full docs are at http://swift.openstack.org/overview_backing_store.html

Data placement updates for smaller, unbalanceable clusters
Swift's data placement now accounts for device weight. This allows operators to gradually add new zones and regions without immediately causing a large amount of data to be moved. Also, if a cluster is inbalanced (eg a two-zone cluster where one zone has twice the capacity of the other), Swift will more efficiently use the available space and warn when replicas are placed without enough dispersion in the cluster.

Global cluster replication improvements
Replication between regions will now only move one replica per replication run. This gives the remote region a chance to replicate internally and thus avoid more data moving over the WAN

Known Issues

 * As a beta release, EC support is nearly fully feature complete, but it is lacking support for some features (like multi-range reads) and has not had a full performance characterization. This feature relies on ssync for durability. Deployers are urged to do extensive testing and not deploy production data using an erasure code storage policy.

Upgrade Notes
As always, you can upgrade to this version of Swift with no end-user downtime


 * In order to support erasure codes, Swift has a new dependency on PyECLib (and liberasurecode, transitively). Also, the minimum required version of eventlet has been raised.

API v2.1

 * We have the first release of the next generation of the Nova API, v2.1. The v2.1 is designed to be backwards compatible with v2.0 with the addition of strong API validation. All changes to the API are discoverable via the advertised microversion. For more details see: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html


 * For kilo, by default we are still using v2.0 API code to server v2.0 API requests. It is hoped that in liberty that v2.1 will be used to serve requests for both v2.0 and v2.1.


 * For liberty v2.0 is now frozen, and all new features will now be added into the v2.1 API using the microversions mechanism. Microversion increments released with kilo are:
 * Extending the keypair API to support for x509 certificates, to be used with Windows WinRM, is one of the first API features added as a microversion in the v2.1 API.
 * Exposing additional attributes in os-extended-server-attributes


 * python-novaclient does not yet have support for the v2.1 API


 * The policy enforcement of Nova v2.1 API get improvement.
 * Policy only enforce at the entry of API.
 * Without duplicated rules for single one API anymore.
 * All the v2.1 API policy rule use 'os_compute_api' as prefix which distinguish with v2 API.
 * Due to hard-code permission checks at db layer, part of Nova API isn't configurable by policy before. It's always required admin user. Part of Nova v2.1 API's hard-code permission checks is removed which make API policy configurable. The rest of hard-code permission checks will be removed at Liberty.

Upgrade Support

 * We have reduced the data migrations that happen in the DB migration scripts, this now happens in a "lazy" way inside the DB objects code. There are nova-manage commands to help force migration of the data. For more details see: http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/flavor-from-sysmeta-to-blob.html


 * Change https://review.openstack.org/#/c/97946/ adds database migration 267 which scans for null instances.uuid records and will fail if any are found since the migrationultimately needs to make instances.uuid non-nullable and adds a UniqueConstraint on that column. A helper script is provided to search for null instances.uuid records before running the database migrations.  Before running 'nova-manage db sync', run the helper script 'nova-manage db null_instance_uuid_scan' which, by default, will just search and dump results, it does not change anything.  Pass the --delete option to the null_instance_uuid_scan command to automatically remove any null records where instances.uuid is null.

Scheduler

 * A selection of performance optimisations
 * We are in the process of making structural changes to the scheduler that will help improve our ability to evolve and improve scheduling. This should not be visible from an end user perspective.

Cells v2

 * There are some initial parts of cell v2 supported added, but this feature is not yet ready to use.
 * new 'nova-manage api_db sync' and 'nova-manage api_db version' commands for working with the new api database for cells, but nothing is using this database yet so it is not necessary to set it up.

Hyper-V

 * Support for generation 2 VMs: https://blueprints.launchpad.net/nova/+spec/hyper-v-generation-2-vms
 * Support for SMB based volumes, to sit along side existing iSCSI volume support: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/hyper-v-smbfs-volume-support.html
 * Support for x509 certificate based key pairs: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/keypair-x509-certificates.html
 * Host power actions now work with Hyper-V: https://blueprints.launchpad.net/nova/+spec/hyper-v-host-power-actions

Libvirt (KVM)

 * NFV related features:
 * NUMA based scheduling: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/input-output-based-numa-scheduling.html
 * Pinning guest vCPUs: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-cpu-pinning.html
 * Large page support: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/virt-driver-large-pages.html
 * vhostuser VIF driver: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/libvirt_vif_vhostuser.html
 * Support for running KVM on IBM System z: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/libvirt-kvm-systemz.html
 * Support for parallels: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/pcs-support.html
 * Support for SMB based volumes: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/libvirt-smbfs-volume-support.html
 * Quiesce using QEMU guest agent: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/quiesced-image-snapshots-with-qemu-guest-agent.html
 * Quobyte volume support: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/quobyte-nova-driver.html
 * Support for QEMU iSCSI initiator: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/qemu-built-in-iscsi-initiator.html

VMware

 * Support for Ephemeral disks: http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/vmware-ephemeral-disk-support.html
 * Support fo vSAN: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/vmware-vsan-support.html
 * Support for based OVA images: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/vmware-driver-ova-support.html
 * Support for SPBM based storage policies: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/vmware-spbm-support.html
 * The capability to run multiple vmware clusters from a single nova-compute service via the vmware.cluster_name config option is deprecated in Kilo and may be removed in Liberty. See: https://review.openstack.org/#/c/163426/

Ironic

 * Support to pass flavor capabilities to ironic: http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/pass-flavor-capabilities-to-ironic-virt-driver.html

Known Issues

 * Evacuate recovery code has the potential to destroy data. On nova-compute startup, instances reported by the hypervisor are examined to see if they have moved (i.e. been evacuated) from the current host during the outage. If the determination is made that they were, then they are destroyed locally. This has the potential to choose incorrectly and destroy instances unexpectedly. On libvirt-like nodes, this can be triggered by changing the system hostname. On vmware-like nodes, this can be triggered by attempting to manage a single vcenter deployment from two different hosts (with different hostnames). This will be fixed properly in Liberty, but for now deployments that wish to disable this behavior as a preventive measure can set workarounds.destroy_after_evacuate=False. NOTE: This is not a regression and has been a flaw in the design of the evacuate feature since its introduction. There is no easy fix for this, hence this workaround to limit the potential for damage. The proposed fix in liberty is here: https://review.openstack.org/#/c/161444/.


 * When using libvirt 1.2.2 live snapshots fail intermittently under load so live snapshots are disabled by default. Deployers can set workarounds.disable_libvirt_livesnapshot=True in nova.conf to enable live snapshot support. See https://bugs.launchpad.net/nova/+bug/1334398 for details.


 * Performance issues have been reported with rootwrap, see bug https://bugs.launchpad.net/nova/+bug/1415106. Deployers can disable rootwrap and fallback to just using sudo by setting workarounds.disable_rootwrap=True.


 * The generated nova.conf.sample is possibly missing some oslo related configuration options.

Upgrade Notes
Below are changes you should be aware of when upgrading. Where possible, The git commit hash is provided for you to find more information:


 * Neutron ports are no longer deleted after your server is deleted, if you created them outside of Nova: 1153a46738fc3ffff98a1df9d94b5a55fdd58777
 * EC2 API support has been deprecated, and is likely to be removed in kilo, see f098398a836e3671c49bb884b4a1a1988053f4b2
 * Websocket proxies need to be upgraded in a lockstep with the API nodes, as older API nodes will not be sending the access_url when authorizing console access, and newer proxy services (this commit and onward) would fail to authorize such requests 9621ccaf05900009d67cdadeb1aac27368114a61
 * After fully upgrading to kilo (i.e. all nodes are running kilo code), you should start a background migration of flavor information from its old home to its new home. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. This is critical to complete before the Liberty release, where support for the old location will be dropped. Use "nova-manage db migrate_flavor_data " to perform this transition.
 * NOTE: The online flavor migration will put extra load on conductor nodes. You may need to scale those out during Kilo to handle that extra load until you upgrade to Liberty where the online flavor migrations are dropped.
 * Since the v2.1 API hasn't been officially supported before, the policy.conf has changed quite a bit so that most things are configurable via the policy.conf rather than hard-coded admin checks in the database API. When deploying the v2.1 API, one should use the sample policy file provided with Nova.
 * VMware rescue VM behaviour no longer creates a new VM and instead happens in place: cd1765459a24e52e1b933c8e05517fed75ac9d41
 * force_config_drive = always has been deprecated, and force_config_drive = True should be used instead: c12a78b35dc910fa97df888960ef2b9a64557254
 * Running hyper-v, if you deployed code that was past this commit: b4d57ab65836460d0d9cb8889ec2e6c3986c0a9b but before this commit: c8e9f8e71de64273f10498c5ad959634bfe79975 you make have problems to manually resolve see: c8e9f8e71de64273f10498c5ad959634bfe79975
 * Changed the default value of: multi_instance_display_name_template see: 609b2df339785bff9e30a9d67d5c853562ae3344
 * Please use "nova-manage db null_instance_uuid_scan" to ensure the DB migrations will apply cleanly, see: c0ea53ce353684b48303fc59393930c3fa5ade58

Key New Features

 * Using graduated oslo.policy. Accounts changes to config options and updates the in-tree etc/config files. http://specs.openstack.org/openstack/glance-specs/specs/kilo/pass-targets-to-policy-enforcer.html
 * Ability to deactivate an image. Adds 2 new API calls and may require policy changes. http://specs.openstack.org/openstack/glance-specs/specs/kilo/deactivate-image.html
 * Basic support for Image conversion during the import process of an Image. http://specs.openstack.org/openstack/glance-specs/specs/kilo/conversion-of-images.html
 * Glance sorting enhancements. Images v2 API supports new sorting syntax including ability to specify the sort dir for each key. http://specs.openstack.org/openstack/glance-specs/specs/kilo/sorting-enhancements.html
 * Notifications support for metadefs. http://specs.openstack.org/openstack/glance-specs/specs/kilo/metadefs-notifications.html
 * Multiple datastore support for VMware Storage driver. http://specs.openstack.org/openstack/glance-specs/specs/kilo/vmware-store-multiple-datastores.html
 * Glance Image Introspection during the import process of an Image. http://specs.openstack.org/openstack/glance-specs/specs/kilo/introspection-of-images.html
 * Support in Metadefs for multivalue operators. http://specs.openstack.org/openstack/glance-specs/specs/kilo/metadata-multivalue-operators-support.html
 * Adding new taskflow executor and removing the old eventlet executor. http://specs.openstack.org/openstack/glance-specs/specs/kilo/taskflow-integration.html
 * Digest algorithm is now configurable. SHA-1 not being suitable for general-purpose digital signature applications that require 112 bits of security as per FIPS, we provide with a configuration to chose between these standards.
 * Metadef Tag support. http://specs.openstack.org/openstack/glance-specs/specs/kilo/metadefs-tags.html
 * Allow None values to be returned from the API. Glance's API v2 now also returns fields that have value None.
 * Catalog Index Service experimental API.http://specs.openstack.org/openstack/glance-specs/specs/kilo/catalog-index-service.html
 * More granular capabilities optional support to storage drivers. http://specs.openstack.org/openstack/glance-specs/specs/kilo/store-capabilities.html
 * Semver Utility for DB storage. http://specs.openstack.org/openstack/glance-specs/specs/kilo/semver-support.html
 * Reload configuration files on SIGHUP signal. Zero downtime config reload. http://specs.openstack.org/openstack/glance-specs/specs/kilo/sighup-conf-reload.html
 * Software Metadata Definitions. http://specs.openstack.org/openstack/glance-specs/specs/kilo/software-metadefs.html
 * Glance Swift Store to use Multiple Containers for Storing Images. http://specs.openstack.org/openstack/glance-specs/specs/kilo/swift-store-multiple-containers.html

Known Issues

 * Adding image member throws 500 when the member name is longer than 255 characters. https://bugs.launchpad.net/glance/+bug/1424038
 * Glance v2 API is incompatible with v1 API for owner change. https://bugs.launchpad.net/glance/+bug/1420008
 * Glance scrubber doesn't work when registry operates in trusted-auth mode. https://bugs.launchpad.net/glance/+bug/1439666

Upgrade Notes

 * Removed deprecated option db_enforce_mysql_charset. Corresponding commit: efeb69f9033a57a1c806f71ee3ed9fd3f4d2475e
 * Notifications for metadef resources are now supported. Corresponding commit: fd547e3717dc4a3a92c1cb2104c18608a4f4872a
 * VMware multiple datastores can be enabled by a few config changes. Corresponding commit: 96fb31d7459bd4e05e052053177dce4d38cdaf90
 * Removed the eventlet executor and added a new taskflow executor for async tasks. Corresponding commits: ae3135e1d67df77697a24fddaee3efeadb34a0dd and a39debfd55f6872e5f4f955b75728c936d1cee4b
 * Replace snet config with endpoint config. Corresponding commit: 41a9a065531ec946b4a9baf999f97d10fa493826
 * Digest algorithm is now configurable. Corresponding commit: 82194e0c422966422f7a4e2157125c7ad8fbc5b5
 * Cleanup chunks for deleted image that was in 'saving' state while deleting. Corresponding commit: 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
 * Glance now uses graduated oslo.policy. Corresponding commit: cb7d5a4795bbdaf4dc3eaaf0a6fb1add52c09011
 * An image can now be deactivated. A new state called deactivated has been added to the Image data asset. Corresponding commit: b000c85b7fabbe944b4df3ab57ff73883328f40d

Key New Features

 * Support for Federated authentication via Web Single-Sign-On -- When configured in keystone, the user will be able to choose the authentication mechanism to use from those support by the deployment. This feature must be enabled by changes to local_settings.py to be utilized. The related settings to enable and configure can be found here.


 * Support for Theming -- A simpler mechanism to specify a custom theme for Horizon has been included. Allowing for use of CSS values for Bootstrap and Horizon variables, as well as the inclusion of custom CSS. More details available here.


 * Sahara UX Improvements -- Dramatic improvements to the Sahara user experience have been made with the addition of guided cluster creation and guided job creation pages.


 * Launch Instance Wizard (beta) -- A full replacement for the launch instance workflow has been implemented in AngularJS to address usability issues in the existing launch instance workflow. Due to the late inclusion date and limited testing, this feature is marked as beta for Kilo and not enabled by default. To use the new workflow, the following change to local_settings.py is required: . Additionally, you can disable the default launch instance wizard with the following:   . This new work is a view into future development in Horizon.


 * Nova
 * allow service disable/enable on Hypervisor
 * Migrate all instances from host
 * expose serial console


 * Cinder
 * Cinder v2 by default
 * Managed/Unmanaged volume support -- allows admin to manage existing volumes not managed by cinder, as well as unmanage volumes.
 * Volume transfer support between projects
 * Volume encryption metadata support


 * Glance
 * View added to allow administrators to view/add/update Glance Metadata definitions


 * Heat
 * Stack Template view
 * Orchestration Resources Panel
 * Suspend/Resume actions for Stacks
 * Preview Stack view allows users to preview stacks specified in templates before creating them.


 * Trove
 * Resizing of Trove instances -- changing instance flavor


 * Ceilometer
 * Display IPMI meters values from Ceilometer


 * New Reusable AngularJS widgets in Horizon:
 * AngularJS table implementation
 * Table drawers -- expandable table content
 * improved client/server search
 * Transfer table widget


 * Configurable web root for Horizon beyond just '/'

Known Issues

 * Volumes created from snapshots are empty - https://bugs.launchpad.net/horizon/+bug/1447288
 * Django 1.8 is not fully supported yet.

Upgrade Notes

 * Django 1.7 is now supported.

Hierarchical multitenancy
Projects can be nested under other projects by setting the  attribute to an existing project when creating a new project. You can also discovery the parent-child hierarchy through the existing  API.

Role assignments can now be assigned to both users and groups on subtrees in the project hierarchy.

This feature will require corresponding support across other OpenStack services (such as hierarchical quotas) in order to become broadly useful.

Fernet tokens
Unlike UUID tokens which must be persisted to a database, Fernet tokens are entirely non-persistent. Deployers can enable the Fernet token provider using  in.

Fernet tokens require symmetric encryption keys which can be established using  and periodically rotated using. These keys must be shared by all Keystone nodes in a multi-node (or multi-region) deployment, such that tokens generated by one node can be immediately validated against another.

Identity federation

 * Keystone can now act as a federated identity provider (IdP) for another instance of Keystone by issuing SAML assertions for local users, which may be ECP-wrapped.
 * Added support for OpenID Connect as a federated identity authentication mechanism.
 * Added the ability to associate many "Remote IDs" to a single identity provider in Keystone. This will help in a case where many identity providers use a common mapping.
 * Added the ability for a user to authenticate via a web browser with an existing IdP, through a Single Sign-On page.
 * Federated tokens now use the  authentication method, although both   and   remain available.
 * Federated users may now be mapped to existing local identities.
 * Groups specified in the mapping rulesets can be identified by name and domain.
 * Groups appearing in federated identity assertions may now be automatically mapped as locally existing groups with local user membership mappings (filtered by white and blacklists).

LDAP

 * Filter parameters specified by API users are now processed by LDAP itself, instead of by Keystone.
 * Experimental support was added to store domain-specific identity backend configuration in SQL using the HTTP API. The primary use case for this is to create a new domain with the HTTP API, and then immediately configure a domain-specific LDAP driver for it without restarting Keystone.

Authorization

 * The "assignment" backend has been split into a "resource" backend (containing domains, projects, and roles) and an "assignment" backend, containing the authorization mapping model.
 * Added support for trust redelegation. If allowed when the trust is initially created, a trustee can redelegate the roles from the trust via another trust.
 * Added support for explicitly requesting an unscoped token from Keystone, even if the user has a  attribute set.
 * Deployers may now opt into disallowing the re-scoping of scoped tokens by setting  in.

Upgrade Notes

 * XML support in Keystone has been removed as of Kilo. When upgrading from Juno to Kilo, it is recommended that references to XML and XmlBodyMiddleware be removed from the Keystone Paste configuration. This includes removing the XML middleware filters and the references from the public_api, admin_api, api_v3, public_version_api, admin_version_api and any other pipelines that may contain the XML filters.
 * All previous extensions (OS-FEDERATION, OS-OAUTH1, OS-ENDPOINT-POLICY and OS-EP-FILTER) are now enabled by default, and are correspondingly marked as either "experimental" or "stable".
 * SQL Schema Downgrades are no longer supported. This change is the result of evaluation that downward SQL migrations are not well tested and become increasingly difficult to support with the volume of data-change that occurs in many of the migrations.
 * The following python libraries are now required: cryptography, msgpack-python, pysaml2 and oauthlib.
 * is now deprecated in favor of  and will be removed in Liberty.
 * Eventlet-specific configuration options such as,  ,  ,  ,  ,  ,  ,  ,   have been moved from the   configuration section to a new configuration section called  . Similarly, Eventlet-specific SSL configuration options such as  ,  ,  ,  ,   have been moved from the   configuration section to a new configuration section called.
 * has been removed in favor of.
 * has been removed in favor of.
 * has been removed in favor of.
 * has been removed in favor of.
 * has been removed in favor of.
 * has been removed in favor of external tooling.
 * has been removed in favor of.
 * has been removed in favor of external access logging.
 * has been removed in favor of.
 * has been removed from  as part of a related security hardening effort.
 * has been removed from  in favor of.

Key New Features

 * Subnet allocation feature allows creating subnets from a pre-defined pool of addresses instead of providing explicit addresses.
 * Support in python-neutron-client (subnetpool-create/update/list/show/delete and subnet-create --subnetpool).
 * API documentation work is underway.
 * Horizon support is tracked with neutron-subnet-allocation blueprint.
 * DVR now supports VLANs in addition to VXLAN/GRE
 * ML2 Hierarchical Port Binding
 * New LBaaS Version 2 API (experimental)
 * Portsecurity support for the OVS ML2 Driver
 * Multiple Prefixes for IPv6 (tenant guest interfaces can now get multiple IPv6 addresses)
 * IPv6 Router (allows guests to get global IPv6 addresses and talk to the world, without NAT)
 * API extension for MTU Selection and Advertisement
 * API extension to check for support of VLAN trunking networks for NFV


 * New Plugins supported in Kilo include the following:
 * A10 Networks LBaaS V2 Driver
 * Brocade LBaaS V2 Driver
 * Brocade ML2 driver for MLX and ICX switches
 * Brocade L3 routing plugin for MLX switch
 * Brocade Vyatta vRouter L3 Plugin
 * Brocade Vyatta vRouter Firewall Driver
 * Brocade Vyatta vRouter VPN Driver
 * Cisco CSR VPNaaS Driver
 * Dragonflow SDN based Distributed Virtual Router L3 Plugin
 * Freescale FWaaS Driver
 * Intel Mcafee NGFW FWaaS Driver
 * IPSEC Strongswan VPNaaS Driver

Known Issues

 * The Firewall-as-a-Service project is still marked as experimental for the Kilo release.
 * Bug 1438819
 * When a new subnet is created on an external network, all existing routers with gateways on the network will get a new address allocated from it. For IPv4 networks, this could consume the entire subnet for router gateway ports.

Upgrade Notes
From Havana, Neutron no longer supported an explicit lease database (https://bugs.launchpad.net/bugs/1202392). This left dead code including unused environment variable. In order to remove the dead code (https://review.openstack.org/#/c/152398/), a change to the dhcp.filter is required, so that line:

dnsmasq: EnvFilter, dnsmasq, root, NEUTRON_NETWORK_ID=

Be replaced by:

dnsmasq: CommandFilter, dnsmasq, root

After advanced services were split into separate packages and received their own service configuration files (specifically, etc/neutron/neutron_lbaas.conf, etc/neutron/neutron_fwaas.conf and etc/neutron/neutron_vpnaas.conf), active service provider configuration can be different after upgrade (specifically, default load balancer (haproxy) and vpn (openswan) providers can be enabled for you even though you previously disabled them in neutron.conf). Please make sure you review configuration after upgrade so that it reflects the desired state of service providers.

Note: this will have no effect if the related service plugin is not loaded in neutron.conf.


 * The neutron.allow_duplicate_networks config option is deprecated in Kilo and will be removed in Liberty where the default behavior will be to just allow multiple ports attached to an instance on the same network in Neutron. (https://review.openstack.org/163581)
 * The linuxbridge agent now enables VXLAN by default (https://review.openstack.org/160826)
 * neutron-ns-metadata-proxy can now be run as non-root (https://review.openstack.org/147437)

Other Notes (Deprecation/EOL etc)

 * Deprecation
 * Brocade Monolithic plugin for Brocade's VDX/VCS series of hardware switches will be deprecated in the L-Release. The functionality provided by this plugin is now addressed by the ML2 Driver available for the VDX series of hardware. The plugin is slated for removal after this release cycle.
 * The monolithic Cisco Meta plugin for Nexus1000V will be deprecated in the L-Release. The functionality provided by this plugin is now available with the Cisco Nexus1000V ML2 mechanism driver. The monolithic plugin is slated for removal after this release cycle.

Key New Features

 * From this point forward any new database schema upgrades will not require restarting Cinder services right away. The services are now independent of schema upgrades. This is part one to Cinder supporting rolling upgrades!
 * Ability to add/remove volumes from an existing consistency group. Read docs for more info.
 * Ability to create a consistency group from an existing consistency group snapshot. Read docs for more info.
 * Create more fine tuned filters/weighers to set how the scheduler will choose a volume backend. Read the docs for more info.
 * Encrypted volumes can now be backed up using the Cinder backup service. Read the docs for more info.
 * Ability to create private volume types. This is perfect when you want to make volume types available to only a specific tenant or to test it before making available to your cloud. To do so use the cinder type-create --is-public.
 * Do incremental backups of volumes. Read docs for more info.
 * Oversubscription with thin provision is configurable. Read docs for more info.
 * Ability to add descriptions to volume types. To do so use ''cinder type-create
 * Cinder now can return multiple iSCSI paths information so that the connector can attach volumes even when the primary path is down (when connector's multipath feature is enabled or not enabled).
 * Add ability to specify a local lvm.conf file. When using the LVM driver this option allows you to set a Cinder specific lvm.conf file in /etc/cinder/lvm.conf.  This enables specific LVM settings and filters that are only picked up and used by Cinder.  See the following changes for details:
 * Cinder: https://review.openstack.org/#/c/148747/
 * Devstack: https://review.openstack.org/#/c/149046/ and https://review.openstack.org/#/c/165281/
 * The Cinder backup service can now backup to an NFS exported filesystem. Read the docs for more info.

Upgrade Notes

 * If you're using HA proxy, the new configuration option 'public_endpoint' should be set to the HA proxy's URL. Cinder client version 1.2.0 introduces version discovery which needs this.
 * The 'host' config option for multiple-storage backends in cinder.conf is renamed to 'backend_host' in order to avoid a naming conflict with the 'host' to locate redis. If you use this option, please ensure your configuration files are updated.
 * The following drivers were removed due to not meeting Cinder's Continuous Integration requirement to verify they're functional in this release. They can be re-added if the requirement is met in the Liberty release:
 * Zadara
 * Fujitsu
 * Nexenta

Key New Features

 * Support to add jitter to polling cycles to ensure pollsters are not querying service's api at the same time
 * Ceilometer API RBAC support
 * Improved Event support:
 * Multi-pipeline support to enable unique processing and publishing of events
 * Enabled ability to capture raw notification messages for auditing and postmortem analysis
 * Support for persisting events into ElasticSearch
 * Publishing support to database, http, file, kafka and oslo.messaging supported message queues
 * Option to split off the events persistence into a separate database
 * Telemetry now supports to collect and store all the event type meters as events. A new option, disable_non_metric_meters, was added to the configuration in order to provide the possibility to turn off storing these events as samples. For further information please see the Telemetry Configuration Reference
 * The Administrator Guide in OpenStack Manuals was updated with a new Events section, where you can find further information about this functionality.
 * Improved pipeline publishing support:
 * Support to publish events and samples to Kafka or HTTP targets
 * Publish data to multiple queues
 * Additional meters
 * memory and disk meters for Hyper-V
 * disk meters for LibVirt
 * power and thermal related IPMI meters, more meters from NodeManager
 * ability to meter Ceph
 * IPv6 support enabled in Ceilometer udp publisher and collector
 * Gnocchi dispatch support for ceilometer-collector
 * Self-disabled pollster mechanism

Upgrade Notes
statistics: ceilometer statistics -m instance -g resource_metadata.instance_type
 * Deprecated meters:
 * The instance: meter is deprecated in the Kilo release. In order to retrieve samples or statistics based on flavor you can use the following queries:

samples: ceilometer sample-list -m instance -q metadata.instance_type=
 * Middleware used to meter Swift was previously packaged in Ceilometer and is now deprecated. It is now separated into it's own library: ceilometermiddleware.
 * Juno configuration: http://docs.openstack.org/juno/install-guide/install/apt/content/ceilometer-swift.html
 * Kilo configuration: http://docs.openstack.org/kilo/install-guide/install/apt/content/ceilometer-swift.html

Key New Features

 * Improved scaling using nested stacks
 * Heat will RPC actions on any resource that is based on a template. This should help to spread the load when dealing with large complex stacks.
 * oslo versioned objects
 * The database layer now uses oslo versioned objects to aid in future upgrades. This will allow a newly upgraded heat-engine to use a database with an older schema. Note that this will not help with upgrading to kilo.
 * New template functions
 * There is a new HOT template version "20150430" which includes two new functions "digest" and "repeat".
 * Multiregion stacks
 * http://docs.openstack.org/hot-reference/content/OS__Heat__Stack.html
 * Access to Heat services
 * The admin now has similar access to services as other projects. This is in the form of "heat-manage service-list" and via horizon. This feature reports the active heat-engines.
 * Improved validation for nova and neutron properties.
 * Pause stack creation/update on a given resource (stack hooks)
 * http://specs.openstack.org/openstack/heat-specs/specs/juno/stack-breakpoint.html
 * http://docs.openstack.org/developer/heat/template_guide/environment.html?highlight=hooks#pause-stack-creation-update-on-a-given-resource
 * New contributed resources
 * Mistral resources
 * gnocchi alarms https://blueprints.launchpad.net/heat/+spec/ceilometer-gnocchi-alarm
 * keystone resources supported with Keystone v3 server for Project, Role, User and Group
 * Stack lifecycle scheduler hints
 * Software-config improvements
 * Option to use Swift TempURLs for deployment signals http://specs.openstack.org/openstack/heat-specs/specs/kilo/software-config-swift-signal.html
 * The ability to create and monitor a deployment from the heat command line, outside of the stack http://specs.openstack.org/openstack/heat-specs/specs/kilo/software-config-trigger.html

Upgrade Notes

 * The default of the configuration option "num_engine_workers" has changed from 1 to a number based on the the number of CPUs. This is now the same as the way other projects set the number of workers.
 * The default for the configuration option "max_nested_stack_depth" has been increased to 5.
 * There is a new configuration option "convergence" it is by default off. This feature is not yet complete and this option should remain off.
 * In preparation of an upcoming major feature (convergence) there have been some significant DB schema changes. It is suggested that the heat-engine is shutdown during schema upgrades.

Deprecation

 * The follow resources are deprecated OS::Heat::HARestarter and OS::Heat::CWLiteAlarm
 * The CloudWatch API (heat-api-cw)

Key New Features

 * Support for a new replication strategy based on async GTID replication (new in MySQL 5.6)
 * We now support for creating n-replicas from a single master in one API call
 * We also support for failover from an unresponsive master to the most up-to-date slave can now be achieved using the new 'eject-master' API
 * Support for Trove guest managers to support the following new datastores:
 * Vertica, and Vertica Cluster
 * DB2
 * CouchDB
 * Extended current management API layer :
 * We now have a new management API to support listing and viewing deleted trove instances
 * We also added a new management API to ping a datastore guestagent via the RPC mechanism
 * Horizon updates to support resize of Trove instances.
 * Users now have the ability to edit/update the names of Trove instances
 * Integration with the cross-project OpenStack profiling library (OSProfiler)

Upgrade Notes

 * We migrated from deprecated oslo-incubator messaging code to the official oslo.messaging python module. Please look at git.openstack.org/cgit/openstack/trove/tree/etc/trove/trove.conf.sample#n18 for more details on the changed config values that were added to support this, (Change)
 * Datastores and strategies that are not currently being tested by any CI have been moved into an 'experimental' section in their respective modules. Once these datastores and strategies have appropriate tests exercising and gating against them in CI, they will be graduated to 'stable'.
 * Added new documentation to help with the process of building trove guest images for different datastores at http://docs.openstack.org/developer/trove/dev/building_guest_images.html

Key New Features

 * New plugins, their features and versions:
 * MAPR
 * Apache Storm
 * Apache Hadoop 2.6.0 was added, Apache Hadoop 2.4.1 deprecated
 * New services for CDH plugin added up to HDFS, YARN, Spark, Oozie, HBase, ZooKeeper and other services
 * Added indirect VM access for better utilization of floating IPs
 * Added event log support to have detailed info about provisioning progress
 * Optional default node group and cluster templates per plugin
 * Horizon updates:
 * Guided cluster creation and job execution
 * Filtering on search for objects
 * Editing of Node Group templates and Cluster templates implemented
 * Added Shell Job Type for clusters running Oozie
 * New Job Types endpoint to query list of the supported Job Types

Upgrade Notes
Details: http://docs.openstack.org/developer/sahara/userdoc/upgrade.guide.html#juno-kilo


 * Sahara now requires policy.json configuration file.

State Machine
Ironic now uses a formal model for the logical state of each node it manages (New Ironic State Machine). This has enabled the addition of two new processes: cleaning and inspection.
 * Automatic disk erasure between tenants is now enabled by default. This may be extended to perform additional cleaning steps, such as re-applying firmware, resetting BIOS settings, etc (Node Cleaning).
 * Both in-band and out-of-band methods are available to inspect hardware. These methods may be used to update Node properties automatically (Hardware Inspection).

Version Headers
The Ironic REST API expects a new X-OpenStack-Ironic-API-Version header be passed with each HTTP[S] request. This header allows client and server to negotiate a mutually supported interface (REST API "micro" versions). In the absence of this header, the REST service will default to a compatibility mode and yield responses compatible with Juno clients. This mode, however, prevents access to most features introduced in Kilo.

Hardware Driver Changes
The following new drivers were added:
 * AMT
 * iRMC
 * VirtualBox (testing driver only)

The following enhancements were made to existing drivers:
 * Configdrives may be used with the "agent" drivers in lieu of a metadata service, if desired.
 * SeaMicro driver supports serial console
 * iLO driver supports UEFI secure boot
 * iLO driver supports out-of-band node inspection
 * iLO driver supports resetting ilo and bios during cleaning

Support for third-party and out-of-tree drivers is enhanced by the following two changes:
 * Drivers may store their own "internal" information about Nodes.
 * Drivers may register their own periodic tasks to be run by the Conductor.
 * vendor_passthru methods now support additional HTTP methods (eg, PUT and POST).
 * vendor_passthru methods are now discoverable in the REST API. See node vendor passthru and driver vendor passthru

Other Changes

 * Logical names may be used to address Nodes, in addition to their canonical UUID.
 * For servers with varied local disks, hints may be supplied that affect which disk device the OS is provisioned to.
 * Support for fetching kernel, ramdisk, and instance images from HTTP[S] sources directly has been added to remove the dependency on Glance. Using ironic as a standalone service
 * Nodes may be placed into maintenance mode via REST API calls. An optional maintenance reason may be specified when doing so.

Known Issues

 * Running more than one nova-compute process is not officially supported.
 * While Ironic does include a ClusteredComputeManager, which allows running more than one nova-compute process with Ironic, it should be considered experimental and has many known problems.
 * Drivers using the "agent" deploy mechanism do not support "rebuild --preserve-ephemeral"

Upgrade Notes

 * IPMI Passwords are now obfuscated in REST API responses. This may be disabled by changing API policy settings.
 * The "agent" class of drivers now support both whole-disk and partition based images.
 * The driver_info parameters of "pxe_deploy_kernel" and "pxe_deploy_ramdisk" are deprecated in favour of "deploy_kernel" and "deploy_ramdisk".
 * Drivers implementing their own version of the vendor_passthru method has been deprecated in favour of the new @passthru decorator.

Juno to Kilo
The recommended upgrade process is documented here:
 * http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html#upgrading-from-juno-to-kilo

Upgrading from Icehouse "nova-baremetal"
An upgrade from an Icehouse Nova installation using the "baremetal" driver directly to Kilo Ironic is untested and unsupported. Instead, please follow the following upgrade path:
 * 1) Icehouse Nova "baremetal" -> Juno Nova "baremetal"
 * 2) Juno Nova "baremetal" -> Juno Ironic
 * 3) Juno Ironic -> Kilo Ironic

Documentation for steps 1 and 2 is available at: https://wiki.openstack.org/wiki/Ironic/NovaBaremetalIronicMigration

OpenStack Documentation

 * New docs.openstack.org landing page and new web design for End User Guide and Admin User Guide
 * First release of the Networking Guide
 * Migration to RST for End User Guide and Admin User Guide
 * New specialty teams:
 * Install Guides
 * High Availability Guide
 * Networking Guide
 * User Guides (Admin and End User)
 * First App Tutorial sprint
 * Driver documentation clarification and connections