Latest revision as of 16:34, 18 September 2014

Possibly outdated content
most OpenStack libraries already switched to requests so the content on this page may be outdated

Securing OpenStack Client Connections

The OpenStack client repositories (or packages) include both the Python API bindings and the reference command line interface (CLI) implementation to communicate with the OpenStack APIs. Client support for modern encrypted connections, i.e SSLv3 and/or TLSv1, has been spotty at best. Most of the clients are capable of using SSL for encryption but often the certificate verification part of the protocol did not work properly for tier-2 or privately signed certificates, prompting the addition of the --insecure option to some of the clients. In addition, most of the clients had no mechanism to specify an alternate CA bundle file to enable certificate verification. Python has historically had incomplete X.509 certificate support in its standard library. For example, ssl and httplib do not verify the hostname as part of certificate verification. Four of the clients (keystone, nova, cinder, quantum) use httplib2 which had no hostname verification before 0.7.0 and what it currently has is incomplete. The other two clients (glance and swift) use httplib directly and either have no hostname verification (swift) or implement it locally (glance).

Issue Summary

Python's ssl module does no certificate hostname verification in 2.7.x; it has been added in 3.2 but will not be backported. [1]
ssl is pinned to using protocol version SSLv23 and must be directly patched to override it.
httplib uses ssl and adds no hostname verification. In addition it only uses the default SSLV23 protocol version set by ssl. We don't want this as SSLv2 is deprecated and insecure. Patching that value directly into the ssl module works but is suboptimal.
httplib2 implements a basic hostname verification but it has some problems such as only checking commonName if the certificate's subjectAltNames is not present and handling wildcards differently than specified in RFC-2818.
httplib2 uses an internal ca bundle (cacert.txt) if the ca_certs argument is not given to HTTPSConnectionWithTimeout.<u>init</u>().
httplib2 only supports 3xx redirects for the GET method.

Additional Notes

glanceclient has patched the ssl module out of httplib in favor of pyOpenSSL. Stuart McLaren added http.VerifiedHTTPSConnection.host_matches_cert() to validate commonName and subjectAltName for httplib connections but it doesn't handle wildcards.

The CLI Solution

The current round of patches to the CLIs is to get them all up to the same level of support for TLSv1 for authentication, at a minimum. Glance and Swift continue to use httplib directly for their data transfer connections (really, all connections to their respective services) and these already support SSL.v?.

Why requests?

The requests module [2] backported match_hostname() from Python 3.2. Like all of the other modules here it does not handle the iPAddress attribute in subjectAltName. This is mostly relevant in development and testing use cases like with DevStack. The 3.2 match_hostname() implementation however does allow IP addresses as a dNSName.Requests also brings a number of other features to the table that may or may not have been implemented individually in the existing clients such as JSON encoding/decoding and 3xx redirection support for POST, PUT, PATCH DELETE, and HEAD. Plus it is stable (notwithstanding the recent 1.0 release) and the developer is known in the OS community.

Original Client HTTP Modules

Client	Module	Client object	CLI Arg
keystone	httplib2	class.HTTPClient(httplib2.Http)	--os-cacert
nova	httplib2	class.HTTPClient(httplib2.Http)	n/a
cinder	httplib2	class.HTTPClient(httplib2.Http)	n/a
glance	httplib	class.HTTPClient(object)	--ca-file
swift	httplib	class.HTTPClient(object)	n/a
quantum	httplib2	class.HTTPClient(httplib2.Http)	n/a

The approach taken for the httplib2 subclasses is to change the parent class to object and rework the request() method to call requests.request(). Some of the differences for requests leaked out of that method but have been mostly containd within the HTTPClient class. All four of the clients (formerly) using httplib2 have implemented one or more features that can easily be handled by requests (such as redirection) or should also be propogated to the other clients. This is ripe for a refactor of HTTPClient to a common module but that effort is not in scope here.

Client Update Status

keystoneclient (complete) https://review.openstack.org/17624

replace httplib2 with requests

novaclient (complete) https://review.openstack.org/18257

replace httplib2 with requests
add --os-cacert and OS_CACERT support
provide ca_cert to keystone client for authentication

cinderclient (complete) https://review.openstack.org/18278

replace httplib2 with requests
add --os-cacert and OS_CACERT support
provide ca_cert to keystone client for authentication

glanceclient (complete) https://review.openstack.org/17698

rename --ca-cert to --os-cacert and add OS_CACERT
provide ca_cert to keystone client for authentication

swiftclient (complete) https://review.openstack.org/18393

add --os-cacert and OS_CACERT support
provide ca_cert to keystone client for authentication

quantumclient (not started)

replace httplib2 with requests
add --os-cacert and OS_CACERT support
provide ca_cert to keystone client for authentication

Testing

Aside from the usual unit tests, support for a TLS proxy is being added to DevStack to demonstrate and test a TLS-enabled OpenStack configuration. It uses stud as the TLS endpoint that proxies to the usual service endpoints. The most interesting challenge here is doing it all on a single host and making the service catalog work. Yay! The TLS-in-DevStack also builds a two-tiered CA (root and intermediate) for testing proper certificate chain validation.

Links

glanceclient host_matches_cert():
- https://review.openstack.org/#/c/16305/4/glanceclient/common/http.py
request's urllib3.ssl_match_hostname.match_hostname():
- https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L72
- https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/packages/ssl_match_hostname/init.py#L23

[1]:http://bugs.python.org/issue1589, the comments about 2.x begin at http://bugs.python.org/issue1589#msg120946

[2]:The patches for the OpenStack CLIs were engineered and implemented before the release of requests 1.0 which is a significantly different implementation and is untested in our application.

-- dtroyer@gmail.com

@@ Line 1: / Line 1: @@
-__NOTOC__
+{{note|Possibly outdated content|most OpenStack libraries already switched to requests so the content on this page may be outdated}}
-<<[[TableOfContents]]()>>
+= Securing OpenStack Client Connections =
+__TOC__
-= Securing [[OpenStack]] Client Connections =
+The [[OpenStack]] client repositories (or packages) include both the Python API bindings and the reference command line interface (CLI) implementation to communicate with the [[OpenStack]] APIs.   Client support for modern  encrypted connections, i.e SSLv3 and/or TLSv1, has been spotty at best.  Most of the clients are capable of using SSL for encryption but often the certificate verification part of the protocol did not work properly for  tier-2 or privately signed certificates, prompting the addition of the <code><nowiki>--insecure</nowiki></code> option to some of the clients.  In addition, most of the clients had no mechanism to specify an alternate CA bundle file to enable certificate verification.  Python has historically had incomplete X.509 certificate support in its standard library.  For example, ssl and httplib do not verify the hostname as part of certificate verification.  Four of the clients (keystone, nova, cinder, quantum) use <code><nowiki>httplib2</nowiki></code> which had no hostname verification before 0.7.0 and what it currently has is incomplete. The other two clients (glance and swift) use <code><nowiki>httplib</nowiki></code> directly and either have no hostname verification (swift) or implement it locally (glance).
-The  [[OpenStack]] client repositories (or packages) include both the Python API  bindings and the reference command line interface (CLI) implementation  to communicate with the [[OpenStack]] APIs.   Client support for modern  encrypted connections, i.e SSLv3 and/or TLSv1, has been spotty at best.Most  of the clients are capable of using SSL for encryption but often the  certificate verification part of the protocol did not work properly for  tier-2 or privately signed certificates, prompting the addition of the  --insecure option to some of the clients.  In addition, most of the  clients had no mechanism to specify an alternate CA bundle file to  enable certificate verification.Python  has historically had incomplete X.509 certificate support in its  standard library.  For example, ssl and httplib do not verify the  hostname as part of certificate verification. Four of the clients  (keystone, nova, cinder, quantum) use httplib2 which had no hostname  verification before 0.7.0 and what it currently has is incomplete. The  other two clients (glance and swift) use httplib directly and either  have no hostname verification (swift) or implement it locally (glance).
 == Issue Summary ==
-* Python's ssl module does no certificate hostname verification in 2.7.x;  it has been added in 3.2 but will not be backported.[1]* ssl is pinned to using protocol version SSLv23 and must be directly patched to override it.*  httplib uses ssl and adds no hostname verification.  In addition it  only uses the default SSLV23 protocol version set by ssl.  We don't want  this as SSLv2 is deprecated and insecure.  Patching that value directly  into the ssl module works but is suboptimal.*  httplib2 implements a basic hostname verification but it has some  problems such as only checking commonName if the certificate's  subjectAltNames is not present and handling wildcards differently than  specified in RFC-2818.*  httplib2 uses an internal ca bundle (cacert.txt) if the ca_certs  argument is not given to HTTPSConnectionWithTimeout.<u>init</u>().* In other news, httplib2 only supports 3xx redirects for GET method.
+* Python's <code><nowiki>ssl</nowiki></code> module does no certificate hostname verification in 2.7.x;  it has been added in 3.2 but will not be backported.[[#footnote1| [1] ]]
+* <code><nowiki>ssl</nowiki></code> is pinned to using protocol version SSLv23 and must be directly patched to override it.
+* <code><nowiki>httplib</nowiki></code> uses <code><nowiki>ssl</nowiki></code> and adds no hostname verification.  In addition it only uses the default <code><nowiki>SSLV23</nowiki></code> protocol version set by <code><nowiki>ssl</nowiki></code>.  We don't want this as <code><nowiki>SSLv2</nowiki></code> is deprecated and insecure.  Patching that value directly into the <code><nowiki>ssl</nowiki></code> module works but is suboptimal.
+* <code><nowiki>httplib2</nowiki></code> implements a basic hostname verification but it has some problems such as only checking commonName if the certificate's subjectAltNames is not present and handling wildcards differently than specified in RFC-2818.
+* <code><nowiki>httplib2</nowiki></code> uses an internal ca bundle (cacert.txt) if the ca_certs argument is not given to <code><nowiki>HTTPSConnectionWithTimeout.<u>init</u>()</nowiki></code>.
+* <code><nowiki>httplib2</nowiki></code> only supports 3xx redirects for the GET method.
-== Additional Notes ==
+=== Additional Notes ===
-glanceclient  has patched the ssl module out of httplib in favor of pyOpenSSL.   Stuart [[McLaren]] added http.VerifiedHTTPSConnection.host_matches_cert() to  validate commonName and subjectAltName for httplib connections but it  doesn't handle wildcards.
+glanceclient has patched the <code><nowiki>ssl</nowiki></code> module out of httplib in favor of pyOpenSSL.   Stuart [[McLaren]] added <code><nowiki>http.VerifiedHTTPSConnection.host_matches_cert()</nowiki></code> to  validate commonName and subjectAltName for <code><nowiki>httplib</nowiki></code> connections but it doesn't handle wildcards.
-=== Why requests? [2] ===
+== The CLI Solution ==
-The  requests module backported match_hostname() from Python 3.2.  Like all  of the other modules here it does not handle the iPAddress attribute in  subjectAltName. This is mostly relevant in development and testing use  cases like with [[DevStack]].  The 3.2 match_hostname() implementation  however does allow IP addresses as a dNSName.Requests  also brings a number of other features to the table that may or may not  have been implemented individually in the existing clients such as JSON  encoding/decoding and 3xx redirection support for POST, PUT, PATCH  DELETE, and HEAD.  Plus it is stable (notwithstanding the recent 1.0  release) and the developer is known in the OS community.
+The current round of patches to the CLIs is to get them all up to the same level of support for TLSv1 for authentication, at a minimum.  Glance and Swift continue to use <code><nowiki>httplib</nowiki></code> directly for their data transfer  connections (really, all connections to their respective services) and these already support SSL.v?.
-=== The CLI Solution ===
+== Why requests? ==
-The  current round of patches to the CLIs is to get them all up to the same  level of support for TLSv1 for authentication, at a minimum.  Glance and  Swift continue to use httplib directly for their data transfer  connections (really, all connections to their respective services) and  these already support SSL.v?.
+The requests module [[#footnote2| [2] ]] backported <code><nowiki>match_hostname()</nowiki></code> from Python 3.2.  Like all of the other modules here it does not handle the iPAddress attribute in subjectAltName. This is mostly relevant in development and testing use cases like with DevStack.  The 3.2 <code><nowiki>match_hostname()</nowiki></code> implementation however does allow IP addresses as a dNSName.Requests also brings a number of other features to the table that may or may not have been implemented individually in the existing clients such as JSON encoding/decoding and 3xx redirection support for POST, PUT, PATCH  DELETE, and HEAD.  Plus it is stable (notwithstanding the recent 1.0 release) and the developer is known in the OS community.
 === Original Client HTTP Modules ===
 {| border="1" cellpadding="2" cellspacing="0"
 |< >|Client
 | Module
 | Client object
 | CLI Arg
 |-
 | keystone
 | httplib2
 | class.HTTPClient(httplib2.Http)
 | --os-cacert
 |-
 | nova
 | httplib2
 | class.HTTPClient(httplib2.Http)
 | n/a
 |-
 | cinder
 | httplib2
 | class.HTTPClient(httplib2.Http)
 | n/a
 |-
 | glance
 | httplib
 | class.HTTPClient(object)
 | --ca-file
 |-
 | swift
 | httplib
 | class.HTTPClient(object)
 | n/a
 |-
 | quantum
 | httplib2
 | class.HTTPClient(httplib2.Http)
 | n/a
 |}
-The  approach taken for the httplib2 subclasses is to change the parent  class to object and reqork the request() method to call  requests.request().  Some of the differences for requests leaked out of  that method but have been mostly containd within the HTTPClient class.   All four of the clients (formetly) using httplib2 have implemented one  or more features that can easily be handled by requests (redirection) or  should also be propogated to the other clients.  This is ripe for a  refactor of HTTPClient to a common module but that effort is not in  scope here.keystoneclient (complete) https://review.openstack.org/#/c/17624/ * replace httplib2 with requests
+The  approach taken for the <code><nowiki>httplib2</nowiki></code> subclasses is to change the parent class to object and rework the <code><nowiki>request()</nowiki></code> method to call <code><nowiki>requests.request()</nowiki></code>.  Some of the differences for <code><nowiki>requests</nowiki></code> leaked out of  that method but have been mostly containd within the <code><nowiki>HTTPClient</nowiki></code> class.   All four of the clients (formerly) using <code><nowiki>httplib2</nowiki></code> have implemented one or more features that can easily be handled by <code><nowiki>requests</nowiki></code> (such as redirection) or should also be propogated to the other clients.  This is ripe for a refactor of <code><nowiki>HTTPClient</nowiki></code> to a common module but that effort is not in scope here.
-novaclient (complete) https://review.openstack.org/#/c/18257/ * replace httplib2 with requests
+=== Client Update Status ===
+keystoneclient (complete) https://review.openstack.org/17624
+* replace httplib2 with requests
+novaclient (complete) https://review.openstack.org/18257
+* replace httplib2 with requests
 * add --os-cacert and OS_CACERT support
-* provide ca_cert to keystone clinet for authentication
+* provide ca_cert to keystone client for authentication
-cinderclient (complete) https://review.openstack.org/#/c/18278/ * replace httplib2 with requests
+cinderclient (complete) https://review.openstack.org/18278
+* replace httplib2 with requests
 * add --os-cacert and OS_CACERT support
-* provide ca_cert to keystone clinet for authentication
+* provide ca_cert to keystone client for authentication
+glanceclient (complete) https://review.openstack.org/17698
-glanceclient (complete) https://review.openstack.org/#/c/17698/ * rename --ca-cert to --os-cacert and add OS_CACERT
+* rename --ca-cert to --os-cacert and add OS_CACERT
 * provide ca_cert to keystone client for authentication
-swiftclient (complete) https://review.openstack.org/#/c/18393/ * add --os-cacert and OS_CACERT support
+swiftclient (complete) https://review.openstack.org/18393
+* add --os-cacert and OS_CACERT support
 * provide ca_cert to keystone client for authentication
-quantumclient (not started) * replace httplib2 with requests
+quantumclient (not started)
+* replace httplib2 with requests
 * add --os-cacert and OS_CACERT support
-* provide ca_cert to keystone clinet for authentication
+* provide ca_cert to keystone client for authentication
 === Testing ===
-Aside  from the usual unit tests, support for a TLS proxy is being added to  [[DevStack]] to demonstrate and test a TLS-enabled [[OpenStack]] configuration.   It uses stud as the TLS endpoint that proxies to the usual service  endpoints.  The most interesting challenge here is doing it all on a  single host and making the service catalog work.  Yay!The TLS-in-[[DevStack]] also builds a two-tiered CA (root and intermediate) for testing proper certificate chain validation.
+Aside from the usual unit tests, support for a TLS proxy is being added to DevStack to demonstrate and test a TLS-enabled [[OpenStack]] configuration.   It uses stud as the TLS endpoint that proxies to the usual service endpoints.  The most interesting challenge here is doing it all on a single host and making the service catalog work.  Yay!  The TLS-in-DevStack also builds a two-tiered CA (root and intermediate) for testing proper certificate chain validation.
 === Links ===
-* glanceclient host_matches_cert(): https://review.openstack.org/#/c/16305/4/glanceclient/common/http.py* request's urllib3.ssl_match_hostname.match_hostname(): https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L72https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/packages/ssl_match_hostname/<u>init</u>.py#L23[1]: http://bugs.python.org/issue1589, the comments about 2.x begin at http://bugs.python.org/issue1589#msg120946[2]:  The patches for the [[OpenStack]] CLIs were engineered and implemented  before the release of requests 1.0 which is a significantly different  implementation and is untested in our application.-- dtroyer@gmail.com
+* glanceclient host_matches_cert():
+** https://review.openstack.org/#/c/16305/4/glanceclient/common/http.py
+* request's urllib3.ssl_match_hostname.match_hostname():
+** https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L72
+** https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/packages/ssl_match_hostname/<u>init</u>.py#L23
+<span id="footnote1"></span>[1]:http://bugs.python.org/issue1589, the comments about 2.x begin at http://bugs.python.org/issue1589#msg120946
+<span id="footnote2"></span>[2]:The patches for the [[OpenStack]] CLIs were engineered and implemented  before the release of requests 1.0 which is a significantly different  implementation and is untested in our application.
+--
+dtroyer@gmail.com

Difference between revisions of "SecureClientConnections"