|
|
| Line 1: |
Line 1: |
| − | == Repeated token revocation requests, can lead to service degradation or disruption ==
| + | <pending> |
| − | | |
| − | === Summary ===
| |
| − | There is currently no limit to the frequency of keystone token revocations that
| |
| − | can be made by a single user, in any given time frame. If a user repeatedly
| |
| − | makes token requests, and then immediately revokes the token, a performance
| |
| − | degradation can occur and possible DoS (Denial of Serice) attacks could be
| |
| − | directed towards keystone.
| |
| − | | |
| − | === Affected Services / Software ===
| |
| − | All services using keystone.
| |
| − | Mitaka, Liberty, Kilo, Nova, Juno, Havana, Icehouse, Grizzly, Folsom, Essex.
| |
| − | | |
| − | === Discussion ===
| |
| − | Token revocation can be self-served, with no restrictions enforced on the
| |
| − | number of token revocations made by any user (including service users).
| |
| − | | |
| − | If token revocations are made in quick succession, response times starts to lengthen, due
| |
| − | to the increasing entries made in the revocation_event table.
| |
| − | | |
| − | With no form of rate limiting in place, a single user can cause the OpenStack
| |
| − | auth service to become poor in response time, resulting in a DoS style attack.
| |
| − | | |
| − | A cleanup of revocation events does occur, based on token expiration plus
| |
| − | expiration_buffer (which is 30 minutes by default). However, with the default
| |
| − | token TTL of 3600 seconds, a user can potentially fill up approximately several
| |
| − | thousand events during that time.
| |
| − | | |
| − | === Recommended Actions ===
| |
| − | For current stable OpenStack releases (Mitaka and previous), operators are
| |
| − | recommended to deploy external rate-limiting proxies or web application firewalls, to
| |
| − | provide a front layer of protection to keystone.
| |
| − | | |
| − | The following solutions may be considered, however it is key that the operator
| |
| − | carefully plans and considers the individual performance needs of users
| |
| − | and services within their OpenStack cloud, when configuring any rate limiting
| |
| − | functionality.
| |
| − | | |
| − | ==== Repose ====
| |
| − | | |
| − | ===== Rate Limiting Filter =====
| |
| − | Repose provides a rate limiting filter, that can limit per IP address and
| |
| − | to a specific HTTP method (DELETE in relation to this OSSN).
| |
| − | | |
| − | The following config may be considered for a single node. For more complex
| |
| − | deployments, clusters can be constructed , utilizing a distributed data-store.
| |
| − | | |
| − | ===== system-model.cfg.xml =====
| |
| − | | |
| − | <?xml version="1.0" encoding="UTF-8"?>
| |
| − | <system-model xmlns="<nowiki>http://docs.openrepose.org/repose/system-model/v2.0</nowiki>">
| |
| − | <repose-cluster id="repose">
| |
| − | <nodes>
| |
| − | <node id="repose_node1" hostname="localhost" http-port="8080"/>
| |
| − | </nodes>
| |
| − | <filters>
| |
| − | <filter name="ip-user"/>
| |
| − | <filter name="rate-limiting"/>
| |
| − | </filters>
| |
| − | <services>
| |
| − | </services>
| |
| − | <destinations>
| |
| − | <endpoint id="keystone" protocol="http" hostname="<nowiki>http://idenity-server.acme.com</nowiki>" root-path="/" port="35357" default="true"/>
| |
| − | </destinations>
| |
| − | </repose-cluster>
| |
| − | <phone-home enabled="false"
| |
| − | origin-service-id="your-service"
| |
| − | contact-email="your@service.com"/>
| |
| − | </system-model>
| |
| − | | |
| − | ===== ip-user.cfg.xml =====
| |
| − | | |
| − | <?xml version="1.0" encoding="UTF-8"?>
| |
| − | <ip-user xmlns="<nowiki>http://docs.openrepose.org/repose/ip-user/v1.0</nowiki>">
| |
| − | <user-header name="X-PP-User" quality="0.4"/>
| |
| − | <group-header name="X-PP-Groups" quality="0.4"/>
| |
| − | <group name="ipv4-group">
| |
| − | <cidr-ip>192.168.0.0/24</cidr-ip>
| |
| − | </group>
| |
| − | <group name="match-all">
| |
| − | <cidr-ip>0.0.0.0/0</cidr-ip>
| |
| − | <cidr-ip>0::0/0</cidr-ip>
| |
| − | </group>
| |
| − | </ip-user>
| |
| − | | |
| − | | |
| − | * Note: Using the ip-user filter, will mean each IP address sending requests to
| |
| − | repose, will have its own rate-limit bucket. Therefore any IP exceeding the
| |
| − | limit, will be blocked - but only that IP. If you are sending NAT'ed connections
| |
| − | to repose, then you should consider, they will also be seen as a single IP, and
| |
| − | grouped accordingly.
| |
| − | | |
| − | ===== rate-limiting.cfg.xml =====
| |
| − | | |
| − | <?xml version="1.0" encoding="UTF-8"?>
| |
| − | <rate-limiting xmlns="<nowiki>http://docs.openrepose.org/repose/rate-limiting/v1.0</nowiki>">
| |
| − | <request-endpoint uri-regex="/limits" include-absolute-limits="false"/>
| |
| − | <global-limit-group>
| |
| − | <limit id="global" uri="*" uri-regex=".*" value="1000" unit="MINUTE"/>
| |
| − | </global-limit-group>
| |
| − | <limit-group id="limited" groups="limited" default="true">
| |
| − | <limit id="all" uri="/auth/token" uri-regex="/.*" http-methods="DELETE" unit="MINUTE" value="10"/>
| |
| − | </limit-group>
| |
| − | <limit-group id="unlimited" groups="unlimited" default="false"/>
| |
| − | </rate-limiting>
| |
| − | </code>
| |
| − | | |
| − | Key points to note with the above. The rate limit is limited to DELETE
| |
| − | requests (which is the http method used to revoke a token), and to the URI
| |
| − | /auth/token. Any IP which exceeds 10 revoke requests per minute, will be blocked
| |
| − | for 1 minute.
| |
| − | | |
| − | Further details can be found on the openrepose wiki:
| |
| − | | |
| − | https://repose.atlassian.net/wiki/display/REPOSE/Rate+Limiting+filter
| |
| − | | |
| − | | |
| − | ===Other possible solutions===
| |
| − | | |
| − | ==== NGINX ====
| |
| − | NGINX provides the limit_req_module, which can be used to provide a global rate
| |
| − | limit. Using a map, it can be limited to just the DELETE method.
| |
| − | | |
| − | =====nginx.conf=====
| |
| − | http {
| |
| − | map $request_method $keystone {
| |
| − | default "";
| |
| − | DELETE $binary_remote_addr;
| |
| − | }
| |
| − | limit_req_zone $keystone zone=keystone:10m rate=10r/m;
| |
| − | server {
| |
| − | ...
| |
| − | location /auth/token {
| |
| − | limit_req zone=keystone;
| |
| − | ...
| |
| − | }
| |
| − | }
| |
| − | | |
| − | | |
| − | Further details can be found on the nginx site:
| |
| − | | |
| − | http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
| |
| − | | |
| − | ==== HAProxy ====
| |
| − | | |
| − | HAProxy can provide inherent rate-limiting, using stick-tables, with a General
| |
| − | Purpose Counter (gpc)
| |
| − | | |
| − | =====/etc/init.d/haproxy.cfg=====
| |
| − | # Monitors the number of request sent by an IP over a period of 10 seconds
| |
| − | stick-table type ip size 1m expire 10s store gpc0,http_req_rate(10s)
| |
| − | tcp-request connection track-sc1 src
| |
| − | tcp-request connection reject if { src_get_gpc0 gt 0 }
| |
| − | | |
| − | Further details can be found on the haproxy website:
| |
| − | | |
| − | http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos
| |
| − | | |
| − | ==== Apache ====
| |
| − | A number of solutions can be explored here.
| |
| − | | |
| − | ===== mod_ratelimit =====
| |
| − | http://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html
| |
| − | | |
| − | ===== mod_qos =====
| |
| − | http://opensource.adnovum.ch/mod_qos/dos.html
| |
| − | | |
| − | ===== mod_evasive =====
| |
| − | https://www.digitalocean.com/community/tutorials/how-to-protect-against-dos-and-ddos-with-mod_evasive-for-apache-on-centos-7
| |
| − | | |
| − | ===== mod_security =====
| |
| − | https://www.modsecurity.org/
| |
| − | | |
| − | ==== Contacts / References ====
| |
| − | * Author: Luke Hinds, Red Hat
| |
| − | * This OSSN : https://bugs.launchpad.net/ossn/+bug/1553324
| |
| − | * Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1553324
| |
| − | * OpenStack Security ML : openstack-security@lists.openstack.org
| |
| − | * OpenStack Security Group : https://launchpad.net/~openstack-ossg
| |
