MitmStack

Purpose

OpenStack is capable to manage large scale IT infrastructures. But it becomes a challenge when you want to test the client applications and services for performance before deployment.

Developer might be curious about whether it will perform well against thousands hosts or even more. It is such a luxury to create large scale OpenStack clusters in laboratory that sometimes the only thing developers could do is to pray. Pray for a successful deployment in production environment.

MitmStack aims to simulate large scale openstack cluster with limited hardware. It works together with Vitrage, the RCA service for OpenStack to explore the solution for cheating the client with faked responses crafted from actual traffic.

Proof of Concept

  CUT   |  mitmstack
+-----+ | +-------+   +---------+   +--------+   +-----------+
|     +-+->       +--->         +--->        +--->          |
| CUT | | | Faked |   | Crafter |   | Actual |   | OpenStack |
|     <-+-+       <---+         <---+        <---+          |
+-----+ | +-------+   +---------+   +--------+   +-----------+
        |

CUT

OpenStack client under test

Faked Facade

Cache of actual request and mocked response

• input: actual request from CMS
• output: return cached response if hit, else forward the request to Crafter to get one

The cache is indexed on full request.

Crafter

Fake a response from a matched record in Actual Repository

• input: request missed in Faked Facade
• output: faked response based on matched record in Actual Repository

To create a fake response The request need to be modified in order to route it to a valid devstack instance. It may include not only the content of request, but also the time characteristics.

Actual Repository

Cache of actual request and actual response

• input: modified request by Crafter
• output: return cached response if hit, else forward the request to devstack to get one

The cache can be index on API endpoints, i.e. one record for each API.

devstack

An actual devstack instance to provide base response

• input: request missed in Actual Repository
• output: response from devstack

Faker engine

A fake response could be generated by randomizing the values in a actual response. But it could be difficult to manage the mapping, especially when there are multi-level dependency, e.g. when the OSC received a faked response, it may send a second step request and expecting a response related to the previous one.

Scale Up

Multiple devstack could be used to provide more samples of actual traffic.

Implementation

• Faked Facade: virtual hosts on nginx
• Crafter: mitmproxy with scripting
• Actual Repository: key-value storage in redis