Jump to: navigation, search

Tricircle

Revision as of 01:30, 18 September 2016 by Chaoyi Huang (talk | contribs)

The Tricircle is to provide networking automation across Neutron in multi-region OpenStack deployment.

~~ Attention ~~ During the big-tent application of Tricircle: https://review.openstack.org/#/c/338796/, the proposal is to move API-gateway part away from Tricircle, and form two independent and decoupled projects:

Tricircle: Dedicated for cross Neutron networking automation in multi-region OpenStack deployment, run without or with Trio2o.

Trio2o: Dedicated to provide API gateway for those who need single Nova/Cinder API endpoint in multi-region OpenStack deployment, run without or with Tricircle.

The wiki for Tricircle before splitting is linked here: https://wiki.openstack.org/wiki/tricircle_before_splitting

Use Cases

Cloud Capacity Expansion

When one OpenStack cloud is provisioned with more and more resources, the capacity of one OpenStack instance may not be enough. A new openstack instance has to be added to the cloud. But tenants still want to add VMs to existing network. And they don’t want to add a new network, router or whatever required resources.

Tricircle usecase1


Application high availability running over cloud

In telecom and other industries(for example, MySQL Galera cluster), normally applications tend to be designed as Active-Active, Active-Passive or N-to-1 node configurations. Because those applications need to be available as much as possible. Once those applications are planned to be migrated to cloud infrastructure, active instance(s) need(s) to be deployed in one OpenStack instance first. After that, passive or active instances(s) will be deployed in another (other) OpenStack instance(s) in order for achieving 99.999% availability, if it’s required.

The reason why this deployment style is required is that in general cloud system only achieve 99.99% availability. [3] [4]

To achieve required high availability, design of network architecture (especially Layer 2 and 3) needs to be considered to use state replication, heartbeat.

Tricircle usecase2


The above picture shows an application with Galera DB cluster as backend which are geographically distributed into multiple OpenStack instances.

Isolation of East-West traffic

In financial industry, more than one OpenStack instances will be deployed, some OpenStack instance will be put in DMZ zone, and the other ones in trust zone, so that application or part of the application could be put in different level security zone. Although the tenant’s resources are provisioned in different OpenStack instances, east-west traffic among the tenant’s resources should be isolated.

Tricircle usecase3


Bandwidth sensitive, heavy load App like CAD modeling asked for the cloud close to the end user in each edge site, but the east-west communication with isolation between the tenant’s resources are also required.

Tricircle usecase4


Dual ISPs Load Balancing for internet link

Deploy application in separate OpenStack instance with dual ISPs for internet link redundancy, load balancing, east-west traffic isolation for data/state replication is needed.

Tricircle usecase5


Cross Neutron L2 network for NFV(Network Function Virtualization) area

IN NFV area, network functions like router, NAT, LB etc will be virtualized, The cross Neutron L2 networking capability like IP address space management, IP allocation and L2 network segment global management provided by the Tricircle can help VNFs(virtualized network function) across site inter-connection. For example, vRouter1 in site1, but vRouter2 in site2, these two VNFs could be in one L2 network across site[5].


Overview

The Tricircle provides an OpenStack API gateway and networking automation to allow multiple OpenStack instances, spanning in one site or multiple sites or in hybrid cloud, to be managed as a single OpenStack cloud.

The Tricircle and these managed OpenStack instances will use shared KeyStone (with centralized or distributed deployment) or federated KeyStones for identity management. The Tricircle presents one big region to the end user in KeyStone. And each OpenStack instance called a pod is a sub-region of the Tricircle in KeyStone, and usually not visible to end user directly.

The Tricircle acts as OpenStack API gateway, can handle OpenStack API calls, schedule one proper OpenStack instance if needed during the API calls handling, forward the API calls to the appropriate OpenStack instance, and deal with tenant level L2/L3 networking across OpenStack instances automatically, so that the VMs of the tenant, no matter located in which bottom OpenStack instance, can communicate with each other via L2 or L3.

The end user can see availability zone(AZ) and use AZ to provision VM, Volume, even Network through the Tricircle. One AZ can include many OpenStack instances, the Tricircle can schedule and bind OpenStack instance for the tenant inside one AZ. A tenant's resources could be bound to multiple specific bottom OpenStack instances in one or multiple AZs automatically.

The Tricircle is the formal open source project for OpenStack cascading solution ( https://wiki.openstack.org/wiki/OpenStack_cascading_solution ) but with enhanced and decoupled design.

The Tricircle could be extended to support more powerful capabilities such as support the Tricircle instance being virtually splitted into multiple micro instances which could enable user to have a more fine granularity on the tenancy and service. And the Tricircle also enables OpenStack based hybrid cloud.

Architecture

Now the Tricircle is designed as standalone service which is decoupled from OpenStack existing services like Nova, Cinder, Neutron. The design blueprint has been developed with ongoing improvement in https://docs.google.com/document/d/18kZZ1snMOCD9IQvUKI5NVDzSASpw-QKj7l2zNqMEd3g/edit?usp=sharing,


Tricircle improved architecture design - stateless


Sared KeyStone (centralized or distributed deployment) or federated KeyStones could be used for identity management for the Tricrcle and managed OpenStack instances. The Tricircle presents one big region to the end user in KeyStone. And each OpenStack instance called a pod is a sub-region of the Tricircle in KeyStone, and usually not visible to end user directly.

  • Nova API-GW
  1. A standalone web service to receive all nova API request, and routing the request to appropriate bottom OpenStack instance according to Availability Zone ( during creation ) or VM's uuid ( during operation and query ). If more than one OpenStack instance in one Availability Zone, schedule one and forward the request to proper OpenStack instance, and build the binding relationship beween tenant ID and OpenStack instance.
  2. Nova APIGW is the functionality to trigger networking automation when new VMs are being provisioned.
  3. work as stateless service, and could run with processes distributed in multi-hosts.
  • Cinder API-GW
  1. a standalone web service to receive all cinder API request, and route the request to appropriate bottom OpenStack instance according to Availability Zone ( during creation ) or resource id like volume/backup/snapshot uuid ( during operation and query ). If more than one OpenStack instance in one Availability Zone, schedule one OpenStack instance and forward the request to proper OpenStack instance, and build binding relationship between the tenant ID and OpenStack instance.
  2. Cinder APIGW and Nova APIGW will make sure the volumes for the same VM will co-locate in same OpenStack instance.
  3. work as stateless service, and could run with processes distributed in multi-hosts.
  • Neutron API Server
  1. Neutron API Server is reused from Neutron to receive and handle Neutron API request.
  2. Neutron Tricircle Plugin. It runs under Neutron API server in the same process like OVN Neutron plugin. The Tricircle plugin serve for tenant level L2/L3 networking automation across multi-OpenStack instances. It will use driver interface to call bottom OpenStack Neutron API and L2GW API if needed, especially for cross OpenStack mixed VLAN / VxLAN L2 networking.
  • Admin API
  1. manage sites(bottom OpenStack instances) and availability zone mappings.
  2. Retrieve object uuid routing.
  3. Expose API for maintenance.
  • XJob
  1. Receive and process cross OpenStack functionalities and other async. jobs from Nova API-GW, or Cinder API-GW, Admin API or Neutron Tricircle Plugin. For example, when booting a VM for the first time for the tenant, router, security group rule, FIP and other resources may have not already been created in the bottom OpenStack instance. But it’s required. Not like network,security group, ssh keypair, other resources they must be created before a VM booting. These resources could be created in async. way to accelerate response for the first VM booting request.
  2. Cross OpenStack networking also will be done in async. jobs.
  3. Any of Admin API, Nova API-GW, Cinder API-GW, Neutron Tricircle plugin could send an async. job to XJob through message bus with RPC API provided by XJob.
  • Database
  1. The Tricircle has its own database to store pods, pod-bindings, jobs, resource routing tables.
  2. Neutron Tricircle plugin reuse DB of Neutron, for one tenant’s network, router will be spread into multiple OpenStack instances, and managing tenant level IP/mac address to avoid conflict across different OpenStack instances.

For Glance deployment, there are several choice:

  • Shared Glance, if all OpenStack instances are located inside a high bandwidth, low latency site.
  • Shared Glance with distributed back-end, if OpenStack instances are located in several sites.
  • Distributed Glance deployment, Glance service is deployed distributed in multiple site with distributed back-end
  • Separate Glance deployment, each site is installed with separate Glance instance and back-end, no cross site image sharing is needed.

Value

The motivation to develop the Tricircle open source project:

The cascading solution based on PoC design with enhancement has been running in several production clouds, which showed the value of one OpenStack API gateway layer with networking automation functionality above multiple OpenStack instances, no matter in large scale centralized cloud scenario, or distributed enterprise application located in the distributed edge clouds, even hybrid cloud scenario:

  • OpenStack API eco-system reserved, from CLI, SDK to Heat, Murano, Magum etc, all of these could be reused seamlessly.
  • support modularized capacity expansion in large scale cloud.
  • L2/L3 networking automation across OpenStack instances.
  • Tenant's VMs communicate with each other via L2 or L3 networking across OpenStack instances.
  • Security group applied across OpenStack instances.
  • Tenant level IP/mac addesses management to avoid conflict across OpenStack instances.
  • Tenant level quota control across OpenStack instances.
  • Global resource usage view across OpenStack instances.
  • User level KeyPair management across OpenStack instances.
  • Tenant's data movement across OpenStack instances thanks to the tenant level L2/L3 networking.
  • ...

Installation and Play

Set up the Tricircle in 3 VMs with virtualbox in Ubuntu 14.04 LTS. Install Tricircle in VirtualBox.
Or refer to installation guide in https://github.com/openstack/tricircle for single node/two nodes setup using devstack.

FAQ

Q: What is the difference between Tricircle and OpenStack Cascading?

OpenStack Cascading was mainly a solution used in a PoC done in late 2014 and early 2015, which aims to test out the idea that multiple OpenStack instances COULD be deployed across multiple geo-diverse sites, and managed by an OpenStack API layer, which was based on OpenStack services. After the PoC was carried out successfully, the team then planned to contribute the core idea to the community.

Tricircle was born out of that idea, however got a different shape and focus. Unlike what is implemented in the V1 of OpenStack cascading solution in the PoC, which has plenty twists and plumbers of feature enhancements, Tricircle in its earliest stage tries to build a clean architecture that is extendable, pluggable and reusable in nature.

In short, OpenStack Cascading is a specific deployment solution, while Tricircle represents a standalone project with decoupled group of services, like any other OpenStack project for example Neutron, Nova or Glance, etc, that in the future could be applied to OpenStack Ecosystem.

Q: What is the goal of Tricircle?

In short term, Tricircle would focus on developing a robust architecture and related features, in a long run, we hope we could successfully establish a paradigm that could be applied to the whole OpenStack community

How to read the source code

To read the source code, it's much easier if you follow this blueprint:

Implement Stateless Architecture: https://blueprints.launchpad.net/tricircle/+spec/implement-stateless

This blueprint is to build Tricircle from scratch

Resources


Tricircle is designed to use the same tools for submission and review as other OpenStack projects. As such we follow the OpenStack development workflow. New contributors should follow the getting started steps before proceeding, as a Launchpad ID and signed contributor license are required to add new entries.

Meeting minutes and logs

all meeting logs and minutes could be found in
2016: http://eavesdrop.openstack.org/meetings/tricircle/2016/
2015: http://eavesdrop.openstack.org/meetings/tricircle/2015/

To do list

To do list is in the etherpad: https://etherpad.openstack.org/p/TricircleToDo

Splitting Tricircle into two projects: https://etherpad.openstack.org/p/TricircleSplitting

Team Member

Contact team members in IRC channel: #openstack-tricircle

Current active participants

Joe Huang, Huawei

Khayam Gondal, Dell

Shinobu Kinjo, RedHat

Ge Li, China UnionPay

Vega Cai, Huawei

Pengfei Shi, OMNI Lab

Bean Zhang, OMNI Lab

Yipei Niu, Huazhong University of Science and Technology

Ronghui Cao, Hunan University

Xiongqiu Long, Hunan University

Zhuo Tang, Hunan University

Liuzyu, Hunan University

Jiawei He, Hunan University

KunKun Liu, Hunan University

Yangkai Shi, Hunan University

Yuquan Yue, Hunan University

Howard Huang, Huawei