TricircleHowToReadCode

Experience the multi-region Tricircle networking first
Follow the installation guide: https://github.com/openstack/tricircle/blob/master/doc/source/multi-pod-installation-devstack.rst

Please note: the central Neutron in the source code sometimes is called "top", local neutron as "bottom".

The source code involved in the installation guide is described as follows:

Step 5: Create pod instances
Command issued:

curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \ -H "X-Auth-Token: $token" -d '{"pod": {"region_name": "CentralRegion"}}' curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \ -H "X-Auth-Token: $token" -d '{"pod": {"region_name": "RegionOne", "az_name": "az1"}}'


 * The restful API request will be routed to V1Controller from RootController for the version is 'v1.0':

tricircle/tricircle/api/controllers/root.py

... class RootController(object): ...     @expose def _lookup(self, version, *remainder): if version == 'v1.0': return V1Controller, remainder


 * In V1Controller, because the request is to create pod, so the request will be forwarded to the post function in PodsController. Post function will create the pod and az mapping record in pod table:

tricircle/tricircle/api/controllers/pod.py

... class PodsController(rest.RestController): ...   @expose(generic=True, template='json') def post(self, **kw): context = t_context.extract_context_from_environ ...       new_pod = core.create_resource(                    context, models.Pod,                    {'pod_id': _uuid,                     'region_name': region_name,                     'pod_az_name': pod_az_name,                     'dc_name': dc_name,                     'az_name': az_name})

Step 6: Create network/subnet in central Neutron server
The central_plugin.py will be called because the core plugin of central Neutron is configured like this:

core_plugin = tricircle.network.central_plugin.TricirclePlugin

Command issued:

neutron --os-region-name=CentralRegion net-create net1 neutron --os-region-name=CentralRegion subnet-create net1 10.0.1.0/24


 * The request is handled by create_network when net1 is to be created:

tricircle/tricircle/network/central_plugin.py

... class TricirclePlugin(db_base_plugin_v2.NeutronDbPluginV2,                      security_groups.TricircleSecurityGroupMixin,                      external_net_db.External_net_db_mixin,                      portbindings_db.PortBindingMixin,                      extradhcpopt_db.ExtraDhcpOptMixin,                      l3_db.L3_NAT_dbonly_mixin,                      l3_attrs_db.ExtraAttributesMixin): ...  def create_network(self, context, network): net_data = network[attributes.NETWORK] ...


 * The request is handled by create_subnet when 10.0.1.0/24 subnet is created in net1:

tricircle/tricircle/network/central_plugin.py

... class TricirclePlugin(db_base_plugin_v2.NeutronDbPluginV2,                      security_groups.TricircleSecurityGroupMixin,                      external_net_db.External_net_db_mixin,                      portbindings_db.PortBindingMixin,                      extradhcpopt_db.ExtraDhcpOptMixin,                      l3_db.L3_NAT_dbonly_mixin,                      l3_attrs_db.ExtraAttributesMixin): ... def create_subnet(self, context, subnet): subnet_data = subnet['subnet'] ...

Step 7: Get image ID and flavor ID which will be used in VM booting
Command issued:

glance --os-region-name=RegionOne image-list nova --os-region-name=RegionOne flavor-list

No source code of Tricircle will be involved in this step.

Step 8: Boot virtual machines
Command issued:

nova --os-region-name=RegionOne boot --flavor 1 --image $image1_id --nic net-id=$net1_id vm1

Please note, Nova in RegionOne is configured to use local Neutron in RegionOne, and RegionTwo's Nova will use local Neutron in RegionTwo. So different local Neutron will be called when booting instances in different Region.

In RegionOne local Neutron's configuration, the core plugin is configured like this:

core_plugin = tricircle.network.local_plugin.TricirclePlugin


 * When nova boot is executed in RegionOne Nova, Nova will send networks query with network id to local Neutron, and local plugin will send networks query request to central Neutron:

tricircle/tricircle/network/local_plugin.py

... def get_networks(self, context, filters=None, fields=None,                     sorts=None, limit=None, marker=None, page_reverse=False): ... t_networks = raw_client.list_networks(**params)['networks']


 * raw_client.list_networks is to query networks in central Neutron, if the network existed in central Neutron, local plugin will create network/subnet with same uuid as that in central Neutron:

tricircle/tricircle/network/local_plugin.py

... def get_networks(self, context, filters=None, fields=None,                     sorts=None, limit=None, marker=None, page_reverse=False): ...               b_network = self.core_plugin.create_network(                    context, {'network': network}) subnet_ids = self._ensure_subnet(context, network)


 * later nova in RegionOne will issue a request to create a port in local Neutron in RegionOne, the creation process will create the port in central Neutron first, then create the port and security group in local Neutron with same uuid.

tricircle/tricircle/network/local_plugin.py

... def create_port(self, context, port): ...           self._adapt_port_body_for_client(port['port']) t_port = raw_client.create_port(port)['port'] ....   self._handle_security_group(t_ctx, context, t_port) b_port = self.core_plugin.create_port(context, {'port': t_port})


 * next step, nova in RegionOne will issue a request to update port binding in local Neutron in RegionOne, the binding process in local Neutron plugin will forward the port update to central Neutron, and then update local port too.

tricircle/tricircle/network/local_plugin.py

... def update_port(self, context, _id, port): ...           t_ctx = t_context.get_context_from_neutron_context(context) self.neutron_handle.handle_update(t_ctx, 'port', _id,                                             {'port': update_dict}) return self.core_plugin.update_port(context, _id, port)


 * Once central Neutron receives the port update request, need to create resource routing entries in routing table.

tricircle/tricircle/network/central_plugin.py

... def update_port(self, context, port_id, port): t_ctx = t_context.get_context_from_neutron_context(context) top_port = super(TricirclePlugin, self).get_port(context, port_id) ...          for resource_id, resource_type in entries: if db_api.get_bottom_id_by_top_id_region_name(                       t_ctx, resource_id, pod['region_name'], resource_type): continue db_api.create_resource_mapping(t_ctx, resource_id, resource_id,                                              pod['pod_id'], res['tenant_id'],                                               resource_type)


 * If the network has interface connected to a router, then start asynchronous job to (create if not exist in RegionOne) setup the router in RegionOne.

tricircle/tricircle/network/central_plugin.py

... def update_port(self, context, port_id, port): ...      interfaces = super(TricirclePlugin, self).get_ports(                context,                {'network_id': [res['network_id']],                 'device_owner': [constants.DEVICE_OWNER_ROUTER_INTF]}) interfaces = [inf for inf in interfaces if inf['device_id']] if interfaces: # request may be come from service, we use an admin context # to run the xjob admin_context = t_context.get_admin_context self.xjob_handler.setup_bottom_router(                   admin_context, res['network_id'],                    interfaces[0]['device_id'], pod['pod_id'])


 * If the network has interface connected to a router, xjob_handler.setup_bottom_router will register the asynchronous job and send one RPC message to xmanager.py to trigger the asynchronous job, which will setup the router in RegionOne.

tricircle/tricircle/xjob/xmanager.py

...   @_job_handle(constants.JT_ROUTER_SETUP) def setup_bottom_router(self, ctx, payload): (b_pod_id,        t_router_id, t_net_id) = payload[constants.JT_ROUTER_SETUP].split('#') ...


 * Except the setup_bottom_router async. job, there are still other resources need to be check in update_port function in central_plugin.py, whether to create or update these resources in RegionOne, for example, security group.

tricircle/tricircle/network/central_plugin.py

... def update_port(self, context, port_id, port): ...      self.xjob_handler.configure_security_group_rules(t_ctx,                                                             res['tenant_id'])

xjob_handler.configure_security_group_rules will register the security group rules asynchronous job, and then send RPC message to trigger the async.job in xmanager.py.


 * You can find the configure_security_group_rules job in xmanager.py.

tricircle/tricircle/xjob/xmanager.py

...  @_job_handle(constants.JT_SEG_RULE_SETUP) def configure_security_group_rules(self, ctx, payload): project_id = payload[constants.JT_SEG_RULE_SETUP] top_client = self._get_client sg_filters = [{'key': 'tenant_id', 'comparator': 'eq', 'value': project_id}]

The router setup and security group rules configuration (and other resources will be included later) are handled as async. jobs in XJOB process, the reason is to enhance the experience instances booting. If all this resources are created in local_plugin, then it'll take long time and may lead to the time out of booting, bad user experience and low reliability. At the same time, through the aync. job redo and reliability consideration design, the reliability of the networking automation process is ensured.

If you boot a VM2 in RegionTwo, for example

nova --os-region-name=RegionTwo boot --flavor 1 --image $image2_id --nic net-id=$net2_id vm2

Then the local plugin under RegionTwo local Neutron will be called, the workflow is same as that in RegionOne. And the asynchronous jobs in central Neutron will setup resources like router (if the network has interface on the router) and security group rules, etc in RegionTwo. Please be aware of that if cross OpenStack L2 network is used in the VM2 booting, then resources setup to RegionOne and RegionTwo may be occurred in the asynchronous jobs. This is the networking automation to coordinate multiple OpenStack networking resources.

Step 9: Verify the VMs are connected to the networks
Tricircle is not involved in this step.

Step 10: Create external network and subnet
Command issued: curl -X POST http://127.0.0.1:20001/v2.0/networks -H "Content-Type: application/json" \ -H "X-Auth-Token: $token" \ -d '{"network": {"name": "ext-net", "admin_state_up": true, "router:external": true, "provider:network_type": "vlan", "provider:physical_network": "extern", "availability_zone_hints": ["RegionTwo"]}}' neutron --os-region-name=CentralRegion subnet-create --name ext-subnet --disable-dhcp ext-net 163.3.124.0/24


 * For the external network, it'll be created by cloud admin, and need to specify in which region the external network will reside. create_network will check the network creation is for external network or not, if yes, forward the external network creation request to regarding region. Because RegionTwo is specified, so the request will be forwarded to RegionTwo local Neutron.

tricircle/tricircle/network/central_plugin.py

... def create_network(self, context, network): net_data = network[attributes.NETWORK] tenant_id = net_data['tenant_id'] is_external = self._ensure_az_set_for_external_network(context,                                                              net_data) ...            if is_external: self._fill_provider_info(res, net_data) self._create_bottom_external_network(                   context, net_data, res['id'])

Step 11: Create router and attach subnets in central Neutron server
Command issued

neutron --os-region-name=CentralRegion router-create router neutron --os-region-name=CentralRegion router-interface-add router $subnet1_id neutron --os-region-name=CentralRegion router-interface-add router $subnet2_id

Please note that in step 6, although there is code to setup router in update_port, but because the network has no interface on router in step 6, so the asynchronous job for router setup is not called. When router-interface-add is executed now, the code add_router_interface in central plugin will be called, and start asynchronous job to setup the router in RegionOne and/or RegionTwo, and bridge network will be created if needed:

tricircle/tricircle/network/central_plugin.py

... def add_router_interface(self, context, router_id, interface_info): t_ctx = t_context.get_context_from_neutron_context(context) router = self._get_router(context, router_id) ....       if len(b_pods) == 1: self.xjob_handler.setup_bottom_router(                   t_ctx, net_id, router_id, b_pods[0]['pod_id']) else: self.xjob_handler.setup_bottom_router(                   t_ctx, net_id, router_id, t_constants.POD_NOT_SPECIFIED)

Step 12: Set router external gateway in central Neutron server
Command issued

neutron --os-region-name=CentralRegion router-gateway-set router ext-net

The update_router in central_plugin will be called, and router gateway will be set in central Neutron first, then set router gateway in RegionTwo where the ext-net resides, extra route "destination, next hop" for L3 networking across Neutron will be set too through asynchronous job.

tricircle/tricircle/network/central_plugin.py

... def update_router(self, context, router_id, router): router_data = copy.deepcopy(router['router']) ...       if is_add: ret = super(TricirclePlugin, self).update_router(               context, router_id, router) router_data[l3.EXTERNAL_GW_INFO].update(ret[l3.EXTERNAL_GW_INFO]) self._add_router_gateway(context, router_id, router_data) ...       t_ctx = t_context.get_context_from_neutron_context(context) self.xjob_handler.configure_extra_routes(t_ctx, router_id)

Step 13: Launch VNC console and test connection
No code of Tricircle will be called in this step.

Step 14: Create floating ip in central Neutron server
Command issued

neutron --os-region-name=CentralRegion floatingip-create ext-net

The create_floatingip in central_plugin will be called, and just create a floating IP in central Neutron.

tricircle/tricircle/network/central_plugin.py

... def create_floatingip(self, context, floatingip): # create bottom fip when associating fixed ip       return super(TricirclePlugin, self).create_floatingip(            context, floatingip,            initial_status=constants.FLOATINGIP_STATUS_DOWN) ...

Step 15: Associate floating ip
Command issued

neutron --os-region-name=CentralRegion floatingip-associate $floatingip_id $port_id

The update_floatingip in central_plugin will be called, and associate floating in central Neutron first, then call async. job to associate the floatingip in RegionTwo where the external network resides, shadow port may be created if the port associated to the floatingip does not co-located in same region.

tricircle/tricircle/network/central_plugin.py

... def update_floatingip(self, context, _id, floatingip): ...       org_floatingip_dict = self._make_floatingip_dict(            self._get_floatingip(context, _id)) res = super(TricirclePlugin, self).update_floatingip(           context, _id, floatingip) try: if floatingip['floatingip']['port_id']: self._associate_floatingip(context, _id, floatingip) ... def _associate_floatingip(self, context, _id, floatingip): t_ctx = t_context.get_context_from_neutron_context(context) .....       self.xjob_handler.setup_bottom_router(            t_ctx, net_id, floatingip_db['router_id'], int_net_pod['pod_id'])