Jump to: navigation, search

Difference between revisions of "TricircleHowToReadCode"

(Step 8: Boot virtual machines)
 
(10 intermediate revisions by one other user not shown)
Line 1: Line 1:
== (Work in progress)Learn the source code of Tricircle step by step ==
+
== (WIP) Learn the source code of Tricircle step by step ==
  
 
=== Experience the  multi-region Tricircle networking first ===
 
=== Experience the  multi-region Tricircle networking first ===
Line 15: Line 15:
 
   curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \
 
   curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \
 
       -H "X-Auth-Token: $token" -d '{"pod": {"region_name":  "CentralRegion"}}'
 
       -H "X-Auth-Token: $token" -d '{"pod": {"region_name":  "CentralRegion"}}'
 
 
   curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \
 
   curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \
 
       -H "X-Auth-Token: $token" -d '{"pod": {"region_name":  "RegionOne", "az_name": "az1"}}'
 
       -H "X-Auth-Token: $token" -d '{"pod": {"region_name":  "RegionOne", "az_name": "az1"}}'
Line 201: Line 200:
 
                     interfaces[0]['device_id'], pod['pod_id'])
 
                     interfaces[0]['device_id'], pod['pod_id'])
  
* If the network has interface connected to a router,  xjob_handler.setup_bottom_router will register the asynchronous job and send one RPC message to xmanager,py to trigger the asynchronous job, which will setup the router in RegionOne.
+
* If the network has interface connected to a router,  xjob_handler.setup_bottom_router will register the asynchronous job and send one RPC message to xmanager.py to trigger the asynchronous job, which will setup the router in RegionOne.
  
 
   tricircle/tricircle/xjob/xmanager.py
 
   tricircle/tricircle/xjob/xmanager.py
Line 242: Line 241:
 
   nova --os-region-name=RegionTwo boot --flavor 1 --image $image2_id --nic net-id=$net2_id vm2
 
   nova --os-region-name=RegionTwo boot --flavor 1 --image $image2_id --nic net-id=$net2_id vm2
  
Then the local plugin under RegionTwo local Neutron will be called, the workflow is same as that in RegionOne. And the asynchronous jobs will setup resources like router (if the network has interface on the router) and security group rules, etc in RegionTwo. Please be aware of that if cross OpenStack L2 network is used in the VM2 booting, then  resources setup to RegionOne and RegionTwo may be occurred in the asynchronous jobs. This is the networking automation to coordinate multiple OpenStack networking resources.
+
Then the local plugin under RegionTwo local Neutron will be called, the workflow is same as that in RegionOne. And the asynchronous jobs in central Neutron will setup resources like router (if the network has interface on the router) and security group rules, etc in RegionTwo. Please be aware of that if cross OpenStack L2 network is used in the VM2 booting, then  resources setup to RegionOne and RegionTwo may be occurred in the asynchronous jobs. This is the networking automation to coordinate multiple OpenStack networking resources.
  
 
=== Step 9: Verify the VMs are connected to the networks ===
 
=== Step 9: Verify the VMs are connected to the networks ===
  
 
Tricircle is not involved in this step.
 
Tricircle is not involved in this step.
 +
 +
=== Step 10: Create external network and subnet ===
 +
 +
Command issued:
 +
  curl -X POST http://127.0.0.1:20001/v2.0/networks -H "Content-Type: application/json" \
 +
    -H "X-Auth-Token: $token" \
 +
    -d '{"network": {"name": "ext-net", "admin_state_up": true, "router:external": true,  "provider:network_type": "vlan", "provider:physical_network": "extern", "availability_zone_hints": ["RegionTwo"]}}'
 +
  neutron --os-region-name=CentralRegion subnet-create --name ext-subnet --disable-dhcp ext-net 163.3.124.0/24
 +
 +
* For the external network, it'll be created by cloud admin, and need to specify in which region the external network will reside. create_network will check the network creation is for external network or not, if yes, forward the external network creation request to regarding region. Because RegionTwo is specified, so the request will be forwarded to RegionTwo local Neutron.
 +
 +
  tricircle/tricircle/network/central_plugin.py
 +
 +
  ...
 +
  def create_network(self, context, network):
 +
        net_data = network[attributes.NETWORK]
 +
        tenant_id = net_data['tenant_id']
 +
        is_external = self._ensure_az_set_for_external_network(context,
 +
                                                              net_data)
 +
        ...
 +
            if is_external:
 +
                self._fill_provider_info(res, net_data)
 +
                self._create_bottom_external_network(
 +
                    context, net_data, res['id'])
 +
 +
=== Step 11: Create router and attach subnets in central Neutron server ===
 +
 +
Command issued
 +
 +
  neutron --os-region-name=CentralRegion router-create router
 +
  neutron --os-region-name=CentralRegion router-interface-add router $subnet1_id
 +
  neutron --os-region-name=CentralRegion router-interface-add router $subnet2_id
 +
 +
Please note that in step 6, although there is code to setup router in update_port, but because the network has no interface on router in step 6, so the asynchronous job for router setup is not called. When router-interface-add is executed now, the code add_router_interface in central plugin will be called, and start asynchronous job to setup the router in RegionOne and/or RegionTwo, and bridge network will be created if needed:
 +
 +
tricircle/tricircle/network/central_plugin.py
 +
 +
  ...
 +
  def add_router_interface(self, context, router_id, interface_info):
 +
        t_ctx = t_context.get_context_from_neutron_context(context)
 +
        router = self._get_router(context, router_id)
 +
        ....
 +
        if len(b_pods) == 1:
 +
                self.xjob_handler.setup_bottom_router(
 +
                    t_ctx, net_id, router_id, b_pods[0]['pod_id'])
 +
            else:
 +
                self.xjob_handler.setup_bottom_router(
 +
                    t_ctx, net_id, router_id, t_constants.POD_NOT_SPECIFIED)
 +
 +
=== Step 12: Set router external gateway in central Neutron server ===
 +
 +
Command issued
 +
 +
  neutron --os-region-name=CentralRegion router-gateway-set router ext-net
 +
 +
The update_router in central_plugin will be called, and router gateway will be set in central Neutron first, then set router gateway in  RegionTwo where the ext-net resides, extra route "destination, next hop" for L3 networking across Neutron will be set too through asynchronous job .
 +
 +
  tricircle/tricircle/network/central_plugin.py
 +
 +
  ...
 +
  def update_router(self, context, router_id, router):
 +
        router_data = copy.deepcopy(router['router'])
 +
        ...
 +
        if is_add:
 +
            ret = super(TricirclePlugin, self).update_router(
 +
                context, router_id, router)
 +
            router_data[l3.EXTERNAL_GW_INFO].update(ret[l3.EXTERNAL_GW_INFO])
 +
            self._add_router_gateway(context, router_id, router_data)
 +
        ...
 +
        t_ctx = t_context.get_context_from_neutron_context(context)
 +
        self.xjob_handler.configure_extra_routes(t_ctx, router_id)
 +
 +
=== Step 13: Launch VNC console and test connection ===
 +
 +
No code of Tricircle will be called in this step.
 +
 +
=== Step 14: Create floating ip in central Neutron server ===
 +
 +
Command issued
 +
 +
  neutron --os-region-name=CentralRegion floatingip-create ext-net
 +
 +
The create_floatingip in central_plugin will be called, and just create a floating IP in central Neutron.
 +
 +
  tricircle/tricircle/network/central_plugin.py
 +
 +
  ...
 +
  def create_floatingip(self, context, floatingip):
 +
        # create bottom fip when associating fixed ip
 +
        return super(TricirclePlugin, self).create_floatingip(
 +
            context, floatingip,
 +
            initial_status=constants.FLOATINGIP_STATUS_DOWN)
 +
  ...
 +
 +
=== Step 15: Associate floating ip ===
 +
 +
Command issued
 +
 +
  neutron --os-region-name=CentralRegion floatingip-associate $floatingip_id $port_id
 +
 +
The update_floatingip in central_plugin will be called, and associate floating in central Neutron first, then call async. job to associate the floatingip in RegionTwo where the external network resides, shadow port may be created if the port associated to the floatingip does not co-located in same region.
 +
 +
  tricircle/tricircle/network/central_plugin.py
 +
 +
  ...
 +
def update_floatingip(self, context, _id, floatingip):
 +
...
 +
        org_floatingip_dict = self._make_floatingip_dict(
 +
            self._get_floatingip(context, _id))
 +
        res = super(TricirclePlugin, self).update_floatingip(
 +
            context, _id, floatingip)
 +
        try:
 +
            if floatingip['floatingip']['port_id']:
 +
                self._associate_floatingip(context, _id, floatingip)
 +
  ...
 +
  def _associate_floatingip(self, context, _id, floatingip):
 +
        t_ctx = t_context.get_context_from_neutron_context(context)
 +
        .....
 +
        self.xjob_handler.setup_bottom_router(
 +
            t_ctx, net_id, floatingip_db['router_id'], int_net_pod['pod_id'])

Latest revision as of 03:11, 22 February 2017

(WIP) Learn the source code of Tricircle step by step

Experience the multi-region Tricircle networking first

Follow the installation guide: https://github.com/openstack/tricircle/blob/master/doc/source/multi-pod-installation-devstack.rst

Please note: the central Neutron in the source code sometimes is called "top", local neutron as "bottom".

The source code involved in the installation guide is described as follows:

Step 1 ~ Step 4: no code of Tricircle will be involved

Step 5: Create pod instances

Command issued:

 curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \
     -H "X-Auth-Token: $token" -d '{"pod": {"region_name":  "CentralRegion"}}'
 curl -X POST http://127.0.0.1:19999/v1.0/pods -H "Content-Type: application/json" \
     -H "X-Auth-Token: $token" -d '{"pod": {"region_name":  "RegionOne", "az_name": "az1"}}'


  • The restful API request will be routed to V1Controller from RootController for the version is 'v1.0':
 tricircle/tricircle/api/controllers/root.py
 ...
 class RootController(object):
      ...
     @expose()
     def _lookup(self, version, *remainder):
         if version == 'v1.0':
             return V1Controller(), remainder
  • In V1Controller, because the request is to create pod, so the request will be forwarded to the post function in PodsController. Post function will create the pod and az mapping record in pod table:
 tricircle/tricircle/api/controllers/pod.py
 ...
 class PodsController(rest.RestController):
   ...
   @expose(generic=True, template='json')
   def post(self, **kw):
       context = t_context.extract_context_from_environ()
       ...
       new_pod = core.create_resource(
                   context, models.Pod,
                   {'pod_id': _uuid,
                    'region_name': region_name,
                    'pod_az_name': pod_az_name,
                    'dc_name': dc_name,
                    'az_name': az_name})

Step 6: Create network/subnet in central Neutron server

The central_plugin.py will be called because the core plugin of central Neutron is configured like this:

   core_plugin = tricircle.network.central_plugin.TricirclePlugin

Command issued:

 neutron --os-region-name=CentralRegion net-create net1
 neutron --os-region-name=CentralRegion subnet-create net1 10.0.1.0/24
  • The request is handled by create_network when net1 is to be created:
 tricircle/tricircle/network/central_plugin.py
 ...
 class TricirclePlugin(db_base_plugin_v2.NeutronDbPluginV2,
                     security_groups.TricircleSecurityGroupMixin,
                     external_net_db.External_net_db_mixin,
                     portbindings_db.PortBindingMixin,
                     extradhcpopt_db.ExtraDhcpOptMixin,
                     l3_db.L3_NAT_dbonly_mixin,
                     l3_attrs_db.ExtraAttributesMixin):
  ...
  def create_network(self, context, network):
       net_data = network[attributes.NETWORK]
       ...
  • The request is handled by create_subnet when 10.0.1.0/24 subnet is created in net1:
 tricircle/tricircle/network/central_plugin.py
 ...
 class TricirclePlugin(db_base_plugin_v2.NeutronDbPluginV2,
                     security_groups.TricircleSecurityGroupMixin,
                     external_net_db.External_net_db_mixin,
                     portbindings_db.PortBindingMixin,
                     extradhcpopt_db.ExtraDhcpOptMixin,
                     l3_db.L3_NAT_dbonly_mixin,
                     l3_attrs_db.ExtraAttributesMixin):
 ...
 def create_subnet(self, context, subnet):
       subnet_data = subnet['subnet']
       ...

Step 7: Get image ID and flavor ID which will be used in VM booting

Command issued:

 glance --os-region-name=RegionOne image-list
 nova --os-region-name=RegionOne flavor-list

No source code of Tricircle will be involved in this step.

Step 8: Boot virtual machines

Command issued:

 nova --os-region-name=RegionOne boot --flavor 1 --image $image1_id --nic net-id=$net1_id vm1

Please note, Nova in RegionOne is configured to use local Neutron in RegionOne, and RegionTwo's Nova will use local Neutron in RegionTwo. So different local Neutron will be called when booting instances in different Region.

In RegionOne local Neutron's configuration, the core plugin is configured like this:

 core_plugin = tricircle.network.local_plugin.TricirclePlugin
  • When nova boot is executed in RegionOne Nova, Nova will send networks query with network id to local Neutron, and local plugin will send networks query request to central Neutron:
 tricircle/tricircle/network/local_plugin.py
 ...
 def get_networks(self, context, filters=None, fields=None,
                    sorts=None, limit=None, marker=None, page_reverse=False):
 ...
 t_networks = raw_client.list_networks(**params)['networks']
  • raw_client.list_networks is to query networks in central Neutron, if the network existed in central Neutron, local plugin will create network/subnet with same uuid as that in central Neutron:
 tricircle/tricircle/network/local_plugin.py
 ...
 def get_networks(self, context, filters=None, fields=None,
                    sorts=None, limit=None, marker=None, page_reverse=False):
       ...
               b_network = self.core_plugin.create_network(
                   context, {'network': network})
               subnet_ids = self._ensure_subnet(context, network)
  • later nova in RegionOne will issue a request to create a port in local Neutron in RegionOne, the creation process will create the port in central Neutron first, then create the port and security group in local Neutron with same uuid.
 tricircle/tricircle/network/local_plugin.py
 ...
 def create_port(self, context, port):
 ...
           self._adapt_port_body_for_client(port['port'])
           t_port = raw_client.create_port(port)['port']
  ....
   self._handle_security_group(t_ctx, context, t_port)
   b_port = self.core_plugin.create_port(context, {'port': t_port})
  • next step, nova in RegionOne will issue a request to update port binding in local Neutron in RegionOne, the binding process in local Neutron plugin will forward the port update to central Neutron, and then update local port too.
 tricircle/tricircle/network/local_plugin.py
 ...
 def update_port(self, context, _id, port):
 ...
           t_ctx = t_context.get_context_from_neutron_context(context)
           self.neutron_handle.handle_update(t_ctx, 'port', _id,
                                             {'port': update_dict})
       return self.core_plugin.update_port(context, _id, port)
  • Once central Neutron receives the port update request, need to create resource routing entries in routing table.
 tricircle/tricircle/network/central_plugin.py
 ...
 def update_port(self, context, port_id, port):
       t_ctx = t_context.get_context_from_neutron_context(context)
       top_port = super(TricirclePlugin, self).get_port(context, port_id)
       ...
          for resource_id, resource_type in entries:
               if db_api.get_bottom_id_by_top_id_region_name(
                       t_ctx, resource_id, pod['region_name'], resource_type):
                   continue
               db_api.create_resource_mapping(t_ctx, resource_id, resource_id,
                                              pod['pod_id'], res['tenant_id'],
                                              resource_type)
  • If the network has interface connected to a router, then start asynchronous job to (create if not exist in RegionOne) setup the router in RegionOne.
 tricircle/tricircle/network/central_plugin.py
 ...
 def update_port(self, context, port_id, port):
      ...
      interfaces = super(TricirclePlugin, self).get_ports(
               context,
               {'network_id': [res['network_id']],
                'device_owner': [constants.DEVICE_OWNER_ROUTER_INTF]})
           interfaces = [inf for inf in interfaces if inf['device_id']]
           if interfaces:
               # request may be come from service, we use an admin context
               # to run the xjob
               admin_context = t_context.get_admin_context()
               self.xjob_handler.setup_bottom_router(
                   admin_context, res['network_id'],
                   interfaces[0]['device_id'], pod['pod_id'])
  • If the network has interface connected to a router, xjob_handler.setup_bottom_router will register the asynchronous job and send one RPC message to xmanager.py to trigger the asynchronous job, which will setup the router in RegionOne.
  tricircle/tricircle/xjob/xmanager.py
  ...
   @_job_handle(constants.JT_ROUTER_SETUP)
   def setup_bottom_router(self, ctx, payload):
       (b_pod_id,
        t_router_id, t_net_id) = payload[constants.JT_ROUTER_SETUP].split('#')
        ...
  • Except the setup_bottom_router async. job, there are still other resources need to be check in update_port function in central_plugin.py, whether to create or update these resources in RegionOne, for example, security group.
 tricircle/tricircle/network/central_plugin.py
 ...
 def update_port(self, context, port_id, port):
      ...
      self.xjob_handler.configure_security_group_rules(t_ctx,
                                                            res['tenant_id'])

xjob_handler.configure_security_group_rules will register the security group rules asynchronous job, and then send RPC message to trigger the async.job in xmanager.py.

  • You can find the configure_security_group_rules job in xmanager.py.
    tricircle/tricircle/xjob/xmanager.py
  ...
  @_job_handle(constants.JT_SEG_RULE_SETUP)
   def configure_security_group_rules(self, ctx, payload):
       project_id = payload[constants.JT_SEG_RULE_SETUP]
       top_client = self._get_client()
       sg_filters = [{'key': 'tenant_id', 'comparator': 'eq',
                      'value': project_id}] 

The router setup and security group rules configuration (and other resources will be included later) are handled as async. jobs in XJOB process, the reason is to enhance the experience instances booting. If all this resources are created in local_plugin, then it'll take long time and may lead to the time out of booting, bad user experience and low reliability. At the same time, through the aync. job redo and reliability consideration design, the reliability of the networking automation process is ensured.

If you boot a VM2 in RegionTwo, for example

  nova --os-region-name=RegionTwo boot --flavor 1 --image $image2_id --nic net-id=$net2_id vm2

Then the local plugin under RegionTwo local Neutron will be called, the workflow is same as that in RegionOne. And the asynchronous jobs in central Neutron will setup resources like router (if the network has interface on the router) and security group rules, etc in RegionTwo. Please be aware of that if cross OpenStack L2 network is used in the VM2 booting, then resources setup to RegionOne and RegionTwo may be occurred in the asynchronous jobs. This is the networking automation to coordinate multiple OpenStack networking resources.

Step 9: Verify the VMs are connected to the networks

Tricircle is not involved in this step.

Step 10: Create external network and subnet

Command issued:

 curl -X POST http://127.0.0.1:20001/v2.0/networks -H "Content-Type: application/json" \
    -H "X-Auth-Token: $token" \
    -d '{"network": {"name": "ext-net", "admin_state_up": true, "router:external": true,  "provider:network_type": "vlan", "provider:physical_network": "extern", "availability_zone_hints": ["RegionTwo"]}}'
 neutron --os-region-name=CentralRegion subnet-create --name ext-subnet --disable-dhcp ext-net 163.3.124.0/24
  • For the external network, it'll be created by cloud admin, and need to specify in which region the external network will reside. create_network will check the network creation is for external network or not, if yes, forward the external network creation request to regarding region. Because RegionTwo is specified, so the request will be forwarded to RegionTwo local Neutron.
 tricircle/tricircle/network/central_plugin.py
 ...
 def create_network(self, context, network):
       net_data = network[attributes.NETWORK]
       tenant_id = net_data['tenant_id']
       is_external = self._ensure_az_set_for_external_network(context,
                                                              net_data)
       ...
            if is_external:
               self._fill_provider_info(res, net_data)
               self._create_bottom_external_network(
                   context, net_data, res['id'])

Step 11: Create router and attach subnets in central Neutron server

Command issued

 neutron --os-region-name=CentralRegion router-create router
 neutron --os-region-name=CentralRegion router-interface-add router $subnet1_id
 neutron --os-region-name=CentralRegion router-interface-add router $subnet2_id

Please note that in step 6, although there is code to setup router in update_port, but because the network has no interface on router in step 6, so the asynchronous job for router setup is not called. When router-interface-add is executed now, the code add_router_interface in central plugin will be called, and start asynchronous job to setup the router in RegionOne and/or RegionTwo, and bridge network will be created if needed:

tricircle/tricircle/network/central_plugin.py
 ...
 def add_router_interface(self, context, router_id, interface_info):
       t_ctx = t_context.get_context_from_neutron_context(context)
       router = self._get_router(context, router_id)
       ....
       if len(b_pods) == 1:
               self.xjob_handler.setup_bottom_router(
                   t_ctx, net_id, router_id, b_pods[0]['pod_id'])
           else:
               self.xjob_handler.setup_bottom_router(
                   t_ctx, net_id, router_id, t_constants.POD_NOT_SPECIFIED)

Step 12: Set router external gateway in central Neutron server

Command issued

  neutron --os-region-name=CentralRegion router-gateway-set router ext-net

The update_router in central_plugin will be called, and router gateway will be set in central Neutron first, then set router gateway in RegionTwo where the ext-net resides, extra route "destination, next hop" for L3 networking across Neutron will be set too through asynchronous job .

 tricircle/tricircle/network/central_plugin.py
 ...
 def update_router(self, context, router_id, router):
       router_data = copy.deepcopy(router['router'])
       ...
       if is_add:
           ret = super(TricirclePlugin, self).update_router(
               context, router_id, router)
           router_data[l3.EXTERNAL_GW_INFO].update(ret[l3.EXTERNAL_GW_INFO])
           self._add_router_gateway(context, router_id, router_data)
       ...
       t_ctx = t_context.get_context_from_neutron_context(context)
       self.xjob_handler.configure_extra_routes(t_ctx, router_id)

Step 13: Launch VNC console and test connection

No code of Tricircle will be called in this step.

Step 14: Create floating ip in central Neutron server

Command issued

 neutron --os-region-name=CentralRegion floatingip-create ext-net

The create_floatingip in central_plugin will be called, and just create a floating IP in central Neutron.

 tricircle/tricircle/network/central_plugin.py
 ...
 def create_floatingip(self, context, floatingip):
       # create bottom fip when associating fixed ip
       return super(TricirclePlugin, self).create_floatingip(
           context, floatingip,
           initial_status=constants.FLOATINGIP_STATUS_DOWN)
 ...

Step 15: Associate floating ip

Command issued

 neutron --os-region-name=CentralRegion floatingip-associate $floatingip_id $port_id

The update_floatingip in central_plugin will be called, and associate floating in central Neutron first, then call async. job to associate the floatingip in RegionTwo where the external network resides, shadow port may be created if the port associated to the floatingip does not co-located in same region.

 tricircle/tricircle/network/central_plugin.py
 ...
def update_floatingip(self, context, _id, floatingip):
...
       org_floatingip_dict = self._make_floatingip_dict(
           self._get_floatingip(context, _id))
       res = super(TricirclePlugin, self).update_floatingip(
           context, _id, floatingip)
       try:
           if floatingip['floatingip']['port_id']:
               self._associate_floatingip(context, _id, floatingip)
 ...
 def _associate_floatingip(self, context, _id, floatingip):
       t_ctx = t_context.get_context_from_neutron_context(context)
       .....
       self.xjob_handler.setup_bottom_router(
           t_ctx, net_id, floatingip_db['router_id'], int_net_pod['pod_id'])