Revision as of 15:00, 10 August 2016

The OpenStack Networking L3 DVR Sub-team holds public meetings as advertised on OpenStack IRC Meetings Calendar. If you are unable to attend, please check the most recent logs.

Meetings

• Weekly on Wednesday at 1500 UTC
• IRC channel: #openstack-meeting-alt on freenode
• Chair: Brian Haley (haleyb), Swaminathan Vasudevan (Swami)
• Meetings, with their notes and logs, will be found under http://eavesdrop.openstack.org/meetings/neutron_dvr

Agenda

Meeting August 10th, 2016

Announcements (haleyb)

• When adding items below I'd like to try to get feedback on whether they are a MUST FIX, SHOULD FIX, or GOOD TO HAVE

Topics for Discussion

Bugs (Swami)

All DVR bugs should be tagged and listed here: https://bugs.launchpad.net/neutron/+bugs?field.tag=l3-dvr-backlog

New Bugs this week

• https://bugs.launchpad.net/neutron/+bug/1609540 (MUST FIX) - CSNAT port fails due to no fixed ips. (partial workaround proposed).
  • https://review.openstack.org/350783
• https://bugs.launchpad.net/neutron/+bug/1597461 (MUST FIX) - Two masters after reboot of controller when HA enabled. Also seen with DVR
• https://bugs.launchpad.net/neutron/+bug/1602320 (SHOULD FIX) - Keepalive process kill vrrp child process with l3/dvr/ha
• https://bugs.launchpad.net/neutron/+bug/1606741 (SHOULD FIX) - Metadata error with dvr_snat on compute hosts.
  • https://review.openstack.org/352686 (Needs review)
• https://bugs.launchpad.net/neutron/+bug/1595043 (SHOULD FIX) - Generic DVR portbinding useful for HA ports.
  • https://review.openstack.org/324302 (Need review)
• https://bugs.launchpad.net/neutron/+bug/1602614 (MUST FIX) - DVR+L3 HA Loss during failover is higher (Need to triage)
• https://bugs.launchpad.net/neutron/+bug/1593354 (MUST FIX) - SNAT HA failed due to missing NAT rule and sg- interface. (May be a duplicate)
• https://bugs.launchpad.net/neutron/+bug/1596473 (MUST FIX) - Packet loss with DVR and IPv6 (Need to triage)
• https://bugs.launchpad.net/neutron/+bug/1506567 (SHOULD FIX) - No information from metering agent
• https://bugs.launchpad.net/neutron/+bug/1599287 (MUST FIX) - Cleanup of ip rule and tables for stale snat
  • https://review.openstack.org/337855

Categorized Bugs

Gate Test Failures

More Functional test failures seen in Gate.

RFE

• https://bugs.launchpad.net/neutron/+bug/1563879 (RFE)
• https://bugs.launchpad.net/neutron/+bug/1557290 (RFE)
• https://bugs.launchpad.net/neutron/+bug/1577488 (RFE)
  • https://review.openstack.org/#/c/283757/
  • https://review.openstack.org/#/c/297468/ (WIP)
• https://bugs.launchpad.net/neutron/+bug/1583694 (RFE) (This RFE may not move forward in the newton time frame, but we might have further discussion during the mid-cycle meetup)
  • https://review.openstack.org/#/c/320669/ (WIP)(Server side patch)
  • https://review.openstack.org/#/c/323618/ (WIP)(Agent side patch)

Existing Functionality Broken Bugs

• https://bugs.launchpad.net/neutron/+bug/1590041 (SHOULD-FIX) - Regression, snat_namespace object creation ahead of time does not solve some external_gateway_update condition.
  • https://review.openstack.org/#/c/326729/2

• https://bugs.launchpad.net/neutron/+bug/1583266 (SHOULD FIX) - watch_log_file=True causes fip to slow down. Symptom seen more with DVR routers.
• https://bugs.launchpad.net/neutron/+bug/1585165 (SHOULD FIX) - Floating ip not reachable after VM live migration (Seen in Liberty with DVR) - Seems to me like a duplicate but should triage it more on the symptoms.

• https://bugs.launchpad.net/neutron/+bug/1571676 (MUST FIX) - Static routes not added to FloatingIP Namespace.
  • https://review.openstack.org/#/c/308068/
  • This approach can't really work because adding a static route in the FIP namespace for one tenant could impact traffic for another tenant.

• https://bugs.launchpad.net/neutron/+bug/1564776 - (MUST FIX) (Snat Namespace error when namespace deleted)
  • Patch: https://review.openstack.org/#/c/300358/

• https://bugs.launchpad.net/neutron/+bug/1564757 - (MUST FIX) (FloatingIPError while agent restarts)

• https://bugs.launchpad.net/neutron/+bug/1526855 - (SHOULD FIX) (Metadata error)

• https://bugs.launchpad.net/neutron/+bug/1541406 - IPv6 Prefix Delegation does not work with DVR (SHOULD FIX)
  • https://review.openstack.org/#/c/277657/ (Patch) - Needs update to create a better abstraction that allows clients to get the namespace for the right part of the router.

• https://bugs.launchpad.net/neutron/+bug/1456073 (High) - Block Migration with FIP breaks with DVR (MUST FIX)
  • https://review.openstack.org/275073 (Nova side change) - Needs review
  • https://review.openstack.org/#/c/286855/ (Tempest change) - Needs work - must merge before Nova patch
  • https://review.openstack.org/275420 (Merged) - Neutron Server side DVR change
  • https://review.openstack.org/#/c/260738/ (Merged) - L3 Agent side patch

• https://bugs.launchpad.net/neutron/+bug/1414559 (OVS drops RARP packets)
  • in neutron https://review.openstack.org/#/c/246898/ (Needs review) Notify vif plugged event (related)
  • and nova https://review.openstack.org/#/c/246910/ (Needs review) Notify vif plugged event (related)

• https://bugs.launchpad.net/neutron/+bug/1557290 - (MUST FIX) DVR FIP agent gateway does not pass traffic directed at fixed IP
  • https://review.openstack.org/#/c/297468/

• https://bugs.launchpad.net/neutron/+bug/1447227 (Low) - Connecting two or more distributed routers to a subnet. (No one is working on it) (GOOD TO HAVE)

• https://bugs.launchpad.net/neutron/+bug/1552070 (GOOD TO HAVE) - Functional test changes to address dual stack for DVR

Scale and Performance Impact Bugs

New Features Bugs

• https://bugs.launchpad.net/neutron/+bug/1557290 (SHOULD FIX) - DVR not able to forward traffic to private IP from FIP namespace
• https://bugs.launchpad.net/neutron/+bug/1504039 - LinuxBridge DVR (Spec & code)

Refactor or Cleanup Bugs

• https://bugs.launchpad.net/neutron/+bug/1531065 - (Low) duplicate fetch subnet_id in get_subnet_for_dvr (GOOD TO HAVE)
  • https://review.openstack.org/#/c/263563/ - (Needs review)
• https://bugs.launchpad.net/neutron/+bug/1538369 - Refactor request (Not a real bug)

WishList Bugs

• https://bugs.launchpad.net/neutron/+bug/1518819 - Ability to specify a gateway when adding a subnet without cidr (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1529439 - unify validate_agent_router_combination exceptions for dvr agent_mode (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1450067 (Low) - ML2 and L3 plugin exposes dvr extension even if ovs is unused. (No owners yet) (GOOD TO HAVE)

WatchList Bugs

• https://bugs.launchpad.net/neutron/+bug/1444014 (Incomplete) (StaleDataError) - on Watch
• https://bugs.launchpad.net/neutron/+bug/1521815 (Low)- (DVR functional Job failure) - on Watch state

Bugs Closed Recently

• https://bugs.launchpad.net/neutron/+bug/1602794 (MUST FIX) - Itemallocator class can throw a ValueError
  • https://review.openstack.org/341794
• https://bugs.launchpad.net/neutron/+bug/1599089 (MUST FIX) - Floatingip move within the same node does not handle fixed ip properly
  • https://review.openstack.org/#/c/337591/
• https://bugs.launchpad.net/neutron/+bug/1597561 (MUST FIX) - Occurences of Duplicate fg port in fip namespace
  • https://review.openstack.org/#/c/335755/
• https://bugs.launchpad.net/neutron/+bug/1569918 (MUST FIX) (Allowed_address_pair delayed FIP association)
  • https://review.openstack.org/#/c/304905/

• https://bugs.launchpad.net/neutron/+bug/1462154 (With DVR Pings to floating IPs replied with fixed-ips if VMs are on the same network)
  • https://review.openstack.org/246855 (Merged) (Reverted)

Performance/Scalability

Gate failures (haleyb)

• Current failure rate: https://goo.gl/L1WODG
  • Upstream multinode job failures seen, root cause identified.
  • Check job failures were mostly due to the ipv6 dual net test case failure, cause identified and workaround proposed.
  • Functional test failures seen in the Check queue, root cause yet to be identified.
  • Single node check job failure rates are more when compared to neutron full job.
  • Multi-node check job failure rate have gone up from last week
    • Bugs in Nova:
      • https://bugs.launchpad.net/nova/+bug/1524898 (Volume based live migration aborted unexpectedly)
      • https://bugs.launchpad.net/nova/+bug/1535232 (live-migration ci failure on nfs shared storage)
• Old bug to help in debugging the gate
  • https://bugs.launchpad.net/neutron/+bug/1515360 - Tempest "SSHTimeout" failures (only for debugging)

Stable backports (haleyb)

Ihar created a page tracking all the potential backports from Mitaka to the stable releases. I have been going through it with the help of Swami to get stable/liberty, well, more stable. Bugs are removed from list as they merge.

• https://etherpad.openstack.org/p/stable-bug-candidates-from-master

We need to continue to be aggressive at proactively backporting fixes to the stable branches

• http://docs.openstack.org/project-team-guide/stable-branches.html#proactive-backports

This is a list of bugs with a fix that has been committed to the master branch, and are tagged with 'neutron-proactive-backport-potential+l3-dvr-backlog' (the wiki thinks the link is spam so I had to put a space in it)

• https://goo. gl/sx0KL5

Open Discussion

Wanted to talk about some related patches, so we come up with a good answer going forward These two both do similar things around ip_lib.get_devices() code:

• https://review.openstack.org/#/c/348372 (L3 agent: check router namespace existence before delete)
• https://review.openstack.org/#/c/309050/ (Check if namespace exists before getting devices)

And this is a change that also checks existence so not throw exceptions:

• https://review.openstack.org/#/c/326729/ (DVR: Clean stale snat-ns by checking its existence when agent restarts)

Need to escalate the nova patch for the live migration. Since we have the tempest test running right now, we need someone from the nova team to take a look at the nova live migration patch.

• https://review.openstack.org/#/c/275073/

Meeting commands

/join #openstack-meeting-alt
#startmeeting neutron_dvr
#topic Announcements
#undo topic
#link https://wiki.openstack.org/wiki/Meetings/Neutron-DVR
#action haleyb will get something specific done this week
#chair Swami
...
#endmeeting