Revision as of 15:00, 21 September 2016

The OpenStack Networking L3 DVR Sub-team holds public meetings as advertised on OpenStack IRC Meetings Calendar. If you are unable to attend, please check the most recent logs.

Meetings

• Weekly on Wednesday at 1500 UTC
• IRC channel: #openstack-meeting-alt on freenode
• Chair: Brian Haley (haleyb), Swaminathan Vasudevan (Swami)
• Meetings, with their notes and logs, will be found under http://eavesdrop.openstack.org/meetings/neutron_dvr

Agenda

Meeting September 21st, 2016

Announcements (haleyb)

• When adding items below I'd like to try to get feedback on whether they are a MUST FIX, SHOULD FIX, or GOOD TO HAVE and TRIAGE ME

Topics for Discussion

Bugs (Swami)

All DVR bugs should be tagged and listed here: https://bugs.launchpad.net/neutron/+bugs?field.tag=l3-dvr-backlog

New Bugs this week

• https://bugs.launchpad.net/neutron/+bug/1625333 (NEED-TRIAGE) - FloatingIP GARP fails. Log seen in the l3 agent and VM not able to ping
  • Need a reproducer

High Priority Bugs in progress

• https://bugs.launchpad.net/neutron/+bug/1612192 (MUST FIX) - L3 DVR: Unable to complete operation on subnet (Related to HA)
  • Not sure this is critical any more, as it's mostly seen in the SFC and OVN jobs, ~0 in the dvr-multinode-full job

• https://bugs.launchpad.net/neutron/+bug/1612804 (MUST FIX) - test_shelve_instance fails with sshtimeout (One instance of failure seen in gate)
  • Lowered to High since not seen recently

• https://bugs.launchpad.net/neutron/+bug/1403455 (MUST FIX) - neutron netns cleanup does not cleanup all subprocesses

• https://bugs.launchpad.net/neutron/+bug/1593354 (MUST FIX) - SNAT HA failed due to missing NAT rule and sg- interface. (May be a duplicate)
  • Not reproducible with Newton

• https://bugs.launchpad.net/neutron/+bug/1606741 (SHOULD FIX) - Metadata error with dvr_snat on compute hosts.
  • https://review.openstack.org/352686 (Needs review) - No updated in two weeks

• https://bugs.launchpad.net/neutron/+bug/1596473 (MUST FIX) - Packet loss with DVR and IPv6 (Need to triage)
• https://bugs.launchpad.net/neutron/+bug/1506567 (SHOULD FIX) - No information from metering agent

• https://bugs.launchpad.net/neutron/+bug/1580648 (SHOULD FIX) - Two HA routers in master state during functional test
  • Patch already pushed in but the issue is still seen.

• https://bugs.launchpad.net/neutron/+bug/1571676 (Re-opened)
  • https://review.openstack.org/#/c/308068/ (Needs review)

Categorized Bugs

Gate Test Failures

More Functional test failures seen in Gate.

RFE

• https://bugs.launchpad.net/neutron/+bug/1563879 (RFE)
• https://bugs.launchpad.net/neutron/+bug/1557290 (RFE)

• https://bugs.launchpad.net/neutron/+bug/1577488 (RFE)
  • https://review.openstack.org/#/c/283757/ (NEEDS REVIEW)
  • https://review.openstack.org/#/c/355062/ (NEEDS REVIEW)

• https://bugs.launchpad.net/neutron/+bug/1583694 (RFE) (This RFE may not move forward in the newton time frame, but we might have further discussion during the mid-cycle meetup)
  • https://review.openstack.org/#/c/320669/ (WIP)(Server side patch)
  • https://review.openstack.org/#/c/323618/ (WIP)(Agent side patch)

Existing Functionality Broken Bugs

• https://bugs.launchpad.net/neutron/+bug/1583266 (SHOULD FIX) - watch_log_file=True causes fip to slow down. Symptom seen more with DVR routers.
• https://bugs.launchpad.net/neutron/+bug/1585165 (SHOULD FIX) - Floating ip not reachable after VM live migration (Seen in Liberty with DVR) - Seems to me like a duplicate but should triage it more on the symptoms.

• https://bugs.launchpad.net/neutron/+bug/1571676 (MUST FIX) - Static routes not added to FloatingIP Namespace.
  • https://review.openstack.org/#/c/308068/
  • This approach can't really work because adding a static route in the FIP namespace for one tenant could impact traffic for another tenant.

• https://bugs.launchpad.net/neutron/+bug/1564776 - (MUST FIX) (Snat Namespace error when namespace deleted)
  • Patch: https://review.openstack.org/#/c/300358/

• https://bugs.launchpad.net/neutron/+bug/1564757 - (MUST FIX) (FloatingIPError while agent restarts)

• https://bugs.launchpad.net/neutron/+bug/1526855 - (SHOULD FIX) (Metadata error)

• https://bugs.launchpad.net/neutron/+bug/1541406 - IPv6 Prefix Delegation does not work with DVR (SHOULD FIX)
  • https://review.openstack.org/#/c/277657/ (Patch) - Needs update to create a better abstraction that allows clients to get the namespace for the right part of the router.

• Live migration
  • https://review.openstack.org/#/c/286855/ (Tempest change) - Needs work - must merge before Nova patch
  • https://review.openstack.org/275420 (Merged) - Neutron Server side DVR change
  • https://review.openstack.org/#/c/260738/ (Merged) - L3 Agent side patch

• https://bugs.launchpad.net/neutron/+bug/1414559 (OVS drops RARP packets)
  • in neutron https://review.openstack.org/#/c/246898/ (Needs review) Notify vif plugged event (related)
  • and nova https://review.openstack.org/#/c/246910/ (Needs review) Notify vif plugged event (related)

• https://bugs.launchpad.net/neutron/+bug/1557290 - (MUST FIX) DVR FIP agent gateway does not pass traffic directed at fixed IP
  • https://review.openstack.org/#/c/297468/

• https://bugs.launchpad.net/neutron/+bug/1447227 (Low) - Connecting two or more distributed routers to a subnet. (No one is working on it) (GOOD TO HAVE)

• https://bugs.launchpad.net/neutron/+bug/1552070 (GOOD TO HAVE) - Functional test changes to address dual stack for DVR

Scale and Performance Impact Bugs

New Features Bugs

• https://bugs.launchpad.net/neutron/+bug/1557290 (SHOULD FIX) - DVR not able to forward traffic to private IP from FIP namespace
• https://bugs.launchpad.net/neutron/+bug/1504039 - LinuxBridge DVR (Spec & code)

Refactor or Cleanup Bugs

• https://bugs.launchpad.net/neutron/+bug/1531065 - (Low) duplicate fetch subnet_id in get_subnet_for_dvr (GOOD TO HAVE)
  • https://review.openstack.org/#/c/263563/ - (Needs review)
• https://bugs.launchpad.net/neutron/+bug/1538369 - Refactor request (Not a real bug)

WishList Bugs

• https://bugs.launchpad.net/neutron/+bug/1518819 - Ability to specify a gateway when adding a subnet without cidr (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1529439 - unify validate_agent_router_combination exceptions for dvr agent_mode (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1450067 (Low) - ML2 and L3 plugin exposes dvr extension even if ovs is unused. (No owners yet) (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1476469 (TRIAGE ME) - with DVR, a VM can't use floating IP and VPN at the same time
  • This is not a bug, since floating IP can't be used with the VPN

• https://bugs.launchpad.net/neutron/+bug/1620824 (TRIAGED) - Neutron DVR(SNAT) steals FIP traffic
  • https://review.openstack.org/#/c/366297/ (needs review) (Currently not using reference L2pop and using tcp_loose=0).

WatchList Bugs

• https://bugs.launchpad.net/neutron/+bug/1444014 (Incomplete) (StaleDataError) - on Watch
• https://bugs.launchpad.net/neutron/+bug/1521815 (Low)- (DVR functional Job failure) - on Watch state

Bugs Closed Recently

• https://bugs.launchpad.net/neutron/+bug/1619312 (FIXED) - dvr: can't migrate legacy router to DVR
  • Need to turn-off guard in transaction
  • https://review.openstack.org/370430

• https://bugs.launchpad.net/neutron/+bug/1602320 (FIXED) - Keepalive process kill vrrp child process with l3/dvr/ha
  • https://review.openstack.org/#/c/342730/

• https://bugs.launchpad.net/neutron/+bug/1602614 (MUST FIX) - DVR+L3 HA Loss during failover is higher

• https://bugs.launchpad.net/neutron/+bug/1595043 (SHOULD FIX) - Generic DVR portbinding useful for HA ports.
  • https://review.openstack.org/#/c/255237 (MERGED)

• https://bugs.launchpad.net/neutron/+bug/1590041 (SHOULD-FIX) - Regression, snat_namespace object creation ahead of time does not solve some external_gateway_update condition.
  • https://review.openstack.org/#/c/326729/2

• https://bugs.launchpad.net/neutron/+bug/1456073 (High) - Block Migration with FIP breaks with DVR (MUST FIX)
  • https://review.openstack.org/275073 (Nova side change) - Needs review

• https://bugs.launchpad.net/neutron/+bug/1597461 (FIXED) - Two masters after reboot of controller when HA enabled. Also seen with DVR
  • https://review.openstack.org/357458 (merged)
• https://bugs.launchpad.net/neutron/+bug/1607381 (FIXED) - HA router in l3 dvr_snat/legacy agent has no ha_port
  • https://review.openstack.org/365653 (merged)

• https://bugs.launchpad.net/neutron/+bug/1599287 (MUST FIX) - Cleanup of ip rule and tables for stale snat
  • https://review.openstack.org/337855 (merged)
• https://bugs.launchpad.net/neutron/+bug/1609540 (MUST FIX) - CSNAT port fails due to no fixed ips. (partial workaround proposed).
  • https://review.openstack.org/350783 (Merged)

• https://bugs.launchpad.net/neutron/+bug/1602794 (MUST FIX) - Itemallocator class can throw a ValueError
  • https://review.openstack.org/341794
• https://bugs.launchpad.net/neutron/+bug/1599089 (MUST FIX) - Floatingip move within the same node does not handle fixed ip properly
  • https://review.openstack.org/#/c/337591/
• https://bugs.launchpad.net/neutron/+bug/1597561 (MUST FIX) - Occurences of Duplicate fg port in fip namespace
  • https://review.openstack.org/#/c/335755/
• https://bugs.launchpad.net/neutron/+bug/1569918 (MUST FIX) (Allowed_address_pair delayed FIP association)
  • https://review.openstack.org/#/c/304905/

• https://bugs.launchpad.net/neutron/+bug/1462154 (With DVR Pings to floating IPs replied with fixed-ips if VMs are on the same network)
  • https://review.openstack.org/246855 (Merged) (Reverted)

Performance/Scalability

Gate failures (haleyb)

• Neutron Failure Rate dashboard - http://grafana.openstack.org/dashboard/db/neutron-failure-rate
  • A number of patches have merged in both Neutron and Infra repositories that fix issues, including:
    • DHCP (multiple)
    • Underlay network possibly using same VNI as overlay, causing packets to go out wrong interfaces
  • DVR multinode failure rates have lowered - 5% in gate queue, 15% in check queue
    • VM failing to get DHCP address typical problem, need to re-confirm that's still the case
  • Bugs in Nova:
    • https://bugs.launchpad.net/nova/+bug/1524898 (Volume based live migration aborted unexpectedly)
    • https://bugs.launchpad.net/nova/+bug/1535232 (live-migration ci failure on nfs shared storage)

Stable backports (haleyb)

Ihar created a page tracking all the potential backports from Mitaka to the stable releases. I have been going through it with the help of Swami to get stable/liberty, well, more stable. Bugs are removed from list as they merge.

• https://etherpad.openstack.org/p/stable-bug-candidates-from-master

We need to continue to be aggressive at proactively backporting fixes to the stable branches

• http://docs.openstack.org/project-team-guide/stable-branches.html#proactive-backports

This is a list of bugs with a fix that has been committed to the master branch, and are tagged with 'neutron-proactive-backport-potential+l3-dvr-backlog' (the wiki thinks the link is spam so I had to put a space in it)

• https://goo. gl/sx0KL5

Open Discussion

Need to escalate the nova patch for the live migration. Since we have the tempest test running right now, we need someone from the nova team to take a look at the nova live migration patch.

• https://review.openstack.org/#/c/275073/

Meeting commands

/join #openstack-meeting-alt
#startmeeting neutron_dvr
#chair Swami
#topic Announcements
#undo topic
#link https://wiki.openstack.org/wiki/Meetings/Neutron-DVR
#action haleyb will get something specific done this week
...
#endmeeting