Revision as of 14:25, 28 September 2016

The OpenStack Networking L3 DVR Sub-team holds public meetings as advertised on OpenStack IRC Meetings Calendar. If you are unable to attend, please check the most recent logs.

Meetings

• Weekly on Wednesday at 1500 UTC
• IRC channel: #openstack-meeting-alt on freenode
• Chair: Brian Haley (haleyb), Swaminathan Vasudevan (Swami)
• Meetings, with their notes and logs, will be found under http://eavesdrop.openstack.org/meetings/neutron_dvr

Agenda

Meeting September 28th, 2016

Announcements (haleyb)

• When adding items below I'd like to try to get feedback on whether they are a MUST FIX, SHOULD FIX, or GOOD TO HAVE and TRIAGE ME

Topics for Discussion

Bugs (Swami)

All DVR bugs should be tagged and listed here: https://bugs.launchpad.net/neutron/+bugs?field.tag=l3-dvr-backlog

New Bugs this week

High Priority Bugs in progress

• https://bugs.launchpad.net/neutron/+bug/1625333 (NEED-TRIAGE) - FloatingIP GARP fails. Log seen in the l3 agent and VM not able to ping
  • Need a reproducer, only on RHEL

• https://bugs.launchpad.net/neutron/+bug/1612192 (MUST FIX) - L3 DVR: Unable to complete operation on subnet (Related to HA)
  • Not sure this is critical any more, as it's mostly seen in the SFC and OVN jobs, ~0 in the dvr-multinode-full job

• https://bugs.launchpad.net/neutron/+bug/1612804 (MUST FIX) - test_shelve_instance fails with sshtimeout (One instance of failure seen in gate)
  • Lowered to High since not seen recently

• https://bugs.launchpad.net/neutron/+bug/1403455 (MUST FIX) - neutron netns cleanup does not cleanup all subprocesses

• https://bugs.launchpad.net/neutron/+bug/1593354 (MUST FIX) - SNAT HA failed due to missing NAT rule and sg- interface. (May be a duplicate)
  • Not reproducible with Newton

• https://bugs.launchpad.net/neutron/+bug/1606741 (SHOULD FIX) - Metadata error with dvr_snat on compute hosts.
  • https://review.openstack.org/352686 (Needs review) - No updated in two weeks

• https://bugs.launchpad.net/neutron/+bug/1596473 (MUST FIX) - Packet loss with DVR and IPv6 (Need to triage)
• https://bugs.launchpad.net/neutron/+bug/1506567 (SHOULD FIX) - No information from metering agent

• https://bugs.launchpad.net/neutron/+bug/1580648 (SHOULD FIX) - Two HA routers in master state during functional test
  • Patch already pushed in but the issue is still seen.

• https://bugs.launchpad.net/neutron/+bug/1571676 (Re-opened)
  • https://review.openstack.org/#/c/308068/ (Needs review)

Categorized Bugs

Gate Test Failures

More Functional test failures seen in Gate.

RFE

• https://bugs.launchpad.net/neutron/+bug/1563879 (RFE)
• https://bugs.launchpad.net/neutron/+bug/1557290 (RFE)

• https://bugs.launchpad.net/neutron/+bug/1577488 (RFE)
  • https://review.openstack.org/#/c/283757/ (NEEDS REVIEW)
  • https://review.openstack.org/#/c/355062/ (NEEDS REVIEW)

• https://bugs.launchpad.net/neutron/+bug/1583694 (RFE) (This RFE may not move forward in the newton time frame)
  • https://review.openstack.org/#/c/320669/ (WIP)(Server side patch)
  • https://review.openstack.org/#/c/323618/ (WIP)(Agent side patch)

Existing Functionality Broken Bugs

• https://bugs.launchpad.net/neutron/+bug/1583266 (SHOULD FIX) - watch_log_file=True causes fip to slow down. Symptom seen more with DVR routers.
• https://bugs.launchpad.net/neutron/+bug/1585165 (SHOULD FIX) - Floating ip not reachable after VM live migration (Seen in Liberty with DVR) - Seems to me like a duplicate but should triage it more on the symptoms.

• https://bugs.launchpad.net/neutron/+bug/1571676 (MUST FIX) - Static routes not added to FloatingIP Namespace.
  • https://review.openstack.org/#/c/308068/
  • This approach can't really work because adding a static route in the FIP namespace for one tenant could impact traffic for another tenant.

• https://bugs.launchpad.net/neutron/+bug/1564776 - (MUST FIX) (Snat Namespace error when namespace deleted)
  • Patch: https://review.openstack.org/#/c/300358/

• https://bugs.launchpad.net/neutron/+bug/1564757 - (MUST FIX) (FloatingIPError while agent restarts)

• https://bugs.launchpad.net/neutron/+bug/1526855 - (SHOULD FIX) (Metadata error)

• https://bugs.launchpad.net/neutron/+bug/1541406 - IPv6 Prefix Delegation does not work with DVR (SHOULD FIX)
  • https://review.openstack.org/#/c/277657/ (Patch) - Needs update to create a better abstraction that allows clients to get the namespace for the right part of the router.

• Live migration
  • https://review.openstack.org/#/c/286855/ (Tempest change) - Needs work - must merge before Nova patch
  • https://review.openstack.org/275420 (Merged) - Neutron Server side DVR change
  • https://review.openstack.org/#/c/260738/ (Merged) - L3 Agent side patch

• https://bugs.launchpad.net/neutron/+bug/1414559 (OVS drops RARP packets)
  • in neutron https://review.openstack.org/#/c/246898/ (MERGED) Notify vif plugged event
  • and nova https://review.openstack.org/#/c/246910/ (not making progress) Notify vif plugged event

• https://bugs.launchpad.net/neutron/+bug/1557290 - (MUST FIX) DVR FIP agent gateway does not pass traffic directed at fixed IP
  • https://review.openstack.org/#/c/297468/

• https://bugs.launchpad.net/neutron/+bug/1447227 (Low) - Connecting two or more distributed routers to a subnet. (No one is working on it) (GOOD TO HAVE)

• https://bugs.launchpad.net/neutron/+bug/1552070 (GOOD TO HAVE) - Functional test changes to address dual stack for DVR

Scale and Performance Impact Bugs

New Features Bugs

• https://bugs.launchpad.net/neutron/+bug/1557290 (SHOULD FIX) - DVR not able to forward traffic to private IP from FIP namespace
• https://bugs.launchpad.net/neutron/+bug/1504039 - LinuxBridge DVR (Spec & code)

Refactor or Cleanup Bugs

• https://bugs.launchpad.net/neutron/+bug/1531065 - (Low) duplicate fetch subnet_id in get_subnet_for_dvr (GOOD TO HAVE)
  • https://review.openstack.org/#/c/263563/ - (Needs review)
• https://bugs.launchpad.net/neutron/+bug/1538369 - Refactor request (Not a real bug)

WishList Bugs

• https://bugs.launchpad.net/neutron/+bug/1518819 - Ability to specify a gateway when adding a subnet without cidr (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1529439 - unify validate_agent_router_combination exceptions for dvr agent_mode (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1450067 (Low) - ML2 and L3 plugin exposes dvr extension even if ovs is unused. (No owners yet) (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1476469 (TRIAGE ME) - with DVR, a VM can't use floating IP and VPN at the same time
  • This is not a bug, since floating IP can't be used with the VPN

• https://bugs.launchpad.net/neutron/+bug/1620824 (TRIAGED) - Neutron DVR(SNAT) steals FIP traffic
  • https://review.openstack.org/#/c/366297/ (needs review) (Currently not using reference L2pop and using tcp_loose=0).

WatchList Bugs

• https://bugs.launchpad.net/neutron/+bug/1444014 (Incomplete) (StaleDataError) - on Watch
• https://bugs.launchpad.net/neutron/+bug/1521815 (Low)- (DVR functional Job failure) - on Watch state

Bugs Closed Recently

• https://bugs.launchpad.net/neutron/+bug/1619312 (FIXED) - dvr: can't migrate legacy router to DVR
  • https://review.openstack.org/370430

• https://bugs.launchpad.net/neutron/+bug/1602320 (FIXED) - Keepalive process kill vrrp child process with l3/dvr/ha
  • https://review.openstack.org/#/c/342730/

• https://bugs.launchpad.net/neutron/+bug/1602614 (MUST FIX) - DVR+L3 HA Loss during failover is higher

• https://bugs.launchpad.net/neutron/+bug/1595043 (SHOULD FIX) - Generic DVR portbinding useful for HA ports.
  • https://review.openstack.org/#/c/255237 (MERGED)

• https://bugs.launchpad.net/neutron/+bug/1456073 (High) - Block Migration with FIP breaks with DVR (MUST FIX)
  • https://review.openstack.org/275073 (Nova side change) - Needs review

• https://bugs.launchpad.net/neutron/+bug/1597461 (FIXED) - Two masters after reboot of controller when HA enabled. Also seen with DVR
  • https://review.openstack.org/357458 (merged)

• https://bugs.launchpad.net/neutron/+bug/1607381 (FIXED) - HA router in l3 dvr_snat/legacy agent has no ha_port
  • https://review.openstack.org/365653 (merged)

• https://bugs.launchpad.net/neutron/+bug/1462154 (With DVR Pings to floating IPs replied with fixed-ips if VMs are on the same network)
  • https://review.openstack.org/246855 (Merged) (Reverted)

Performance/Scalability

Gate failures (haleyb)

• Neutron Failure Rate dashboard - http://grafana.openstack.org/dashboard/db/neutron-failure-rate
  • DVR multinode failure rates have lowered - very infrequent in gate queue, 15% in check queue
    • VM failing to get DHCP address typical problem
  • Bugs in Nova:
    • https://bugs.launchpad.net/nova/+bug/1524898 (Volume based live migration aborted unexpectedly)
    • https://bugs.launchpad.net/nova/+bug/1535232 (live-migration ci failure on nfs shared storage)

Stable backports (haleyb)

Ihar created a page tracking all the potential backports from Mitaka to the stable releases. I have been going through it with the help of Swami to get stable/liberty, well, more stable. Bugs are removed from list as they merge.

• https://etherpad.openstack.org/p/stable-bug-candidates-from-master

We need to continue to be aggressive at proactively backporting fixes to the stable branches

• http://docs.openstack.org/project-team-guide/stable-branches.html#proactive-backports

This is a list of bugs with a fix that has been committed to the master branch, and are tagged with 'neutron-proactive-backport-potential+l3-dvr-backlog' (the wiki thinks the link is spam so I had to put a space in it)

• https://goo. gl/sx0KL5

Open Discussion

Meeting commands

/join #openstack-meeting-alt
#startmeeting neutron_dvr
#chair Swami
#topic Announcements
#undo topic
#link https://wiki.openstack.org/wiki/Meetings/Neutron-DVR
#action haleyb will get something specific done this week
...
#endmeeting