Revision as of 14:48, 3 February 2016

Meetings

• Weekly on Wednesday at 1500 UTC
• IRC channel: #openstack-meeting-alt on freenode
• Chair: Brian Haley (haleyb), Swaminathan Vasudevan (Swami)
• Meetings, with their notes and logs, will be found under http://eavesdrop.openstack.org/meetings/neutron_dvr

Agenda

Meeting February 3rd, 2016

Announcements (haleyb)

• We still have a lot of bugs (below)
  • When going through them I'd like to try to get feedback on whether they are a MUST FIX, SHOULD FIX, or GOOD TO HAVE
  • If it needs a rebase, do it soon after Jenkins notifies you
  • If you just need reviews, ping people on irc or post a comment in the review

Topics for Discussion

Live Migration issues with DVR (Swami)

• We have started a discussion on the live migration issue with DVR and how to address it from the neutron side. As a team we need to get consensus on this ideas before we work with the Nova team on this issue.
  • https://bugs.launchpad.net/neutron/+bug/1456073 (discussion thread)
  • Any updates - Are we all in sync with nova-neutron notifier requirement and also with the changes proposed for the neutron side to address the live migration.
  • Live migration was discussed with the Nova team during the mid-cycle meet-up. Based on the discussion we have upload a couple of patches for early review ( Work in progress patches)
  • https://review.openstack.org/275073 (Nova change)
  • https://review.openstack.org/275420 (Neutron Server side DVR change)
  • https://review.openstack.org/#/c/260738/ ( Neutron Agent side DVR change)
  • NOTE: Based on the current internal tests, these patches seem to fix the Live migration and floatingip issues.

Bugs (Swami)

All DVR bugs should be tagged and listed here: https://bugs.launchpad.net/neutron/+bugs?field.tag=l3-dvr-backlog

New Bugs this week

• https://bugs.launchpad.net/neutron/+bug/1541406 - IPv6 Prefix Delegation does not work with DVR
• https://bugs.launchpad.net/neutron/+bug/1540259 - Use list should be True DVR portbinding
• https://bugs.launchpad.net/neutron/+bug/1538369 - Refactor request ( Not a real bug)

Categorized Bugs

Gate Test Failures

• https://bugs.launchpad.net/neutron/+bug/1522824 (High) - Test_shelve_instance test failure. (MUST FIX)
  • https://review.openstack.org/#/c/253569/ - Set PENDING_BUILD status for ports after update in DB - Abandoned in favor of https://review.openstack.org/#/c/215467
• https://bugs.launchpad.net/neutron/+bug/1450604 (Medium) - Fix or stabilize Multi-node Job

Existing Functionality Broken Bugs

• https://bugs.launchpad.net/neutron/+bug/1456073 (High) - Block Migration with FIP breaks with DVR (MUST FIX)
  • related bug https://bugs.launchpad.net/neutron/+bug/1414559 with fixes:
    • in neutron https://review.openstack.org/#/c/246898/ (Needs review)
    • and nova https://review.openstack.org/#/c/246910/ (Needs review)
    • https://review.openstack.org/#/c/259299/ (WIP)
    • https://review.openstack.org/#/c/259171/ (WIP)
    • https://review.openstack.org/#/c/260738/ (WIP)

• https://bugs.launchpad.net/neutron/+bug/1462154 (High) - DVR pings to floatingip replied with fixed ips (SHOULD FIX)
  • https://review.openstack.org/246855
  • Doesn't seem to be making progress?
• https://bugs.launchpad.net/neutron/+bug/1445255 (Low) - DVR Floatingip to unbound port does not work (Allowed address pair) (SHOULD FIX)
  • https://review.openstack.org/#/c/254439/ (Needs review)
• https://bugs.launchpad.net/neutron/+bug/1447227 (Low) - Connecting two or more distributed routers to a subnet. (No one is working on it) (GOOD TO HAVE)

Scale and Performance Impact Bugs

New Features Bugs

• https://bugs.launchpad.net/neutron/+bug/1365473 (High) - (DVR SNAT HA - Server Side) (Should change this to HIGH) Right now it is WishList
  • https://review.openstack.org/#/c/143169/ (Needs core blessings) (MUST FIX)

Refactor or Cleanup Bugs

• https://bugs.launchpad.net/neutron/+bug/1531065 - (Low) duplicate fetch subnet_id in get_subnet_for_dvr (GOOD TO HAVE)
  • https://review.openstack.org/#/c/263563/ - (Needs review)

WishList Bugs

• https://bugs.launchpad.net/neutron/+bug/1504039 (Linux bridge support for DVR) (SHOULD FIX or GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1518819 - Ability to specify a gateway when adding a subnet without cidr (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1529439 - unify validate_agent_router_combination exceptions for dvr agent_mode (GOOD TO HAVE)
• https://bugs.launchpad.net/neutron/+bug/1450067 (Low) - ML2 and L3 plugin exposes dvr extension even if ovs is unused. (No owners yet) (GOOD TO HAVE)

WatchList Bugs

• https://bugs.launchpad.net/neutron/+bug/1444014 (Incomplete) (StaleDataError) - on Watch
• https://bugs.launchpad.net/neutron/+bug/1521815 (Low)- (DVR functional Job failure) - on Watch state

Bugs Closed

• https://bugs.launchpad.net/neutron/+bug/1526175 (Medium) - L3 HA routers end up in DVR Node (medium) (Abandoned)
• https://bugs.launchpad.net/neutron/+bug/1510796 (Low) - Function sync_routers always calls get_dvr_sync_data in ha scenario Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1538163 (UNDECIDED) - Race condition in removing a router on dvr service port delete - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1524908 (Undecided - May be a Low) - Router may be removed from SNAT agent by accident - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1536110 (Undecided) - OVS agent should fail if can't get DVR mac address - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1535928 (Undecided) - Duplicate IPtables rule detected - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1508869 (Low) - handle port host change (live migration) - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1524020 (Medium) - DVR ARP update call count can be reduced. - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1524291 (Low) - Code duplication - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1512199 - (Medium) This bug has been confirmed and re-opened this week
  • Seen in Master branch as well - probably introduced due to regression. - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1499785 (Low) - Static routes are not added to the router namespace - Merged (Can be closed)
• https://bugs.launchpad.net/neutron/+bug/1499787 (Low) - Static routes are added to snat namespace without checking for gateway. - Merged (Can be closed)
  • https://review.openstack.org/#/c/228026/ (reviewed - final comments addressed)
• https://bugs.launchpad.net/neutron/+bug/1424096 (Undecided) - Delete port with admin context. - Merged (Can be closed)
  • https://review.openstack.org/#/c/257938/ - (Patch in merge conflict - reviewed)
• https://bugs.launchpad.net/neutron/+bug/1530179 -(Medium) get_subnet_for_dvr() returns wrong gateway mac - Merged (Can be closed)
  • https://review.openstack.org/#/c/262574/ - (Needs review)
• https://bugs.launchpad.net/neutron/+bug/1505575 Fatal memory consumption by neutron-server with DVR at scale - Merged (Can be closed)
  • https://review.openstack.org/#/c/234067/ (sync in chunks)

Discuss a plan to address the gate failures (haleyb)

• Current failure rate: https://goo.gl/L1WODG
  • Single node check job failure rate has recovered from last week issues, same as non-dvr job as of today.
  • Multi-node non-DVR job failures are still over 50% (migrate volume error)
  • Multi-node DVR job over 70%
    • Filed https://bugs.launchpad.net/bugs/1538387 yesterday (OVS l2pop exception)
      • https://review.openstack.org/272986 out for review
    • Next noticeable issue: "DVR: Unable to retrieve subnet information for subnet_id"
      • Will file bug - looks like get_subnet_for_dvr() call in _bind_centralized_snat_port_on_dvr_subnet() needs to check if subnet_info is not {}
• Debug the gate in the current failure state (Identify owners who can own this)
• https://bugs.launchpad.net/neutron/+bug/1515360 - Tempest "SSHTimeout" failures (GOOD TO HAVE) - This patch seem to have some timing issues and so we are seeing more failures for the pings.
  • https://review.openstack.org/#/c/247748/ (add debugging code to l3-agent) - Still pings are not consistent with the tempest.
• https://drive.google.com/file/d/0B4kh-7VVPWlPMkdrQWFFdjNsSnM/view?usp=sharing - Logstash failures captured. Any logstash errors can be captured here

Performance/Scalability (obondarev)

• https://blueprints.launchpad.net/neutron/+spec/improve-dvr-l3-agent-binding (SHOULD HAVE)
  • https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:bp/improve-dvr-l3-agent-binding
  • All changes have been merged!

Open Discussion

Meeting commands

/join #openstack-meeting-alt
#startmeeting neutron_dvr
#topic Announcements
#undo topic
#link https://wiki.openstack.org/wiki/Meetings/Neutron-DVR
#action haleyb will get something specific done this week
#chair Swami
...
#endmeeting