OpenStack - User contributions [en]

Meetings/QATeamMeeting

2015-06-17T18:00:54Z

David-kranz: /* Weekly QA Team meeting */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on eavesdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for June 11th 2015 (1700 UTC) ===
* Specs Reviews
* Blueprints
** Status Updates
* Deprecation of user account options in tempest.conf
** Really only need test-accounts for real clouds with cleanup
** Tenant isolation is good for isolating failures
** How/when can we deprecate?
*** Add gate job with test accounts
*** Stop trying to support non-isolated user/demo accounts in tools (cleanup, tempest config, etc.)
* DevStack
* Grenade
* Bugs
** https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for June 11th 2015 (2200 UTC) ===
* Specs Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
** https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for June 4th 2015 (1700 UTC) ===
* Specs Reviews
** https://review.openstack.org/184992 - Tempest Plugin Interface
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 28th 2015 (2200 UTC) ===
* Specs Reviews
** https://review.openstack.org/184992 - Tempest Plugin Interface (mtreinish)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 14th 2015 (2200 UTC) ===
* Summit Session Etherpads
** https://wiki.openstack.org/wiki/Design_Summit/Liberty/Etherpads#QA
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 7th 2015 (1700 UTC) ===
* Summit Session Planning
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 30th 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 23rd 2015 (1700 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Tempest-4 tag (mtreinish)
** Things worth waiting for:
*** Updates to config guide: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:tempest-config-guide,n,z
*** requirements sync removing oslo caps: https://review.openstack.org/#/c/176851/
*** tempest-lib 0.5.0 removals??: https://review.openstack.org/#/c/166022/ and https://review.openstack.org/#/c/176039/
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews
* Tempest with python 3 (mtreinish)
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:python3-support,n,z
** https://review.openstack.org/176869

=== Agenda for April 16th 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 9th 2015 (1700 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 2nd 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* os-testr added to QA (mtreinish)
** http://git.openstack.org/cgit/openstack/os-testr/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
**https://bugs.launchpad.net/tempest/+bug/1436314 (gmann)
* Critical Reviews

=== Agenda for March 19th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
** Discuss plans for making grenade compatible with devstack plugins. Devstack patch https://review.openstack.org/#/c/161436/ is blocked by this. (ajmiller)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for March 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Can we freeze development of Neutron api tests? (marun)
* How do we make sure tempest can continue as the OpenStack regression suite as api tests are moved out? (dkranz)
** Define a canonical module path for api tests in projects.
** Define a mechanism for how tempest will locate tests that can be seen with 'testr list tests'.
** Create a job that can run tempest as is done now, but pulling and discovering api tests from projects as well.

=== Agenda for March 5th 2015 (2200 UTC) ===
* Tempest test removal proposal reviews
** https://wiki.openstack.org/wiki/QA/Tempest-test-removal
** https://etherpad.openstack.org/p/tempest-test-removals

Regular agenda if there is sufficient time after removal reviews:

* DevStack
* Grenade
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** http://docs.openstack.org/developer/tempest/configuration.html
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Blueprints
** Status Updates
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 26th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** idempotent_ids (uuids) for tests (hogepodge)
***https://blueprints.launchpad.net/tempest/+spec/test-uuid
*** https://review.openstack.org/#/c/157273/
** Status Updates
* DevStack
* Grenade
* Codifying test removal procedure: (mtreinish)
** [[QA/Tempest-test-removal]]
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** https://review.openstack.org/157935
* Tempest Python 2.6 Support (mtreinish)
** Desire to clarify messaging around tempest 2.6 support
** https://review.openstack.org/158355
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 19th 2015 (2200 UTC) ===
* QA End of Cycle Code Sprint
** http://lists.openstack.org/pipermail/openstack-dev/2015-February/057168.html
* Spec Reviews
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value (dkranz)
*** Swift status unclear, little response to ml message. Propose leaving swift as is, the usual outlier, and closing bp.
*** Final reviews: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z
* DevStack
* Grenade
* test_server_basic_ops testscenarios usage (mtreinish)
** I added a comment on https://review.openstack.org/#/c/157064/ with ideas related to this (andreaf)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
** https://bugs.launchpad.net/tempest/+bug/1296955 (dkranz)
*** How much do we care about this? There was a reference in Sean's proposal https://review.openstack.org/#/c/150653 that admin tests not be in tempest.
* Critical Reviews
* Open Discussion
** Dims needs Mentors and Project ideas for GSoC 2015 - https://wiki.openstack.org/wiki/GSoC2015 (dims OR dims__)

=== Agenda for February 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These are subject to rebase hell so would appreciate getting them in
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z

=== Agenda for February 5th 2015 (2200 UTC) ===
* Spec Reviews
** QA Program changes going into the big tent WIP draft: https://review.openstack.org/152683
* Blueprints
** Status Updates
* DevStack
* Grenade
* Tempest-lib
** Service clients (oomichi) Sorry for my absence again, but I'd like to share service clients development status with this note. As one development item, we are removing configuration values from service clients. The work is almost done, and there are still three patches for it. https://review.openstack.org/#/c/149910/ https://review.openstack.org/#/c/151530/ https://review.openstack.org/#/c/151540/ They requires a new release of tempest-lib which includes https://review.openstack.org/#/c/151070/ for merging. BTW, I found another working items for migrating service clients as library, and wrote them on the spec(https://review.openstack.org/#/c/150356/). Current spec requires a lot of patches/reviews, so I'd like to get feedback before doing that. 
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/137239/
*** This patch has several things such as "Support for IPv6", refactoring, fixing comments etc.. So I'd like to know others opinion.

=== Agenda for January 22nd 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-06-03T19:57:46Z

David-kranz: /* Proposed Agenda for June 6th 2015 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on eavesdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for June 4th 2015 (1700 UTC) ===
* Specs Reviews
** https://review.openstack.org/184992 - Tempest Plugin Interface
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for May 28th 2015 (2200 UTC) ===
* Specs Reviews
** https://review.openstack.org/184992 - Tempest Plugin Interface (mtreinish)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 14th 2015 (2200 UTC) ===
* Summit Session Etherpads
** https://wiki.openstack.org/wiki/Design_Summit/Liberty/Etherpads#QA
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 7th 2015 (1700 UTC) ===
* Summit Session Planning
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 30th 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 23rd 2015 (1700 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Tempest-4 tag (mtreinish)
** Things worth waiting for:
*** Updates to config guide: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:tempest-config-guide,n,z
*** requirements sync removing oslo caps: https://review.openstack.org/#/c/176851/
*** tempest-lib 0.5.0 removals??: https://review.openstack.org/#/c/166022/ and https://review.openstack.org/#/c/176039/
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews
* Tempest with python 3 (mtreinish)
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:python3-support,n,z
** https://review.openstack.org/176869

=== Agenda for April 16th 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 9th 2015 (1700 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 2nd 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* os-testr added to QA (mtreinish)
** http://git.openstack.org/cgit/openstack/os-testr/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
**https://bugs.launchpad.net/tempest/+bug/1436314 (gmann)
* Critical Reviews

=== Agenda for March 19th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
** Discuss plans for making grenade compatible with devstack plugins. Devstack patch https://review.openstack.org/#/c/161436/ is blocked by this. (ajmiller)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for March 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Can we freeze development of Neutron api tests? (marun)
* How do we make sure tempest can continue as the OpenStack regression suite as api tests are moved out? (dkranz)
** Define a canonical module path for api tests in projects.
** Define a mechanism for how tempest will locate tests that can be seen with 'testr list tests'.
** Create a job that can run tempest as is done now, but pulling and discovering api tests from projects as well.

=== Agenda for March 5th 2015 (2200 UTC) ===
* Tempest test removal proposal reviews
** https://wiki.openstack.org/wiki/QA/Tempest-test-removal
** https://etherpad.openstack.org/p/tempest-test-removals

Regular agenda if there is sufficient time after removal reviews:

* DevStack
* Grenade
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** http://docs.openstack.org/developer/tempest/configuration.html
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Blueprints
** Status Updates
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 26th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** idempotent_ids (uuids) for tests (hogepodge)
***https://blueprints.launchpad.net/tempest/+spec/test-uuid
*** https://review.openstack.org/#/c/157273/
** Status Updates
* DevStack
* Grenade
* Codifying test removal procedure: (mtreinish)
** [[QA/Tempest-test-removal]]
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** https://review.openstack.org/157935
* Tempest Python 2.6 Support (mtreinish)
** Desire to clarify messaging around tempest 2.6 support
** https://review.openstack.org/158355
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 19th 2015 (2200 UTC) ===
* QA End of Cycle Code Sprint
** http://lists.openstack.org/pipermail/openstack-dev/2015-February/057168.html
* Spec Reviews
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value (dkranz)
*** Swift status unclear, little response to ml message. Propose leaving swift as is, the usual outlier, and closing bp.
*** Final reviews: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z
* DevStack
* Grenade
* test_server_basic_ops testscenarios usage (mtreinish)
** I added a comment on https://review.openstack.org/#/c/157064/ with ideas related to this (andreaf)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
** https://bugs.launchpad.net/tempest/+bug/1296955 (dkranz)
*** How much do we care about this? There was a reference in Sean's proposal https://review.openstack.org/#/c/150653 that admin tests not be in tempest.
* Critical Reviews
* Open Discussion
** Dims needs Mentors and Project ideas for GSoC 2015 - https://wiki.openstack.org/wiki/GSoC2015 (dims OR dims__)

=== Agenda for February 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These are subject to rebase hell so would appreciate getting them in
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z

=== Agenda for February 5th 2015 (2200 UTC) ===
* Spec Reviews
** QA Program changes going into the big tent WIP draft: https://review.openstack.org/152683
* Blueprints
** Status Updates
* DevStack
* Grenade
* Tempest-lib
** Service clients (oomichi) Sorry for my absence again, but I'd like to share service clients development status with this note. As one development item, we are removing configuration values from service clients. The work is almost done, and there are still three patches for it. https://review.openstack.org/#/c/149910/ https://review.openstack.org/#/c/151530/ https://review.openstack.org/#/c/151540/ They requires a new release of tempest-lib which includes https://review.openstack.org/#/c/151070/ for merging. BTW, I found another working items for migrating service clients as library, and wrote them on the spec(https://review.openstack.org/#/c/150356/). Current spec requires a lot of patches/reviews, so I'd like to get feedback before doing that. 
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/137239/
*** This patch has several things such as "Support for IPv6", refactoring, fixing comments etc.. So I'd like to know others opinion.

=== Agenda for January 22nd 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-06-03T19:57:32Z

David-kranz: /* Weekly QA Team meeting */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on eavesdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for June 6th 2015 (1700 UTC) ===
* Specs Reviews
** https://review.openstack.org/184992 - Tempest Plugin Interface
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for May 28th 2015 (2200 UTC) ===
* Specs Reviews
** https://review.openstack.org/184992 - Tempest Plugin Interface (mtreinish)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 14th 2015 (2200 UTC) ===
* Summit Session Etherpads
** https://wiki.openstack.org/wiki/Design_Summit/Liberty/Etherpads#QA
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for May 7th 2015 (1700 UTC) ===
* Summit Session Planning
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 30th 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 23rd 2015 (1700 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Tempest-4 tag (mtreinish)
** Things worth waiting for:
*** Updates to config guide: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:tempest-config-guide,n,z
*** requirements sync removing oslo caps: https://review.openstack.org/#/c/176851/
*** tempest-lib 0.5.0 removals??: https://review.openstack.org/#/c/166022/ and https://review.openstack.org/#/c/176039/
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews
* Tempest with python 3 (mtreinish)
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:python3-support,n,z
** https://review.openstack.org/176869

=== Agenda for April 16th 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 9th 2015 (1700 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for April 2nd 2015 (2200 UTC) ===
* Summit Session Brainstorm
** https://etherpad.openstack.org/p/liberty-qa-summit-topics
* Spec Reviews
* Blueprints
** Status Updates
* os-testr added to QA (mtreinish)
** http://git.openstack.org/cgit/openstack/os-testr/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
**https://bugs.launchpad.net/tempest/+bug/1436314 (gmann)
* Critical Reviews

=== Agenda for March 19th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
** Discuss plans for making grenade compatible with devstack plugins. Devstack patch https://review.openstack.org/#/c/161436/ is blocked by this. (ajmiller)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for March 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Can we freeze development of Neutron api tests? (marun)
* How do we make sure tempest can continue as the OpenStack regression suite as api tests are moved out? (dkranz)
** Define a canonical module path for api tests in projects.
** Define a mechanism for how tempest will locate tests that can be seen with 'testr list tests'.
** Create a job that can run tempest as is done now, but pulling and discovering api tests from projects as well.

=== Agenda for March 5th 2015 (2200 UTC) ===
* Tempest test removal proposal reviews
** https://wiki.openstack.org/wiki/QA/Tempest-test-removal
** https://etherpad.openstack.org/p/tempest-test-removals

Regular agenda if there is sufficient time after removal reviews:

* DevStack
* Grenade
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** http://docs.openstack.org/developer/tempest/configuration.html
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Blueprints
** Status Updates
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 26th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** idempotent_ids (uuids) for tests (hogepodge)
***https://blueprints.launchpad.net/tempest/+spec/test-uuid
*** https://review.openstack.org/#/c/157273/
** Status Updates
* DevStack
* Grenade
* Codifying test removal procedure: (mtreinish)
** [[QA/Tempest-test-removal]]
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** https://review.openstack.org/157935
* Tempest Python 2.6 Support (mtreinish)
** Desire to clarify messaging around tempest 2.6 support
** https://review.openstack.org/158355
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 19th 2015 (2200 UTC) ===
* QA End of Cycle Code Sprint
** http://lists.openstack.org/pipermail/openstack-dev/2015-February/057168.html
* Spec Reviews
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value (dkranz)
*** Swift status unclear, little response to ml message. Propose leaving swift as is, the usual outlier, and closing bp.
*** Final reviews: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z
* DevStack
* Grenade
* test_server_basic_ops testscenarios usage (mtreinish)
** I added a comment on https://review.openstack.org/#/c/157064/ with ideas related to this (andreaf)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
** https://bugs.launchpad.net/tempest/+bug/1296955 (dkranz)
*** How much do we care about this? There was a reference in Sean's proposal https://review.openstack.org/#/c/150653 that admin tests not be in tempest.
* Critical Reviews
* Open Discussion
** Dims needs Mentors and Project ideas for GSoC 2015 - https://wiki.openstack.org/wiki/GSoC2015 (dims OR dims__)

=== Agenda for February 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These are subject to rebase hell so would appreciate getting them in
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z

=== Agenda for February 5th 2015 (2200 UTC) ===
* Spec Reviews
** QA Program changes going into the big tent WIP draft: https://review.openstack.org/152683
* Blueprints
** Status Updates
* DevStack
* Grenade
* Tempest-lib
** Service clients (oomichi) Sorry for my absence again, but I'd like to share service clients development status with this note. As one development item, we are removing configuration values from service clients. The work is almost done, and there are still three patches for it. https://review.openstack.org/#/c/149910/ https://review.openstack.org/#/c/151530/ https://review.openstack.org/#/c/151540/ They requires a new release of tempest-lib which includes https://review.openstack.org/#/c/151070/ for merging. BTW, I found another working items for migrating service clients as library, and wrote them on the spec(https://review.openstack.org/#/c/150356/). Current spec requires a lot of patches/reviews, so I'd like to get feedback before doing that. 
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/137239/
*** This patch has several things such as "Support for IPv6", refactoring, fixing comments etc.. So I'd like to know others opinion.

=== Agenda for January 22nd 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-03-12T15:54:08Z

David-kranz: /* Proposed Agenda for March 12th 2015 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on eavesdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for March 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Can we freeze development of Neutron api tests? (marun)
* How do we make sure tempest can continue as the OpenStack regression suite as api tests are moved out? (dkranz)
** Define a canonical module path for api tests in projects.
** Define a mechanism for how tempest will locate tests that can be seen with 'testr list tests'.
** Create a job that can run tempest as is done now, but pulling and discovering api tests from projects as well.

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for March 5th 2015 (2200 UTC) ===
* Tempest test removal proposal reviews
** https://wiki.openstack.org/wiki/QA/Tempest-test-removal
** https://etherpad.openstack.org/p/tempest-test-removals

Regular agenda if there is sufficient time after removal reviews:

* DevStack
* Grenade
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** http://docs.openstack.org/developer/tempest/configuration.html
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Blueprints
** Status Updates
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 26th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** idempotent_ids (uuids) for tests (hogepodge)
***https://blueprints.launchpad.net/tempest/+spec/test-uuid
*** https://review.openstack.org/#/c/157273/
** Status Updates
* DevStack
* Grenade
* Codifying test removal procedure: (mtreinish)
** [[QA/Tempest-test-removal]]
* Tempest Configuration Doc (mtreinish)
** Start of having a tempest configuration guide
** https://review.openstack.org/157935
* Tempest Python 2.6 Support (mtreinish)
** Desire to clarify messaging around tempest 2.6 support
** https://review.openstack.org/158355
* new tag for interop tests (hogepodge)
** Discuss desire for new [interop] test tag with requirement for a minimal amount of non-admin config information.
* Neutron shared network testing
** http://www.fpaste.org/190978/14249673/
** https://review.openstack.org/#/c/124998
** http://logs.openstack.org/98/124998/17/check/check-tempest-dsvm-neutron-full/01d6009/console.html.gz#_2014-11-21_07_45_29_348
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for February 19th 2015 (2200 UTC) ===
* QA End of Cycle Code Sprint
** http://lists.openstack.org/pipermail/openstack-dev/2015-February/057168.html
* Spec Reviews
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value (dkranz)
*** Swift status unclear, little response to ml message. Propose leaving swift as is, the usual outlier, and closing bp.
*** Final reviews: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z
* DevStack
* Grenade
* test_server_basic_ops testscenarios usage (mtreinish)
** I added a comment on https://review.openstack.org/#/c/157064/ with ideas related to this (andreaf)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
** https://bugs.launchpad.net/tempest/+bug/1296955 (dkranz)
*** How much do we care about this? There was a reference in Sean's proposal https://review.openstack.org/#/c/150653 that admin tests not be in tempest.
* Critical Reviews
* Open Discussion
** Dims needs Mentors and Project ideas for GSoC 2015 - https://wiki.openstack.org/wiki/GSoC2015 (dims OR dims__)

=== Agenda for February 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These are subject to rebase hell so would appreciate getting them in
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z

=== Agenda for February 5th 2015 (2200 UTC) ===
* Spec Reviews
** QA Program changes going into the big tent WIP draft: https://review.openstack.org/152683
* Blueprints
** Status Updates
* DevStack
* Grenade
* Tempest-lib
** Service clients (oomichi) Sorry for my absence again, but I'd like to share service clients development status with this note. As one development item, we are removing configuration values from service clients. The work is almost done, and there are still three patches for it. https://review.openstack.org/#/c/149910/ https://review.openstack.org/#/c/151530/ https://review.openstack.org/#/c/151540/ They requires a new release of tempest-lib which includes https://review.openstack.org/#/c/151070/ for merging. BTW, I found another working items for migrating service clients as library, and wrote them on the spec(https://review.openstack.org/#/c/150356/). Current spec requires a lot of patches/reviews, so I'd like to get feedback before doing that. 
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/137239/
*** This patch has several things such as "Support for IPv6", refactoring, fixing comments etc.. So I'd like to know others opinion.

=== Agenda for January 22nd 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-02-19T14:03:36Z

David-kranz: /* Proposed Agenda for February 19th 2015 (2200 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on eavesdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for February 19th 2015 (2200 UTC) ===
* QA End of Cycle Code Sprint
** http://lists.openstack.org/pipermail/openstack-dev/2015-February/057168.html
* Spec Reviews
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value (dkranz)
*** Swift status unclear, little response to ml message. Propose leaving swift as is, the usual outlier, and closing bp.
*** Final reviews: https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z
* DevStack
* Grenade
* test_server_basic_ops testscenarios usage (mtreinish)
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
** https://bugs.launchpad.net/tempest/+bug/1296955 (dkranz)
*** How much do we care about this? There was a reference in Sean's proposal https://review.openstack.org/#/c/150653 that admin tests not be in tempest.
* Critical Reviews
* Open Discussion
** Dims needs Mentors and Project ideas for GSoC 2015 - https://wiki.openstack.org/wiki/GSoC2015 (dims OR dims__)

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for February 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These are subject to rebase hell so would appreciate getting them in
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z

=== Agenda for February 5th 2015 (2200 UTC) ===
* Spec Reviews
** QA Program changes going into the big tent WIP draft: https://review.openstack.org/152683
* Blueprints
** Status Updates
* DevStack
* Grenade
* Tempest-lib
** Service clients (oomichi) Sorry for my absence again, but I'd like to share service clients development status with this note. As one development item, we are removing configuration values from service clients. The work is almost done, and there are still three patches for it. https://review.openstack.org/#/c/149910/ https://review.openstack.org/#/c/151530/ https://review.openstack.org/#/c/151540/ They requires a new release of tempest-lib which includes https://review.openstack.org/#/c/151070/ for merging. BTW, I found another working items for migrating service clients as library, and wrote them on the spec(https://review.openstack.org/#/c/150356/). Current spec requires a lot of patches/reviews, so I'd like to get feedback before doing that. 
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/137239/
*** This patch has several things such as "Support for IPv6", refactoring, fixing comments etc.. So I'd like to know others opinion.

=== Agenda for January 22nd 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-02-12T16:30:26Z

David-kranz: /* Proposed Agenda for February 12th 2015 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on eavesdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for February 12th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These are subject to rebase hell so would appreciate getting them in
** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/clients-return-one-value,n,z

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for February 5th 2015 (2200 UTC) ===
* Spec Reviews
** QA Program changes going into the big tent WIP draft: https://review.openstack.org/152683
* Blueprints
** Status Updates
* DevStack
* Grenade
* Tempest-lib
** Service clients (oomichi) Sorry for my absence again, but I'd like to share service clients development status with this note. As one development item, we are removing configuration values from service clients. The work is almost done, and there are still three patches for it. https://review.openstack.org/#/c/149910/ https://review.openstack.org/#/c/151530/ https://review.openstack.org/#/c/151540/ They requires a new release of tempest-lib which includes https://review.openstack.org/#/c/151070/ for merging. BTW, I found another working items for migrating service clients as library, and wrote them on the spec(https://review.openstack.org/#/c/150356/). Current spec requires a lot of patches/reviews, so I'd like to get feedback before doing that. 
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/137239/
*** This patch has several things such as "Support for IPv6", refactoring, fixing comments etc.. So I'd like to know others opinion.

=== Agenda for January 22nd 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-01-15T16:44:08Z

David-kranz: /* Proposed Agenda for January 15th 2015 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/146039/

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-01-15T01:42:59Z

David-kranz: /* Weekly QA Team meeting */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for January 15th 2015 (1700 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2015-01-08T13:55:00Z

David-kranz: /* Proposed Agenda for January 8th 2015 (2200 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for January 8th 2015 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** These would be good to merge to avoid rebases
*** https://review.openstack.org/145295
*** https://review.openstack.org/144473

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2014-12-18T16:38:16Z

David-kranz: /* Proposed Agenda for December 18th 2014 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for December 18th 2014 (1700 UTC) ===
* No meeting next 2 weeks (mtreinish)
** Volunteers to run meetings Jan 8th - Jan 29th
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
** https://review.openstack.org/#/c/141014/ (dkranz)

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
** Pre-caching new images first: https://review.openstack.org/#/c/140462/
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2014-12-11T02:23:35Z

David-kranz: /* Proposed Agenda for December 11th 2014 (2200 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for December 11th 2014 (2200 UTC) ===
* Spec Reviews
* Blueprints
** Status Updates
* Proposed plan for moving neutron rest client to tempest lib (dkranz)
** Refactor rest client so it can be separated from tempest
** Same for neutron client, including applying the clients-return-one-value spec
*** This requires a "big bang" so we need to stop merging neutron patches while this is in review
** Copy rest and neutron client to neutron and get it working there, fixing things as needed
** Copy any changes back to tempest
** Move rest and neutron client to tempest-lib
** neutron replaces copied client code with reference to tempest-lib
* DevStack
** Blueprints
*** External Plugins: https://review.openstack.org/#/c/137054/
*** Logging and Service Names: https://review.openstack.org/140843
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for December 4th 2014 (1700 UTC) ===
* Tagging removal of Tempest XML support? (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Future of "large-ops" testing
** does current test need 6 gates?
** alternate test in stress/scenario
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for November 20th 2014 (1700 UTC) ===
* XML testing (mtreinish)
* Spec Reviews
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 30th 2014 (2200 UTC) ===
* No meeting next week (mtreinish)
** Volunteer to run Nov 13th meeting (2200 UTC)
* Blueprints
** Status Updates
***Resource-cleanup: https://review.openstack.org/115353 looking for opinions
* large-ops gate (yfried - feel free to move this down the list - I was unsure were to fit it)
** currently the test could be part of api.compute or move to nova tree as it's not doing any integration. I have a patch to change that but this would require some changes to the gate that I'd like to bring up https://review.openstack.org/#/c/120070/
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Design Summit/Kilo/Etherpads

2014-10-29T14:43:22Z

David-kranz: /* Cross-Project workshops */

[[Category:Summit]]
[[Category:Kilo]]
[[Category:Etherpad]]

<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
__TOC__
</div>

==Cross-Project workshops==
* Tue 1115 – 1245 [https://etherpad.openstack.org/p/kilo-crossproject-api-wg API Working Group]
* Tue 1205 – 1245 [https://etherpad.openstack.org/p/kilo-crossproject-move-func-tests-to-projects Moving Functional Tests to Projects]
* Tue 1640 – 1800 [https://etherpad.openstack.org/p/kilo-crossproject-growth-challenges Growth Challenges]

==Barbican==
* tbd

==Ceilometer==
* tbd

==Cinder==
* tbd

==Designate==
* tbd

==Glance==
* tbd

==Heat==
* tbd

==Horizon==
* Wed 1440 – 1520 [https://etherpad.openstack.org/p/kilo-horizon-operators-users Operators, Deployers and Users]
* Wed 1530 – 1610 [https://etherpad.openstack.org/p/kilo-keystone-horizon-cli-federation-sso Horizon/Keystone Cross Topic Session]

* Thurs 950 – 1030 [https://etherpad.openstack.org/p/kilo-horizon-django-angular Django-Angular Playing Nice]
* Thurs 1100 – 1140 [https://etherpad.openstack.org/p/kilo-horizon-clientside-data Clientside Data]
* Thurs 1630 – 1710 [https://etherpad.openstack.org/p/kilo-horizon-customize-extend Making Horizon easier to customize and extend]
* Thurs 1720 – 1800 [https://etherpad.openstack.org/p/kilo-horizon-ux UX and Horizon]

* Fri 1340 – 1710 [https://etherpad.openstack.org/p/kilo-horizon-contributors-meetup Contributors Meetup]

==Infrastructure==
* Thu 1520 - 1600 [https://etherpad.openstack.org/p/StoryBoard_design_summit_Paris Storyboard]

==Ironic==
* tbd

==Keystone==
* Wed 0900 – 0940 [https://etherpad.openstack.org/p/kilo-keystone-object-lifecycle Object Lifecycle / Object Dependencies]
* Wed 0950 – 1030 [https://etherpad.openstack.org/p/hierarchical-multitenancy-kilo-summit Hierarchical Multitenancy]
* Wed 1630 – 1710 [https://etherpad.openstack.org/p/kilo-keystone-horizon-cli-federation-sso Keystone-Horizon-CLI Federation/SSO]
* Wed 1720 – 1800 [https://etherpad.openstack.org/p/kilo-keystone-operators-deployers-and-devops Operators, Deployers, and DevOps]

* Thu 0900 – 9040 [https://etherpad.openstack.org/p/kilo-keystone-python-keystoneclient python-keystoneclient]
* Thu 1430 – 1510 [https://etherpad.openstack.org/p/kilo-keystone-authorization Authorization]
* Thu 1520 – 1600 [https://etherpad.openstack.org/p/kilo-keystone-policy-model-token-capabilities Policy Model and User/Token Capabilities]

* Fri 1340 – 1710 [https://etherpad.openstack.org/p/kilo-keystone-meetup Keystone Contributor Meetup]

==Manila==
* tbd

==Neutron==
* tbd

==Nova==
* Thurs 1100 – 1230 [https://etherpad.openstack.org/p/kilo-nova-scheduler-rt Scheduler and Resource Tracker]

==Oslo==
* Wed 2014-11-05 11:00 - [https://etherpad.openstack.org/p/kilo-oslo-library-proposals Oslo graduation schedule]
* Wed 2014-11-05 11:50 - [https://etherpad.openstack.org/p/kilo-oslo-oslo.messaging oslo.messaging ]
* Wed 2014-11-05 13:50 - [https://etherpad.openstack.org/p/kilo-oslo-common-quota-library A Common Quota Management Library]
* Thu 2014-11-06 11:50 - [https://etherpad.openstack.org/p/kilo-oslo-taskflow taskflow]
* Thu 2014-11-06 13:40 - [https://etherpad.openstack.org/p/kilo-oslo-alpha-versioning Using alpha versioning for Oslo libraries]
* Thu 2014-11-06 16:30 - [https://etherpad.openstack.org/p/kilo-oslo-python-3 Python 3 support in Oslo]
* Thu 2014-11-06 17:20 - [https://etherpad.openstack.org/p/kilo-oslo-namespace-packages Moving Oslo away from namespace packages]

==QA==
* Wed 9:00 - 9:40 [https://etherpad.openstack.org/p/kilo-summit-gap-analysis-tempest-in-production Gap Analysis for using Tempest in Production]
* Wed 11:00 - 11:40 [https://etherpad.openstack.org/p/kilo-summit-tempest-scope-in-the-brave-new-world Tempest Scope in the Brave New World]
* Wed 14:40 - 13:20
* Wed 15:30 - 16:10 [https://etherpad.openstack.org/p/kilo-summit-post-merge-qa-ci QA and CI After Merge]
* Wed 17:20 - 18:00 [https://etherpad.openstack.org/p/kilo-summit-tempest-lib-moving-forward Tempest-lib moving forward]
* Thu 11:50 - 12:30 Future Auth Interface Design
* Thu 14:30 - 13:10 [https://etherpad.openstack.org/p/kilo-summit-devstack-grenade Devstack/Grenade]

==Release Management==
* Wed 1150 – 1230 [https://etherpad.openstack.org/p/kilo-relmgt-stable-branches Stable branches]
* Thu 0950 – 1030 [https://etherpad.openstack.org/p/kilo-relmgt-vulnerability-management Vulnerability management]

==Sahara==
* tbd

==Swift==
* tbd

==TripleO==
* tbd

==Trove==
Trovesday (Thursday) 
14:30 - 15:10: [https://etherpad.openstack.org/p/kilo-summit-trove-clusters Building Out Trove Clusters] 
15:20 - 16:00: [https://etherpad.openstack.org/p/kilo-summit-trove-replication-v2 Replication v2 and Scheduling Tasks in Trove] 
16:30 - 17:10: [https://etherpad.openstack.org/p/kilo-summit-testing-trove Testing Trove] 
17:20 - 18:00: [https://etherpad.openstack.org/p/kilo-summit-trove-openstack-tighter-integration Trove integration with other OpenStack components] 

==Zaqar==
* Tuesday 11:15 [https://etherpad.openstack.org/p/kilo-zaqar-summit-integration-with-services Zaqar integration with other services]
* Tuesday 12:05 [https://etherpad.openstack.org/p/kilo-zaqar-summit-v2 Zaqar API v2: What? Why? When?]
* Tuesday 14:00 [https://etherpad.openstack.org/p/kilo-zaqar-summit-persistent-transports Zaqar Persistent Transports]
* Tuesday 14:50 [https://etherpad.openstack.org/p/kilo-zaqar-summit-infrastructure Zaqar Infrastructure Session]

==Other Projects==
* tbd

==Event intro/closure==
* Tue 1115 – 1155 [https://etherpad.openstack.org/p/kilo-summit-101 Summit 101]
* Fri 1720 – 1800 [https://etherpad.openstack.org/p/kilo-summit-feedback Design Summit feedback]

==Ops Summit==
* Mon 1140 – 1220 [https://etherpad.openstack.org/p/kilo-summit-ops-pain Top 10 Pain points from the user survey - how to fix them?]
* Mon 1140 – 1220 [https://etherpad.openstack.org/p/kilo-summit-ops-pets What is the best practice for managing 'pets'?]
* Mon 1230 – 1310 [https://etherpad.openstack.org/p/kilo-summit-ops-logging How do we fix logging?]
* Mon 1230 – 1310 [https://etherpad.openstack.org/p/kilo-summit-ops-dvr Distributed Virtual Router - finally neutron HA!]
* Mon 1430 – 1510 [https://etherpad.openstack.org/p/kilo-summit-ops-ironic What do you want from ironic/bare metal?]
* Mon 1430 – 1510 [https://etherpad.openstack.org/p/kilo-summit-ops-app-eco The Application Ecosystem working group]
* Mon 1520 – 1600 [https://etherpad.openstack.org/p/kilo-summit-ops-containers What do we want from containers/docker?]
* Mon 1520 – 1600 [https://etherpad.openstack.org/p/kilo-summit-ops-architecture-upgrades Architecture Show and Tell - Upgrades Special Edition]
* Mon 1620 – 1700 [https://etherpad.openstack.org/p/kilo-summit-ops-ha High availability - how do you do it?]
* Mon 1620 – 1700 [https://etherpad.openstack.org/p/kilo-summit-ops-architecture Architecture Show and Tell]
* Mon 1710 – 1750 [https://etherpad.openstack.org/p/kilo-summit-ops-get-involved How to get involved in Kilo?]
* Mon 1710 – 1750 [https://etherpad.openstack.org/p/kilo-summit-ops-storage Storage - how do you do it?]

* Thu 0900 – 1030 [https://etherpad.openstack.org/p/kilo-summit-ops-log-rationalisation Log Rationalisation]
* Thu 0900 – 1030 [https://etherpad.openstack.org/p/kilo-summit-ops-upgrades Upgrades]
* Thu 0900 – 1030 [https://etherpad.openstack.org/p/kilo-summit-ops-telco Telco WG]

* Thu 1100 – 1230 [https://etherpad.openstack.org/p/kilo-summit-ops-monitoring Monitoring]
* Thu 1100 – 1230 [https://etherpad.openstack.org/p/kilo-summit-ops-docs Documentation]
* Thu 1100 – 1140 [https://etherpad.openstack.org/p/kilo-summit-ops-stable-branch Stable Branch]
* Thu 1150 – 1230 [https://etherpad.openstack.org/p/kilo-summit-ops-chef Chef]

* Thu 1340 – 1550 [https://etherpad.openstack.org/p/kilo-summit-ops-app-eco The Application Ecosystem working group]
* Thu 1340 – 1550 [https://etherpad.openstack.org/p/kilo-summit-ops-blueprints Blueprints Working Group]
* Thu 1340 – 1500 [https://etherpad.openstack.org/p/kilo-summit-ops-tools Ops Tools working group]
* Thu 1510 – 1550 [https://etherpad.openstack.org/p/kilo-summit-ops-puppet Puppet]

* Thu 1630 – 1800 [https://etherpad.openstack.org/p/kilo-summit-ops-large-deployments Large Deployments Team]
* Thu 1630 – 1800 [https://etherpad.openstack.org/p/kilo-summit-ops-api API Working Group]
* Thu 1630 – 1800 [https://etherpad.openstack.org/p/kilo-summit-ops-enterprise Win The Enterprise Working Group]

Meetings/QATeamMeeting

2014-10-23T13:53:03Z

David-kranz: /* Proposed Agenda for October 23rd 2014 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Proposed Agenda for October 23rd 2014 (1700 UTC) ===
* Blueprints
** Status Updates
** https://blueprints.launchpad.net/tempest/+spec/clients-return-one-value
*** Are there any objections to this?
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for October 16th 2014 (2200 UTC) ===
* Summit Planning
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Rally/HowTo

2014-10-15T17:10:33Z

David-kranz: /* Step 2. Benchmarking */

== Usage demo ==

'''''NOTE:''''' Throughout this demo, we assume that you have a configured [[Rally/installation|Rally installation]] and an already existing OpenStack deployment has keystone available at ''<KEYSTONE_AUTH_URL>''.

=== Step 1. Deployment initialization (use existing cloud) ===
First, you have to provide Rally with an Openstack deployment it is going to benchmark. This is done through '''''deployment configuration files'''''. The actual deployment can be either created by Rally (see ''/doc/samples'' for configuration examples) or, as in our example, an already existing one. The configuration file (let's call it '''existing.json''') should contain the deployment strategy (in our case, the deployment will be performed by the so called '''''"ExistingCloud"''''', since the deployment is ready to use) and some specific parameters (for the ''ExistingCloud'', an endpoint with administrator permissions):

<pre>
{
"type": "ExistingCloud",
"auth_url": <KEYSTONE_AUTH_URL>,
"admin": {
"username": <ADMIN_USER_NAME>,
"password": <ADMIN_PASSWORD>,
"tenant_name": <ADMIN_TENANT>
}
}
</pre>

To register this deployment in Rally, use the '''deployment create''' command:

<pre>
$ rally deployment create --filename=existing.json --name=existing
+---------------------------+----------------------------+----------+------------------+
| uuid | created_at | name | status |
+---------------------------+----------------------------+----------+------------------+
| <Deployment UUID> | 2014-02-15 22:00:28.270941 | existing | deploy->finished |
+---------------------------+----------------------------+----------+------------------+
Using deployment : <Deployment UUID>
</pre>

Note the last line in the output. It says that the just created deployment is now used by Rally; that means that all the benchmarking operations from now on are going to be performed on this deployment. In case you want to switch to another deployment, execute the '''use deployment''' command:

<pre>
$ rally use deployment --uuid=<Another deployment UUID>
Using deployment : <Another deployment UUID>
</pre>

Finally, the '''deployment check''' command enables you to verify that your current deployment is healthy and ready to be benchmarked:

<pre>
$ rally deployment check
+----------+-----------+-----------+
| services | type | status |
+----------+-----------+-----------+
| nova | compute | Available |
| cinderv2 | volumev2 | Available |
| novav3 | computev3 | Available |
| s3 | s3 | Available |
| glance | image | Available |
| cinder | volume | Available |
| ec2 | ec2 | Available |
| keystone | identity | Available |
+----------+-----------+-----------+
</pre>

=== Step 2. Benchmarking ===
Now that we have a working and registered deployment, we can start benchmarking it. Again, the sequence of benchmark scenarios to be launched by Rally should be specified in a '''''benchmark task configuration file'''''. Note that there is already a set of nice benchmark tasks examples in ''doc/samples/tasks/'' (assuming that you are in the Rally root directory). The natural thing would be just to try one of these sample benchmark tasks, say, the one that boots and deletes multiple servers (''doc/samples/tasks/scenarios/nova/boot-and-delete.json''). To start a benchmark task, run the task start command:


<pre>
ubuntu@tempeste-test:~$ rally -v task start rally/doc/samples/tasks/scenarios/nova/boot-and-delete.json
================================================================================
Task 392c803b-37fd-4915-9732-3523f4252e9b is started
--------------------------------------------------------------------------------
2014-03-20 06:17:39.994 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Starting: Check cloud.
2014-03-20 06:17:40.123 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Completed: Check cloud.
2014-03-20 06:17:40.123 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Starting: Task validation.
2014-03-20 06:17:40.133 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Starting: Task validation of scenarios names.
2014-03-20 06:17:40.137 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Completed: Task validation of scenarios names.
2014-03-20 06:17:40.138 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Starting: Task validation of syntax.
2014-03-20 06:17:40.140 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Completed: Task validation of syntax.
2014-03-20 06:17:40.140 27502 INFO rally.benchmark.engine [-] Task 392c803b-37fd-4915-9732-3523f4252e9b | Starting: Task validation of semantic.
2014-03-20 06:17:41.098 27502 ERROR glanceclient.common.http [-] Request returned failure status.

================================================================================
Task 392c803b-37fd-4915-9732-3523f4252e9b is failed.
--------------------------------------------------------------------------------
<class 'rally.exceptions.InvalidBenchmarkConfig'>
Task config is invalid.
Benchmark NovaServers.boot_and_delete_server has wrong configuration of args at position 0: {'image_id': '73257560-c59b-4275-a1ec-ab140e5b9979', 'flavor_id': 1}
Reason: Image with id '73257560-c59b-4275-a1ec-ab140e5b9979' not found

For more details run:
rally -vd task detailed 392c803b-37fd-4915-9732-3523f4252e9b
</pre>


This attempt, however, will most likely fail because of an '''''input arguments validation error''''' (due to a non-existing image name). The thing is that the benchmark scenario that boots a server needs to do that using a concrete image available in the OpenStack deployment. In prior iterations of Rally, the images were denoted by UUID (such as "flavor_id", "image_id", etc). Now, these resources are simply denoted by name.

To get started, make a local copy of the sample benchmark task:

<pre>
cp doc/samples/tasks/scenarios/nova/boot-and-delete.json my-task.json
</pre>

and then edit it with the resource names from your OpenStack installation:

<pre>
{
"NovaServers.boot_and_delete_server": [
{
"args": {
"flavor": {
"name": "m1.tiny"
},
"image": {
"name": "CirrOS 0.3.1 (x86_64)"
}
},
"runner": {
"type": "constant",
"times": 10,
"concurrency": 2
},
"context": {
"users": {
"tenants": 3,
"users_per_tenant": 2
}
}
}
]
}
</pre>

To obtain proper image name and flavor name, you can use the subcommand ''show'' of rally.

let's get a proper image name:

<pre>
$ rally show images
+--------------------------------------+-----------------------+-----------+
| UUID | Name | Size (B) |
+--------------------------------------+-----------------------+-----------+
| 8dfd6098-0c26-4cb5-8e77-1ecb2db0b8ae | CentOS 6.5 (x86_64) | 344457216 |
| 2b8d119e-9461-48fc-885b-1477abe2edc5 | CirrOS 0.3.1 (x86_64) | 13147648 |
+--------------------------------------+-----------------------+-----------+
</pre>

and a proper flavor name:

<pre>
$ rally show flavors
+---------------------+-----------+-------+----------+-----------+-----------+
| ID | Name | vCPUs | RAM (MB) | Swap (MB) | Disk (GB) |
+---------------------+-----------+-------+----------+-----------+-----------+
| 1 | m1.tiny | 1 | 512 | | 1 |
| 2 | m1.small | 1 | 2048 | | 20 |
| 3 | m1.medium | 2 | 4096 | | 40 |
| 4 | m1.large | 4 | 8192 | | 80 |
| 5 | m1.xlarge | 8 | 16384 | | 160 |
+---------------------+-----------+-------+----------+-----------+-----------+
</pre>

After you've edited the '''''my-task.json''''' file, you can run this benchmark task again. This time, let's also use the ''--verbose'' parameter that will allow us to retrieve more logging from Rally while it performs benchmarking:

<pre>
$ rally -v task start my-task.json --tag my_task

================================================================================
Task my_task 87eb8ff3-07f9-4941-b1be-63e707aceb1e is started
--------------------------------------------------------------------------------
2014-03-20 06:26:36.431 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Check cloud.
2014-03-20 06:26:36.555 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Check cloud.
2014-03-20 06:26:36.555 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Task validation.
2014-03-20 06:26:36.564 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Task validation of scenarios names.
2014-03-20 06:26:36.568 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Task validation of scenarios names.
2014-03-20 06:26:36.568 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Task validation of syntax.
2014-03-20 06:26:36.571 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Task validation of syntax.
2014-03-20 06:26:36.571 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Task validation of semantic.
2014-03-20 06:26:37.316 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Task validation of semantic.
2014-03-20 06:26:37.316 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Task validation.
2014-03-20 06:26:37.316 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Benchmarking.
2014-03-20 06:26:41.596 27820 INFO rally.benchmark.runners.base [-] ITER: 0 START
2014-03-20 06:26:41.596 27821 INFO rally.benchmark.runners.base [-] ITER: 1 START
2014-03-20 06:26:46.105 27820 INFO rally.benchmark.runners.base [-] ITER: 0 END: Error <class 'rally.exceptions.GetResourceNotFound'>: Resource not found: `404`
2014-03-20 06:26:46.105 27820 INFO rally.benchmark.runners.base [-] ITER: 2 START
2014-03-20 06:26:46.451 27821 INFO rally.benchmark.runners.base [-] ITER: 1 END: Error <type 'exceptions.AttributeError'>: status
2014-03-20 06:26:46.452 27821 INFO rally.benchmark.runners.base [-] ITER: 3 START
2014-03-20 06:26:46.497 27820 INFO rally.benchmark.runners.base [-] ITER: 2 END: Error <class 'novaclient.exceptions.NotFound'>: Instance could not be found (HTTP 404) (Request-ID: req-dfd372e9-728d-49ca-87e1-54cbf593b2be)
2014-03-20 06:26:46.497 27820 INFO rally.benchmark.runners.base [-] ITER: 4 START
2014-03-20 06:26:53.274 27821 INFO rally.benchmark.runners.base [-] ITER: 3 END: OK
2014-03-20 06:26:53.275 27821 INFO rally.benchmark.runners.base [-] ITER: 5 START
2014-03-20 06:26:53.709 27820 INFO rally.benchmark.runners.base [-] ITER: 4 END: OK
2014-03-20 06:26:53.710 27820 INFO rally.benchmark.runners.base [-] ITER: 6 START
2014-03-20 06:26:59.942 27821 INFO rally.benchmark.runners.base [-] ITER: 5 END: OK
2014-03-20 06:26:59.943 27821 INFO rally.benchmark.runners.base [-] ITER: 7 START
2014-03-20 06:27:00.601 27820 INFO rally.benchmark.runners.base [-] ITER: 6 END: OK
2014-03-20 06:27:00.601 27820 INFO rally.benchmark.runners.base [-] ITER: 8 START
2014-03-20 06:27:06.635 27821 INFO rally.benchmark.runners.base [-] ITER: 7 END: OK
2014-03-20 06:27:06.635 27821 INFO rally.benchmark.runners.base [-] ITER: 9 START
2014-03-20 06:27:07.414 27820 INFO rally.benchmark.runners.base [-] ITER: 8 END: OK
2014-03-20 06:27:13.311 27821 INFO rally.benchmark.runners.base [-] ITER: 9 END: OK
2014-03-20 06:27:14.302 27812 WARNING rally.benchmark.context.secgroup [-] Unable to delete secgroup: 43
2014-03-20 06:27:14.336 27812 WARNING rally.benchmark.context.secgroup [-] Unable to delete secgroup: 45
2014-03-20 06:27:14.336 27812 INFO rally.benchmark.context.cleaner [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Cleanup users resources.
2014-03-20 06:27:25.498 27812 INFO rally.benchmark.context.cleaner [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Cleanup users resources.
2014-03-20 06:27:25.498 27812 INFO rally.benchmark.context.cleaner [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Starting: Cleanup admin resources.
2014-03-20 06:27:25.689 27812 INFO rally.benchmark.context.cleaner [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Cleanup admin resources.
2014-03-20 06:27:26.092 27812 INFO rally.benchmark.engine [-] Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e | Completed: Benchmarking.

================================================================================
Task 87eb8ff3-07f9-4941-b1be-63e707aceb1e is finished.
--------------------------------------------------------------------------------

test scenario NovaServers.boot_and_delete_server
args position 0
args values:
{u'args': {u'flavor_id': 1,
u'image_id': u'976dfd41-d8d5-4688-a8c1-8f196316d8b9'},
u'context': {u'users': {u'tenants': 3, u'users_per_tenant': 2}},
u'runner': {u'concurrency': 2, u'times': 10, u'type': u'continuous'}}
+---------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+---------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| nova.boot_server | 0.480 | 0.501 | 0.521 | 0.521 | 0.521 | 100.0% | 10 |
| nova.delete_server | 0.185 | 0.189 | 0.195 | 0.194 | 0.194 | 70.0% | 10 |
| total | 0.666 | 0.690 | 0.715 | 0.715 | 0.715 | 70.0% | 10 |
+---------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+

HINTS:
* To plot HTML graphics with this data, run:
rally task plot2html 87eb8ff3-07f9-4941-b1be-63e707aceb1e --out output.html

* To get raw JSON output of task results, run:
rally task results 87eb8ff3-07f9-4941-b1be-63e707aceb1e
</pre>

== Available Rally facilities ==

To be able to run complex benchmark scenarios on somewhat more sophisticated OpenStack deployment types, you should familiarize yourself with more '''''deploy engines''''', '''''server providers''''' and '''''benchmark scenarios''''' available in Rally.

* List of available Deploy engines (including their description and usage examples): [[Rally/DeployEngines|Deploy engines]]

* List of available Server providers (including their description and usage examples): [[Rally/ServerProviders|Server providers]]

* List of available Benchmark scenarios (including their description and usage examples): [[Rally/BenchmarkScenarios|Benchmark scenarios]]

You can also learn about different Rally entities without leaving the Command Line Interface. There is a special '''search engine''' embedded into Rally, which, for a given ''search query'', prints documentation for the corresponding benchmark scenario/deploy engine/... as fetched from the source code. This is accomplished by the '''rally info find''' command:

<pre>
$ rally info find create_meter_and_get_stats

CeilometerStats.create_meter_and_get_stats (benchmark scenario).

Test creating a meter and fetching its statistics.

Meter is first created and then statistics is fetched for the same
using GET /v2/meters/(meter_name)/statistics.

Parameters:
- name_length: length of generated (random) part of meter name
- kwargs: contains optional arguments to create a meter

$ rally info find Authenticate

Authenticate (benchmark scenario group).

This class should contain authentication mechanism.

For different types of clients like Keystone.

$ rally info find some_non_existing_benchmark

Failed to find any docs for query: 'some_non_existing_benchmark'
</pre>

Meetings/QATeamMeeting

2014-10-08T20:56:55Z

David-kranz: /* Agenda for October 9th 2014 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib (dkranz)
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2014-10-08T20:56:26Z

David-kranz: /* Agenda for October 9th 2014 (1700 UTC) */

= Weekly QA Team meeting =
The OpenStack QA Team holds public weekly meetings in <code><nowiki>#openstack-meeting</nowiki></code>, '''Thursday alternating every week between 17:00 UTC and 22:00 UTC''. 
Everyone interested in testing, quality assurance, performance engineering, etc, should attend!

The meeting minutes of can be found here: [http://eavesdrop.openstack.org/meetings/qa meeting minutes on everdrop]

''Rules for adding to the agenda'':
* anyone is free to add to proposed agenda, please do so by 1 hr before the meeting
* if you add an item to the agenda, please include your ircnick (i.e. sdague) to the end of your item, so we know who did, and who should lead the discussion

=== Agenda for October 9th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
** 7 slots + 1 day mini-midcycle room shared with infra and rel-mgt
*** http://kilodesignsummit.sched.org/
** Next week's dedicated summit planning
* Specs Review
** It would be nice to get started on https://review.openstack.org/#/c/126004/ as a step towards tempest-lib
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews

== Previous meetings ==
All previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/qa/.

=== Agenda for October 2nd 2014 (2200 UTC) ===
* tempest-2 tag (mtreinish)
* Blueprints
** Status Updates
*** bp [https://blueprints.launchpad.net/tempest/+spec/resource-cleanup resource-cleanup]: 4 migrate_* patches to go, plus hacking rule and drop safe_setup, then first phase is complete. Cleanup of resource_cleanup methods going on in parallel. Phase 2 to start next week.
*** bp [https://blueprints.launchpad.net/tempest/+spec/test-accounts test-accounts]: non-isolated jobs verified ok. Using pre-provisioned credentials still fails because of credentials leaks. Wait on resource-cleanup bp phase one to be done, retest and if ok we can merge
*** bp [https://blueprints.launchpad.net/tempest/+spec/tempest-client-scenarios tempest-client-scenarios]: masayuki's bp - I worked last week on the cleanup patches, and thanks to all the reviews, they are all merged, except for the cleanup of requirements.txt proposed by masayuki.
* DevStack
* Grenade
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/112581/
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 25th 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* DevStack
* Grenade
* Bugs
**https://etherpad.openstack.org/p/Tempest-bug-report
* Critical Reviews
* Summit topic brainstorming
** https://etherpad.openstack.org/p/kilo-qa-summit-topics

=== Agenda for September 18th 2014 (2200 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* tempest-lib commit history tracking (mtreinish)
** need a concrete transition plan for tempest-lib
** already conflicting in-flight changes to code now in tempest-lib like https://review.openstack.org/#/c/121537/6
* Specs Review
* Blueprints
** Status Updates
* DevStack
** https://review.openstack.org/105785 - we should write down the criteria of when we might want to make Neutron DevStack's default network
** https://review.openstack.org/98854/ - SSL all the things - looking for feedback on the priority here
** https://review.openstack.org/111734 - move logging up in stack.sh - is it good to go?
* Grenade
* Bugs
* Critical Reviews

=== Agenda for September 11th 2014 (1700 UTC) ===
* Summit topic brainstorming (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Results of Bug Day
** Went from 124 New to 71. Not so much participation.
** We need more real-time triage. Rotating weekly bug triage?
* Specs Review
* Blueprints
** Status Updates
** First tempest-lib release (mtreinish)
* Devstack
* Grenade
* Bugs
* Critical Reviews
** Move success response checking of Swift to tempest clients https://review.openstack.org/#/c/117193/ (dmorita)
*** Swift's core members say Swift's policy of response code is responding 2xx series on success. They believe they should not define one or two specific code(s) for successful response.
*** On the other hand, Kenichi makes an assertion that OpenStack's policy says a response code should not be changed, so we should make tests more strictly.
*** I can understand both assertions but we have to decide which idea to be chosen.

=== Agenda for September 4th 2014 (2200 UTC) ===
* Summit Session Topics (mtreinish)
** https://etherpad.openstack.org/p/kilo-qa-summit-topics
* Specs Review
* Blueprints
** Status Updates
* Devstack
* Grenade
* Bugs
* Critical Reviews

=== Agenda for August 28 2014 (1700 UTC) ===
* Specs Review
* Blueprints
** Status Updates
* git history with tempest-lib migration (mtreinish)
** First draft lib repo: https://github.com/mtreinish/tempest-lib
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 21 2014 (2200 UTC) ===
* Devstack now part of the QA program
* Starting a QA liaison system (like oslo) (mtreinish)
* Can we start using smoke tag for its intended purpose? (dkranz)
** Except in neutron which still uses smoke in icehouse runs
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for August 14 2014 (1700 UTC) ===
* Name for tempest library (mtreinish)
* Specs Review
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for July 31 2014 (1700 UTC) ===
* Specs Review
* Prioritizing Neutron over Nova-Network
* Blueprints
** Status Updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* No meeting next week

=== Agenda for July 24 2014 (2200 UTC) ===
* Volunteer to run next 2 meetings on July 31st and August 7th (mtreinish)
* Mid-cycle Meetup Summary (mtreinish)
** https://etherpad.openstack.org/p/Qa_Infra_Meetup_2014
* Blueprints
** J-2 Milestone Status: https://launchpad.net/tempest/+milestone/juno-2
* Specs Review
** Functional Testing Library: https://review.openstack.org/#/c/108858/ (mtreinish)
* Now that there is agreement to move api/func testing to projects, should we start that (virtually) now? (dkranz)
** Reclaim smoke tag (except for neutron until neutron-full is enabled everywhere)
** Use smoke to tag a small set of sanity api tests or those with some cross-project implications
** Start gating each project with all scenario plus full api tests of its test directory, and smoke for the others
* Grenade
* Neutron testing
* Bugs
* Critical Reviews
* Proposal of temp review policies (andreaf - I'll try to make it for the meeting, else details below)
** New scenario tests shall use tempest client. Rebase on in-flight bp changes?
** New API tests shall not include return code validation. I think introducing return code validation should be a separate patch as it can be reviewed and tested independently
** New scenario / API neutron tests on hold (no +A) until full job voting on neutron and tempest

=== Agenda for July 10 2014 (2200 UTC) ===
* Spec review day outcomes (mtreinish)
* Mid-cycle Meet-up (mtreinish)
** No qa-meeting next week
* Specs Review
* Blueprints
** Volunteer for: https://blueprints.launchpad.net/tempest/+spec/branchless-tempest-extensions
** Status updates
* Changing scenario tests to Tempest Client - http://lists.openstack.org/pipermail/openstack-dev/2014-July/039879.html (sdague)
* Abstraction of test/client interactions http://lists.openstack.org/pipermail/openstack-dev/2014-July/039927.html (dkranz)
* Grenade
* Neutron testing
** mlavalle attending Neutron mid-cycle sprint this week:
*** Trained two new developers on Tempest using http://www.slideshare.net/MiguelLavalle/tempest-tests-class-hierarchy. One of this developers is now writing an API test for LBaaS V2.0. The other developer is writing scenario test for FWaaS and VPNaaS
*** Updated Tempest client for Neutron to support LBaaS V2.0 API. Patchset: https://review.openstack.org/#/c/106089/
*** Met with team implementing DVR (Distributed Virtual Router) in Neutron. Agreed that for the time being, tempest.scenario.test_network_basic_ops is sufficient to test DVR
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/104290/7 is suffering rebase/recheck hell (dkranz)
*** Please do not approve any patches that change the 'identity' service and api directories until this merges
* Glance tests transitioning to tempest. Would like to ask some questions around best practices, place to add the tests (api/scenario), things to watch our for, etc? ref. https://review.openstack.org/106146

=== Agenda for July 3 2014 (1700 UTC) ===
* Spec review day July 9th (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039105.html
* Mid-cycle Meet-up full, registration closed (mtreinish)
** http://lists.openstack.org/pipermail/openstack-dev/2014-July/039209.html
* Specs Review
** How do we abandon approved specs: https://review.openstack.org/#/c/104576/ (mtreinish)
** https://review.openstack.org/#/c/101232/
* Blueprints
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 26 2014 (2200 UTC) ===
* Juno specs proposal cutoff date? (mtreinish)
* Specs Review
* Blueprints
** Future of https://blueprints.launchpad.net/tempest/+spec/nova-api-test-inheritance ?(mtreinish)
** Status updates
* Grenade
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for June 19 2014 (1700 UTC) ===
* Specs Review
** https://review.openstack.org/94473 (tempest config script)
*** What is going on with this spec? (submitted by boris42)
*** Does any one have code they are working on ahead of this spec being approved?
** https://review.openstack.org/#/c/101232/ (restructure scenario - yfried)
* Blueprints
** Status updates
* Grenade
* Neutron full voting on icehouse and juno (afazekas)
* Neutron testing
** scenario specific gate (yfried)
*** sdague objects to some LBaaS scenario patches due to duration. could we move the scenarios to a separate gate?
* Bugs
* Critical Reviews
** Convert scenario test tearDown to addCleanup https://review.openstack.org/#/c/62101/ (mtreinish & yfried)
** Force connection auth as part of RemoteClient in scenario https://review.openstack.org/#/c/92573/ (yfried)
** Adds Cross Host Scenario https://review.openstack.org/77816 (yfried)

=== Agenda for June 12 2014 (2200 UTC) ===
* Specs Review
** Client response checking https://review.openstack.org/93037 -- What about xml?
*** a) Put response code checks using the response part of the schema for json in all xml clients (quite a bit more work)
*** b) Don't touch xml clients and accept that some (perhaps most) response checking for xml is lost when the current explicit checks in tests are removed
* Blueprints
** Status updates
* Disabling nova v3 API tests by default - http://lists.openstack.org/pipermail/openstack-dev/2014-June/037370.html (mriedem)
* Experimental tag (cyeoh)
** tag for APIs which are thought to be stable, but for one cycle in extreme circumstances may have a breaking backwards incompatible change
*** Nova tasks API is probably an example
*** Experimental tag would indicate that the test is only run against the latest project devel tree
* Neutron testing
* Bugs
* Critical Reviews
* Tempest tests for Barbican
** blueprint - https://blueprints.launchpad.net/tempest/+spec/add-basic-tests-for-barbican
* Open Discussion
** Multi-node testing for migrations with Nova API? What needs to happen? (mriedem)

=== Agenda for June 5 2014 (1700 UTC) ===
* Mid-Cycle Meetup (mtreinish)
** https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014
* Unstable API testing in Tempest (mtreinish)
** https://review.openstack.org/95789
* Specs Review
** https://review.openstack.org/#/c/94473/
** https://review.openstack.org/#/c/97589/ (asselin)
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 29 2014 (2200 UTC) ===
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Bugs
* Critical Reviews

=== Agenda for May 22 2014 (1700 UTC) ===
* Blueprint Purge (mtreinish)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
* Summit follow-up (andreaf)

=== Agenda for May 8 2014 (2200 UTC) ===
* Reminder about summit etherpads
** https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA
* Proposal to move success response checking to clients (dkranz)
* Can we turn on voting of ironic jobs (recent creds change broke it)? (adam_g)
* Specs Review
* Blueprints
** Status updates
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews
** https://review.openstack.org/#/c/92573/ - connection verification as part of RemoteClient init

=== Agenda for April 24 2014 (1700 UTC) ===
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Specs Review
* Blueprints
** Current state
*** SSH stress test - https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip (julien-llp)
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 17 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Oslo Liaison (mtreinish)
** https://wiki.openstack.org/wiki/Oslo/ProjectLiaisons
* Blueprints
** Current state
*** https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test There is a question about whether
the returned dict schemas should set additionalProperties to false. Perhaps they should because adding one is supposed
to require an extension, but they do not in these reviews.
** Specs Review
** Status updates from people
* Neutron testing
** Work has continued in reviewing api tests. 19 patchsets merged out of 28 we are tracking for Icehouse
** Following patchsets require only one more +2 to merge
*** https://review.openstack.org/#/c/67547
*** https://review.openstack.org/#/c/63723
*** https://review.openstack.org/#/c/66541
** During past Monday Neutron weekly IRC meeting, a session during the design summit was proposed. Neutron core developers supported the idea. Session added here by mlavalle: http://summit.openstack.org/cfp/details/366
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 10 2014 (1700 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for April 3 2014 (2200 UTC) ===
* Summit sessions (mtreinish)
** https://etherpad.openstack.org/p/Juno-QA-design-summit-topics
* Blueprints
** Current state
** Specs Review
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/cinder-v2-api-tests(zhikunliu)
****we have already enabled basic cinder v2 tests. Should we port all v1 tests to v2 like nova v2 to v3? Is it the right direction?
* Network Scenarios - call to arms (mlavalle&yfried)
* Tempest branching strategy (dkranz)
** Many distros, and now refstack, lag behind OpenStack releases. How can we enable more work to happen on master after a release?
*** Make sure tempest can run against previous releases as we do with client libraries?
*** Initial result of master against stable/havana nova-network (not including heat/ceilo/swift): http://paste.openstack.org/show/74966/
*** Whole transcript with error stacks: http://paste.openstack.org/show/74990/ 148 failures out of 2009
*** Etherpad to track these issues here: https://etherpad.openstack.org/p/master-against-havana-errors
* Critical Reviews

=== Agenda for March 27 2014 (1700 UTC) ===
* QA Specs Repo Proposals (sdague)
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
* Critical Reviews

=== Agenda for March 20 2014 (2200 UTC) ===
* Blueprints
** QA Specs repository
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Bug day review (maurosr)
* Critical Reviews

=== Agenda for March 13 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Running tempest as non-admin (dkranz)
* qa-specs gerrit repository

=== Agenda for March 6 2014 (2200 UTC) ===
* Finding a volunteer to organize a bug day
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Heat testing
* Bugs
** Critical / High bug review
* Critical Reviews
* Strategy for creating schemas for negative auto-gen (dkranz)

=== Agenda for February 27 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews
* What is our strategy for getting fail on log errors turned back on? (dkranz)

=== Agenda for February 20 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/neutron-advanced-scenarios,n,z (yfried)
**** https://review.openstack.org/#/c/73973/ (Minor chages)
**** https://review.openstack.org/#/c/69567/ (Refactor cross_tenant)
**** https://review.openstack.org/#/c/66879/ (Refactor network_basic_ops)
**** https://review.openstack.org/#/c/52994/ (Prevent overlapping CIDRs)
**** https://review.openstack.org/#/c/69361/ (hotplug nic)
*** https://review.openstack.org/#/q/project:openstack/tempest+branch:master+topic:bp/multi-keystone-api-version-tests,n,z (andreaf)
**** https://review.openstack.org/#/c/74699/ (Fixes to auth.py - by maurosr)
**** https://review.openstack.org/#/c/68140/ (Auth unit tests - by maurosr)
**** https://review.openstack.org/#/c/73704/ (Refactor manager base class)
**** https://review.openstack.org/#/c/74387/ (Introduce Credentials class - pending unit tests)
* Neutron testing
** mlavalle won't be able to attend this meeting. This is the update:
*** api test has continued over the past few days, with 14 active contributors
*** Message was sent to the ML with a list of all the api tests patchsets to facilitate review by the neutron and Tempest core developers
**** Already seeing Tempest core reviewers approving and merging code. Thanks! Let's keep it up
* Bugs
** Critical / High bug review
* Critical Reviews
* Other
** Service dependency decorators in tests (andreaf)

=== Agenda for February 13 2014 (1700 UTC) ===
* Removing nose support (mtreinish)
* Blueprints
** Current state
** Proposal
*** Fuzzy testing framework (mkoderer)
** Nova v3 testing in scenario tests (andreaf)
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/make-tempest-pluggable (vponomaryov)
**** There are 2 commits ready for review, that are related to bp:
**** https://review.openstack.org/#/c/62429/ (exceptions)
**** https://review.openstack.org/#/c/69985/ (config)
**** Need to decide, is it acceptable? Its pros and cons?
* Neutron testing
* API tests in the Neutron tree (marun)
** https://review.openstack.org/#/c/72585/ (api test in the neutron tree)
** https://review.openstack.org/#/c/72588/ (api test run with tempest rest clients)
* Graduation requirements for incubated project w.r.t Tempest (malini)
**From TC ' Project must have a basic devstack-gate job set up' - what constitutes a basic devstack-gate ?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for February 6 2014 (2200 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* When can we enable failing jobs with bogus log ERRORs (dkranz)
* Criteria for accepting tests that cannot run normally in the gate (dkranz)
** multi-node, require configuration not present in a current gate job
** What precisely is the policy for accepting in tempest? Is running nightly downstream acceptable?
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 30 2014 (1700 UTC) ===
* Blueprints
** Current state
** Proposal
** Status updates from people
* Neutron testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 23 2014 (2200 UTC) ===
* Gate Updates (sdague)
** Current State of Things
** New Gating model - http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html
* Blueprints
** Current state
** Proposal
** Status updates from people
*** https://blueprints.launchpad.net/tempest/+spec/negative-tests
**** Any major comments/issues?
**** Do we really need to support xml in the future?
**** What is needed to get this in?
**** Should schemas go in classes as in the patch, or elsewhere?
* Neutron testing
** mlavalle to report latest status of api testing
* Bugs
** Critical / High bug review
* Critical Reviews

=== Agenda for January 16 2014 (1700 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
*** Config verification status/discussion (rahmu)
** Proposal
*** Start using Six to prepare for an eventual Python 3 migration (rahmu)
** Status updates from people
* Bugs
** Critical / High bug review
* Neutron testing (mlavalle)
* Marconi testing (malini)
**https://review.openstack.org/#/q/topic:bp/add-basic-marconi-tests,n,z
* Ironic testing (agordeev)
** https://etherpad.openstack.org/p/IronicDevstackTesting
** related [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024332.html discussion in openstack-dev], [http://lists.openstack.org/pipermail/openstack-dev/2014-January/024273.html initial message]
* Critical Reviews

=== Agenda for January 9 2014 (2200 UTC) ===
* Blueprints
** Current state
*** Negative test status/discussion (dkranz)
** Status updates from people
** Reconsideration of https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client, according to changes (vponomaryov):
*** https://review.openstack.org/#/c/62923/
*** https://review.openstack.org/#/c/64948/
* Bugs
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing (mlavalle)
** Gap analysis for api tests is complete
*** Approach taken was very detailed and assumed that, to the extent possible, we want to test every attribute, as many options as possible and every operation
*** Negative tests were removed from the gap analysis, in light of "generative" tool under development in Tempest
*** email was sent on 1/6 to the ML to remind the community about this effort and provide an update
*** Seeing people assigning themselves tests from the gap analysis, but not as many as desirable. mlavalle has started assigning himself tests from the list. We can also reduce the gap next week during the sprint in Montreal
** Progress in creating new stable Tempest tests
*** Recently merged
**** https://review.openstack.org/#/c/59029/ rossella_s (api xml client)
**** https://review.openstack.org/#/c/61529/ rossella_s (api floating ip's)
**** https://review.openstack.org/#/c/62614/ enikanorov (api refactor network client)
**** https://review.openstack.org/#/c/55133/ mouad benchchaoui (api)
*** In review cycle:
**** https://review.openstack.org/#/c/63723/ Ann Kamyshnikova (api floating ip)
**** https://review.openstack.org/#/c/61118/ Ann Kamyshnikova (api lbaas)
**** https://review.openstack.org/#/c/63627/ yfried (scenario floating ip's)
**** https://review.openstack.org/#/c/58697/ Elena Ezhova (scenario lbaas)
**** https://review.openstack.org/#/c/65120/ Miguel Lavalle (lbaas agent scheduler)
**** https://review.openstack.org/#/c/65120/ yfried (api fwaas)
**** https://review.openstack.org/#/c/55146/ yfried (scenario network connectivity)
**** https://review.openstack.org/#/c/62962/ salv-orlando (make router creation for tenant isolation optional)
**** https://review.openstack.org/#/c/60008/ Evgeny Fedoruk (Extending quota support for neutron LBaaS entities)
**** https://review.openstack.org/#/c/50542/ ChenZheng (Test for the neutron api with provider extension)
**** https://review.openstack.org/#/c/63625/ QianLin
**** https://review.openstack.org/#/c/63999/2 Ann Kamyshnikova (external network extension)
**** https://review.openstack.org/#/c/64271 Elena Ezhova (api binding extended attributes for ports)
* Critical Reviews

=== Agenda for December 19 2013 (1700 UTC) ===
* Cancelling meetings for the rest of the year (mtreinish)
* Blueprints
** Current state
** Status updates from people
** Needing some attention (gfidente)
*** https://blueprints.launchpad.net/tempest/+spec/fail-gate-on-log-errors (close or not?)
*** https://blueprints.launchpad.net/tempest/+spec/quantum-basic-api (seems to me pretty hard to track/implement, yet useful)
*** https://blueprints.launchpad.net/tempest/+spec/refactor-rest-client (there is actual work here, approve?)
*** https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api (approve/low?)
*** https://blueprints.launchpad.net/tempest/+spec/multi-keystone-api-version-tests (maybe we should just use latest?)
*** https://blueprints.launchpad.net/tempest/+spec/swift-test-without-keystone (is it a good idea at all?)
* Bugs (sdague)
** Critical / High bug review
** Closing invalid Tempest bugs
* Neutron testing
* Bug status
* On new tests tracking (plus and cons of wiki/etherpad/blueprint) (gfidente)
* Critical Reviews

=== Agenda for December 12 2013 (2200 UTC) ===

* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Report on https://bugs.launchpad.net/neutron/+bug/1251448 (sixth top bug in the gate this week) and its relationship to https://bugs.launchpad.net/tempest/+bug/1259282 (mlavalle)
* Bug status
** How should we handle bug traffic after we finish the current purge to make sure the noise doesn't come back?
** Rotating dispatcher for incoming bugs?
** Ask people to put a logstash query in bug tickets that are about test failures?
** https://etherpad.openstack.org/p/tempest-bug-triage
* Critical Reviews
* We should consider putting a fast-track on heat and ceilometer tests since they are integrated but lacking. (dkranz)
* Should we have more specialization on the Core review team? e.g. I am comfortable with the nova v3 patches, but no where on neutron. (dkranz)

=== Agenda for December 5 2013 ===
* Meeting time change proposal
* Getting additional folks signed up as point people
** Meeting organizer - mtreinish voluteered
** Blueprint Triage point person - need volunteer
** Bug Triage point person - need volunteer
* Blueprints
** Current state
** Status updates from people
* Neutron testing
** Progress on API tests gap analysis in https://etherpad.openstack.org/p/icehouse-summit-qa-neutron (mlavalle)
** Wiki page for new comers on how to contribute Neutron API tests (mlavalle)
* Critical Reviews
* How can we make sure reviews don't slide >1 week (dkranz)

=== Agenda for November 21 2013 ===
* Blueprints
* Negative testing (mkoderer/dkranz)
* Neutron testing: Discussion of what needs to happen for the sprint in January to take place / succeed? What do we need to accomplish by mid-December to make sure the sprint happens? (mlavalle)
* Critical Reviews

=== Agenda for November 14 2013 ===
* Possible Meeting Time change to include .jp (sdague)
* Blueprints
** Blueprints breakdown (sdague)
** Negative testing (mkoderer/dkranz)
* Neutron testing
** Negative tests coverage: neutron status (EmilienM)
** Progress on https://bugs.launchpad.net/swift/+bug/1224001 (mlavalle)
* How should we sync the teams (neutron/scenario test)? One BP with work items, bugs or ML? (mkoderer)
* Onboarding documentation - where should we put it? (mkoderer)
* Critical Reviews (sdague)

=== Agenda for October 24 2013 ===
* Design Summit Schedule (sdague)
** http://icehousedesignsummit.sched.org/ - figure out if there are last minute changes we need (note: many time blocks are fixed due to conflicts, so wiggle room is limited)
* Neutron job status (mtreinish)
* Scope and place for performance testing such as Rally (dkranz)
* Status and roll-out plan for failing the gate on log errors (dkranz)
* State of 'smoke' tagging: can we make it useful? (dkranz)
* Tempest config reorg and cleanup (mtreinish)

=== Agenda for October 17 2013 ===
* Unblocking of Ceilometer QA testing (jd__, sileht)
** https://review.openstack.org/#/c/49394/

=== Agenda for October 10 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning
* Lot's of new negative tests. Do we want that? (dkranz)
** some are "validating strings", like passing long strings or checking for invalid vs. non existent uuids (gfidente)
** some are "traversing" the components, like ensuring we can't attach resources to non existent servers (gfidente)
** I've the feeling they should not be tagged as 'smoke' ... and maybe a lot not even 'gate'?
* Rollout plan for failing builds that pass tempest but with spurious log ERRORs (dkranz)
** https://review.openstack.org/#/c/50795/

=== Agenda for October 3 2013 ===
* Blueprints (sdague)
** started cleaning up blueprints, need to start a purge of all the Unknown items
* Neutron job status (sdague)
* Elastic Recheck Top issues (mtreinish)
* Stable branch timing (sdague)
* Design Summit Initial Planning (sdague)
** QA sessions needed at DS - https://etherpad.openstack.org/icehouse-qa-session-planning

=== Agenda for September 26 2013 ===
* Blueprints (sdague)
** neutron testing status (mlavalle)
** Bogus errors in logs (dkranz)
* rework "skip test" functionality, reprises mkoderer's proposal (gfidente)
* Critical Reviews (sdague)
=== Agenda for September 19 2013 ===
* Blueprints (mtreinish)
** neutron testing status (mlavalle)
* Critical Reviews (mtreinish)
* How to handle bug fixes in launchpad (mtreinish)
* Bogus errors in logs one more time (dkranz)

=== Agenda for September 12 2013 ===
* Leader for next week's meeting - sdague on a plane back from LinuxCon during this time
* New core members (sdague)
* Summit planning (sdague)
** my intent is to wait until October to evaluate the agenda we want, however as ideas come forward now feel free to register
* Blueprints (sdague)
* neutron testing status (mlavalle)
* whitebox test removal (sdague)
** need a volunteer to ensure these tests are covered in Nova unit tests
* Critical Reviews (sdague)
* OpenStack qa repo for test plans that can be pointed at from blueprints (mkollaro)

=== Agenda for August 29 2013 ===

* Blueprints (mtreinish)
** testr status (mtreinish)
*** Do we need a way to "incubate" new tests to avoid flakiness with parallel testr? (dkranz)
*** Now that parallel testr is in place (wow) maybe we should organize another bug triage day? (gfidente)
** stress tests (mkoderer)
** neutron testing status (mlavalle)
* Update on slow heat gating job (dkranz)
* Critical Reviews (mtreinish)
* Testing client library compatibility (dkranz)
* Devstack independent tempest usage (afazekas)
* BP proposal: rework "skip test" functionaly (mkoderer)

=== Agenda for August 22 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)
* Heat test merge (afazekas)
* Devstack independent tempest usage (afazekas)
* Adding test plans for new features into the tempest repo (mkollaro)

=== Agenda for August 15 2013 ===
* Blueprints (mtreinish)
** testr parallel status (mtreinish)
** stress tests (mkoderer)
* Critical Reviews (mtreinish)

=== Agenda for August 8 2013 ===
* Volunteers to run meeting while sdague out (sdague)
* Blueprints (sdague)
* Critical Reviews (sdague)
* Stress tests scope and behaviour, also see reviews on https://review.openstack.org/#/c/39752 (gfidente)
* Status of slow heat tests (dkranz)

=== Agenda of August 1 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-08-01-16.59.html meeting minutes for July 1 on eavesdrop].''

* Blueprints (sdague)
* Critical Reviews (sdague)
* Call for more Core reviewers (dkranz)

=== Agenda of July 25 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-25-17.00.html meeting minutes for July 25 on eavesdrop].''

* Blueprints (sdague)
** Current state of implementation
** Stress Tests https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* WebDav status codes in nova? Consistently using the 404 or 422 on actions.
* Adding test cases with skip attribute vote? Exact rule (afazekas)
* py26 compatibility (afazekas)
* Consistent reviewing (afazekas)
* Critical Reviews (sdague)
* Separate heat job and slow tests in general (dkranz)

===Agenda of July 18 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-18-17.00.html meeting minutes for July 18 on eavesdrop].''

* Blueprints (sdague)
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2 (sdague .. et al)
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest (mtreinish)
** Stress Tests (new Blueprint and findings) - https://blueprints.launchpad.net/tempest/+spec/stress-tests (mkoderer)
* Mailing list move (voting on moving to -dev) (sdague)
* White Box Tests (owner for discussion?)
** Should they really be unit tests?
** Current state of implementation
** Integration in CI
** Adding test cases with skip attribute (afazekas)
* Critical Reviews (sdague)

=== Agenda of July 11 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-07-11-17.03.html meeting minutes for July 11 on eavesdrop].''

* QA Program Business
** PTL Nominations close - see if we need to run election
** Program Description
* Blueprints
** Havana-2 status check in - https://launchpad.net/tempest/+milestone/havana-2
** Testr Progress - https://blueprints.launchpad.net/tempest/+spec/speed-up-tempest
* Critical Reviews
* Stress Tests in CI
* Full Open Stack networking gate job
* New stress tests in tempest --[[User:M-koderer|m-koderer]] ([[User talk:M-koderer|talk]]) 16:36, 11 July 2013 (UTC)

=== Agenda of June 27 2013 ===
''See the [http://eavesdrop.openstack.org/meetings/qa/2013/qa.2013-06-27-17.01.html meeting minutes for June 27 on eavesdrop].''

* ''nothing in Agenda ... only QA as OpenStack Program, QA Leader and testr topics were discussed little bit''

Meetings/QATeamMeeting

2014-10-02T20:05:59Z

David-kranz: /* Proposed Agenda for October 2nd 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-09-18T17:20:29Z

David-kranz: /* Proposed Agenda for September 18th 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-09-11T16:43:05Z

David-kranz: /* Proposed Agenda for September 11th 2014 (1700 UTC) */

Meetings/QATeamMeeting

2014-08-21T13:24:47Z

David-kranz: /* Proposed Agenda for August 21 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-08-21T13:20:23Z

David-kranz: /* Proposed Agenda for August 21 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-07-29T20:40:05Z

David-kranz: /* Proposed Agenda for July 31 2014 (1700 UTC) */

Meetings/QATeamMeeting

2014-07-24T19:01:01Z

David-kranz: /* Proposed Agenda for July 24 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-07-24T16:45:31Z

David-kranz: /* Proposed Agenda for July 24 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-07-10T19:22:31Z

David-kranz: /* Agenda for July 10 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-07-10T17:13:01Z

David-kranz: /* Agenda for July 10 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-06-19T14:18:36Z

David-kranz: /* Proposed Agenda for June 19 2014 (1700 UTC) */

Meetings/QATeamMeeting

2014-06-12T19:32:44Z

David-kranz: /* Proposed Agenda for June 12 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-06-12T16:56:33Z

David-kranz: /* Proposed Agenda for June 12 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-06-04T15:22:05Z

David-kranz: /* Proposed Agenda for June 5 2014 (1700 UTC) */

Meetings/QATeamMeeting

2014-05-08T19:23:23Z

David-kranz: /* Proposed Agenda for May 8 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-05-08T13:52:27Z

David-kranz: /* Proposed Agenda for May 8 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-04-16T21:14:36Z

David-kranz: /* Proposed Agenda for April 17 2014 (2200 UTC) */

Meetings/QATeamMeeting

2014-04-03T18:39:23Z