NeutronGateFailureTriage

This page provides guidelines for spotting and assessing neutron gate failures. Some hints for triaging failures are also provided.

Spotting gate failures

This can be achieved using several tools.

1. Joe Gordon's github.io pages
2. logstash.openstack.org

Even though Joe's script is not an "official" Openstack page it provides a quick snapshot of the current status for the most important jobs This page is built using data available at graphite.openstack.org. If you want to check how that is done go here: https://github.com/jogo/jogo.github.io/tree/master/gate (caveat: the color of the neutron job is very similar to that of the full job with nova-network)

For checking gate failures with logstash the following query will return failures for a specific job:

 build_status:FAILURE AND message:Finished  AND build_name:"check-tempest-dsvm-neutron" AND build_queue:"gate"

And divided by the total number of jobs executed:

 message:Finished  AND build_name:"check-tempest-dsvm-neutron" AND build_queue:"gate"

it will return the failure rate in the selected period for a given job. It is important to remark that failures in the check queue might be misleading as the problem causing the failure is most of the time in the patch being checked. Therefore it is always advisable to work on failures occurred in the gate queue. However, these failures are a precious resource for assessing frequency and determining root cause of failures which manifest in the gate queue.

The step above will provide a quick outlook of where things stand. When the failure rate raises above 10% for a job in 24 hours, it's time to be on alert. 25% is amber alert. 33% is red alert. Anything above 50% means that probably somebody from the infra team has already a contract out on you. Whether you are relaxed, in alert mode, or freaking out because you see a red dot on your chest, it is always a good idea to check status.openstack.org/elastic-recheck