Jump to: navigation, search

GuruMeditationReport

Guru Meditation Reports

When things go wrong in (production) deployments of OpenStack collecting debug data is a key first step in the process of triaging & ultimately resolving the problem. Nova has extensively used logging capabilities which produce a vast amount of data. This does not, however, enable an admin to obtain an accurate view on the current live state of the system. For example, what threads are running, what config parameters are in effect, and more. The eventlet backdoor facility provides an interactive shell interface for any eventlet based process, allowing an admin to telnet to a pre-defined port and execute a variety of commands. This can be used to collect the necessary state information, but is has a number of limitations.

  • Every service running on a host needs to have the backdoor running on a different TCP port and the admin has to remember which process is listening where. Get this wrong and very bad things can happen.
  • The backdoor needs to have been enabled when the process was started. If this was not done before the problem arose, the admin is out of luck because restarting the service to enable the backdoor will loose the critical state that was desired.
  • The backdoor shell is too powerful. By presenting an interactive python shell too much burden is placed on the admin to find the right data, without causing problems.
  • The backdoor shell is an effective "root shell" available over unsecured TCP port, so no sensible admin will ever enable it on production systems.

Error Report Framework

To address the issues described above, this page outlines the design of a general purpose error report generation framework, known as the "guru meditation report" (cf http://en.wikipedia.org/wiki/Guru_Meditation).

  • Models: These classes define structured data for a variety of interesting pieces of state. For example, stack traces, threads, config parameters, package version info, etc. They are capable of being serialized to XML / JSON or a plain text representation
  • Generators: These classes are used to populate the model classes with the current runtime state of the system

There will be a number of standard models / generators available for all OpenStack services

  • StackTraceModel: a base class for any model which includes a stack trace
  • ThreadModel: a class for information about a thread
  • ExceptionModel: a class for information about a caught exception
  • ConfigModel: a class for information about configuration file settings
  • PackageModel: a class for information about vendor/product/version/package information

Each OpenStack project will have the ability to register further generator classes to provide custom project specific data.

Integration with apps

Every long running service process should have a call to install a signal handler which will trigger the guru meditation framework upon receipt of SIGUSR1. This will result in the process dumping a complete report of its current state to stderr.

For processes which deal with RPC processes, it may also be desirable to install some kind of hook in the RPC request dispatcher that will save a guru meditation report whenever the processing of a request results in an uncaught exception. It could save these reports to a well known directory (/var/log/openstack/<project>/<service>/) for later analysis by the sysadmin or automated bug analysis tools.

Example Report

========================================================================
====                     Guru meditation report                     ====
========================================================================
UUID: 6d4faf99-480c-4d0c-90be-b7d090ecd83e
Time: Mon, 18 Feb 2013 18:01:11 +0000
========================================================================
====                             Config                             ====
========================================================================
DEFAULT:
  allow_resize_to_same_host=True
  allow_same_net_traffic=True
  allowed_direct_url_schemes=[]
  allowed_rpc_exception_modules=['nova.openstack.common.exception', 'nova.exception', 'cinder.exception', 'exceptions']
  api_paste_config=/etc/nova/api-paste.ini
  api_rate_limit=True
  auth_strategy=keystone
  auto_assign_floating_ip=False
  backdoor_port=None
  bandwidth_poll_interval=600
  base_dir_name=_base
  bindir=/home/berrange/src/cloud/nova/bin
  block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_NON_SHARED_INC
  boot_script_template=/home/berrange/src/cloud/nova/nova/cloudpipe/bootscript.template
  ca_file=cacert.pem
  ca_path=/home/berrange/src/cloud/data/nova/CA
  cert_manager=nova.cert.manager.CertManager
  checksum_base_images=False
  checksum_interval_seconds=3600
  cinder_api_insecure=False
  cinder_catalog_info=volume:cinder:publicURL
  cinder_cross_az_attach=True
  cinder_endpoint_template=None
  cinder_http_retries=3
  cnt_vpn_clients=0
  compute_api_class=nova.compute.api.API
  compute_driver=libvirt.LibvirtDriver
  compute_manager=nova.compute.manager.ComputeManager
  compute_stats_class=nova.compute.stats.Stats
  compute_topic=compute
  config_dir=None
  config_drive_format=iso9660
  config_drive_skip_versions=1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01
  config_drive_tempdir=None
  config_file=['/etc/nova/nova.conf']
  console_host=mustard.gsslab.fab.redhat.com
  console_manager=nova.console.manager.ConsoleProxyManager
  console_topic=console
  consoleauth_topic=consoleauth
  control_exchange=nova
  create_unique_mac_address_attempts=5
  crl_file=crl.pem
  db_backend=sqlalchemy
  db_driver=nova.db
  debug=True
  default_access_ip_network_name=None
  default_availability_zone=nova
  default_ephemeral_format=None
  default_floating_pool=nova
  default_instance_type=m1.small
  default_log_levels=['amqplib=WARN', 'sqlalchemy=WARN', 'boto=WARN', 'suds=INFO', 'keystone=INFO', 'eventlet.wsgi.server=WARN']
  default_notification_level=INFO
  default_publisher_id=mustard.gsslab.fab.redhat.com
  default_schedule_zone=None
  defer_iptables_apply=False
  dhcp_domain=novalocal
  dhcp_lease_time=120
  dhcpbridge=/home/berrange/src/cloud/nova/bin/nova-dhcpbridge
  dhcpbridge_flagfile=/etc/nova/nova.conf
  disable_process_locking=False
  dmz_cidr=[]
  dmz_mask=255.255.255.0
  dmz_net=10.0.0.0
  dns_server=[]
  dns_update_periodic_interval=-1
  dnsmasq_config_file=
  ec2_listen=0.0.0.0
  ec2_listen_port=8773
  ec2_private_dns_show_ip=False
  ec2_strict_validation=True
  ec2_timestamp_expiry=300
  ec2_workers=None
  enable_new_services=True
  enabled_apis=['ec2', 'osapi_compute', 'metadata']
  enabled_ssl_apis=[]
  fake_call=False
  fake_network=False
  fake_rabbit=False
  fatal_deprecations=False
  fatal_exception_format_errors=False
  firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
  fixed_ip_disassociate_timeout=600
  fixed_range=10.0.0.0/24
  fixed_range_v6=fd00::/48
  flat_injected=False
  flat_interface=eth0
  flat_network_bridge=br100
  flat_network_dns=8.8.4.4
  floating_ip_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver
  force_config_drive=None
  force_dhcp_release=True
  force_raw_images=True
  forward_bridge_interface=['all']
  gateway=None
  gateway_v6=None
  glance_api_insecure=False
  glance_api_servers=['10.33.8.112:9292']
  glance_host=10.33.8.112
  glance_num_retries=0
  glance_port=9292
  glance_protocol=http
  glusterfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt
  heal_instance_info_cache_interval=60
  host=mustard.gsslab.fab.redhat.com
  host_state_interval=120
  image_cache_manager_interval=2400
  image_info_filename_pattern=/home/berrange/src/cloud/data/nova/instances/_base/%(image)s.info
  injected_network_template=/home/berrange/src/cloud/nova/nova/virt/interfaces.template
  instance_build_timeout=0
  instance_dns_domain=
  instance_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver
  instance_format=[instance: %(uuid)s] 
  instance_name_template=instance-%08x
  instance_usage_audit=False
  instance_usage_audit_period=month
  instance_uuid_format=[instance: %(uuid)s] 
  instances_path=/home/berrange/src/cloud/data/nova/instances
  internal_service_availability_zone=internal
  iptables_bottom_regex=
  iptables_top_regex=
  ipv6_backend=rfc2462
  key_file=private/cakey.pem
  keys_path=/home/berrange/src/cloud/data/nova/keys
  keystone_ec2_url=http://localhost:5000/v2.0/ec2tokens
  l3_lib=nova.network.l3.LinuxNetL3
  libvirt_cpu_mode=none
  libvirt_cpu_model=None
  libvirt_disk_prefix=None
  libvirt_images_type=default
  libvirt_images_volume_group=None
  libvirt_inject_key=True
  libvirt_inject_partition=1
  libvirt_inject_password=False
  libvirt_lvm_snapshot_size=1000
  libvirt_nonblocking=True
  libvirt_ovs_bridge=br-int
  libvirt_snapshot_compression=False
  libvirt_snapshots_directory=/home/berrange/src/cloud/data/nova/instances/snapshots
  libvirt_sparse_logical_volumes=False
  libvirt_type=kvm
  libvirt_uri=
  libvirt_use_virtio_for_bridges=True
  libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
  libvirt_volume_drivers=['iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver', 'local=nova.virt.libvirt.volume.LibvirtVolumeDriver', 'fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver', 'rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'nfs=nova.virt.libvirt.volume.LibvirtNFSVolumeDriver', 'aoe=nova.virt.libvirt.volume.LibvirtAOEVolumeDriver', 'glusterfs=nova.virt.libvirt.volume.LibvirtGlusterfsVolumeDriver']
  libvirt_wait_soft_reboot_seconds=120
  linuxnet_interface_driver=nova.network.linux_net.LinuxBridgeInterfaceDriver
  linuxnet_ovs_integration_bridge=br-int
  live_migration_bandwidth=0
  live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER
  live_migration_retry_count=30
  live_migration_uri=qemu+tcp://%s/system
  lock_path=/home/berrange/src/cloud/data/nova
  lockout_attempts=5
  lockout_minutes=15
  lockout_window=15
  log_config=None
  log_date_format=%Y-%m-%d %H:%M:%S
  log_dir=None
  log_file=None
  log_format=%(asctime)s %(levelname)8s [%(name)s] %(message)s
  logfile_mode=0644
  logging_context_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s%(color)s] %(instance)s%(color)s%(message)s
  logging_debug_format_suffix=from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d
  logging_default_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
  logging_exception_prefix=%(color)s%(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s
  max_age=0
  memcached_servers=None
  metadata_host=10.33.8.112
  metadata_listen=0.0.0.0
  metadata_listen_port=8775
  metadata_manager=nova.api.manager.MetadataManager
  metadata_port=8775
  metadata_workers=None
  mkisofs_cmd=genisoimage
  monkey_patch=False
  monkey_patch_modules=['nova.api.ec2.cloud:nova.openstack.common.notifier.api.notify_decorator', 'nova.compute.api:nova.openstack.common.notifier.api.notify_decorator']
  multi_host=False
  my_ip=10.33.8.112
  network_api_class=nova.network.api.API
  network_device_mtu=None
  network_driver=nova.network.linux_net
  network_manager=nova.network.manager.FlatDHCPManager
  network_size=256
  network_topic=network
  networks_path=/home/berrange/src/cloud/data/nova/networks
  nfs_mount_options=None
  nfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt
  non_inheritable_image_properties=['cache_in_nova', 'bittorrent']
  notification_driver=[]
  notify_api_faults=False
  notify_on_any_change=False
  notify_on_state_change=None
  novncproxy_base_url=http://10.33.8.112:6080/vnc_auto.html
  null_kernel=nokernel
  num_aoe_discover_tries=3
  num_iscsi_scan_tries=3
  num_networks=1
  os_region_name=None
  osapi_compute_listen=0.0.0.0
  osapi_compute_listen_port=8774
  osapi_compute_workers=None
  password_length=12
  periodic_enable=True
  periodic_fuzzy_delay=60
  policy_default_rule=default
  policy_file=policy.json
  project_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=project-ca-%.16s-%s
  public_interface=br100
  publish_errors=False
  pybasedir=/home/berrange/src/cloud/nova
  qpid_heartbeat=60
  qpid_hostname=localhost
  qpid_hosts=['localhost:5672']
  qpid_password=
  qpid_port=5672
  qpid_protocol=tcp
  qpid_sasl_mechanisms=
  qpid_tcp_nodelay=True
  qpid_username=
  quota_cores=20
  quota_driver=nova.quota.DbQuotaDriver
  quota_floating_ips=10
  quota_injected_file_content_bytes=10240
  quota_injected_file_path_bytes=255
  quota_injected_files=5
  quota_instances=10
  quota_key_pairs=100
  quota_metadata_items=128
  quota_ram=51200
  quota_security_group_rules=20
  quota_security_groups=10
  rbd_secret_uuid=None
  rbd_user=None
  reboot_timeout=0
  reclaim_instance_interval=0
  remove_unused_base_images=True
  remove_unused_original_minimum_age_seconds=86400
  remove_unused_resized_minimum_age_seconds=3600
  report_interval=10
  rescue_image_id=None
  rescue_kernel_id=None
  rescue_ramdisk_id=None
  rescue_timeout=0
  reservation_expire=86400
  reserved_host_disk_mb=0
  reserved_host_memory_mb=512
  resize_confirm_window=0
  resume_guests_state_on_host_boot=False
  rootwrap_config=/etc/nova/rootwrap.conf
  routing_source_ip=10.33.8.112
  rpc_backend=nova.openstack.common.rpc.impl_qpid
  rpc_cast_timeout=30
  rpc_conn_pool_size=30
  rpc_response_timeout=60
  rpc_thread_pool_size=64
  run_external_periodic_tasks=True
  running_deleted_instance_action=log
  running_deleted_instance_poll_interval=1800
  running_deleted_instance_timeout=0
  scheduler_manager=nova.scheduler.manager.SchedulerManager
  scheduler_topic=scheduler
  security_group_api=nova.compute.api.SecurityGroupAPI
  security_group_handler=nova.network.sg.NullSecurityGroupHandler
  send_arp_for_ha=False
  send_arp_for_ha_count=3
  service_down_time=60
  servicegroup_driver=db
  share_dhcp_address=False
  snapshot_image_format=None
  snapshot_name_template=snapshot-%s
  sql_connection=mysql://root:123456@localhost/nova?charset=utf8
  sql_connection_debug=0
  sql_connection_trace=False
  sql_dbpool_enable=False
  sql_idle_timeout=3600
  sql_max_overflow=None
  sql_max_pool_size=5
  sql_max_retries=10
  sql_min_pool_size=1
  sql_retry_interval=10
  sqlite_db=nova.sqlite
  sqlite_synchronous=True
  ssl_ca_file=None
  ssl_cert_file=None
  ssl_key_file=None
  state_path=/home/berrange/src/cloud/data/nova
  syslog_log_facility=LOG_USER
  tcp_keepidle=600
  teardown_unused_network_gateway=False
  tempdir=None
  until_refresh=0
  update_dns_entries=False
  use_cow_images=True
  use_forwarded_for=False
  use_ipv6=False
  use_network_dns_servers=False
  use_project_ca=False
  use_single_default_gateway=False
  use_stderr=True
  use_syslog=False
  use_usb_tablet=True
  user_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=%.16s-%.16s-%s
  verbose=True
  virt_mkfs=['default=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'linux=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'windows=mkfs.ntfs --force --fast --label %(fs_label)s %(target)s']
  vlan_interface=eth0
  vlan_start=100
  vnc_enabled=True
  vnc_keymap=en-us
  vncserver_listen=127.0.0.1
  vncserver_proxyclient_address=127.0.0.1
  volume_api_class=nova.volume.cinder.API
  volume_usage_poll_interval=0
  vpn_image_id=0
  vpn_instance_type=m1.tiny
  vpn_ip=10.33.8.112
  vpn_key_suffix=-vpn
  vpn_start=1000
  wsgi_log_format=%(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f
  xen_hvmloader_path=/usr/lib/xen/boot/hvmloader
  xvpvncproxy_base_url=http://10.33.8.112:6081/console
cells:
  call_timeout=60
  capabilities=['hypervisor=xenserver;kvm', 'os=linux;windows']
  enable=False
  manager=nova.cells.manager.CellsManager
  name=nova
  topic=cells
conductor:
  manager=nova.conductor.manager.ConductorManager
  topic=conductor
  use_local=False
spice:
  agent_enabled=True
  enabled=False
  html5proxy_base_url=http://10.33.8.112:6082/spice_auto.html
  keymap=en-us
  server_listen=127.0.0.1
  server_proxyclient_address=127.0.0.1
========================================================================
====                         Native threads                         ====
========================================================================
Thread ID 139989669103360
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139989677496064
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990096934656
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990667343616
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990063363840
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990692521728
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990088541952
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990071756544
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990675736320
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990650558208
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990080149248
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990700914432
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990781155072
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139989685888768
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139989694281472
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990684129024
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139991167133504
  /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:81 in get_model
    None
  /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:52 in create_report
    None
  /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:42 in create
    None
  /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:78 in dump
    None
  /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:85 in __handle_signal
    None
  /usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py:61 in do_poll
    return self.poll.poll(seconds)
  /usr/lib/python2.7/site-packages/eventlet/hubs/poll.py:84 in wait
    presult = self.do_poll(seconds)
  /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:226 in run
    self.wait(sleep_time)

Thread ID 139990658950912
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990789547776
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990046578432
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()

Thread ID 139990054971136
  /usr/lib64/python2.7/threading.py:243 in wait
    waiter.acquire()
  /usr/lib64/python2.7/Queue.py:168 in get
    self.not_empty.wait()
  /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
    msg = reqq.get()
  /usr/lib64/python2.7/threading.py:504 in run
    self.__target(*self.__args, **self.__kwargs)
  /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
    self.run()
  /usr/lib64/python2.7/threading.py:524 in __bootstrap
    self.__bootstrap_inner()
========================================================================
====                         Green threads                          ====
========================================================================
Thread ID 0
  /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
    return self.greenlet.switch()
  /usr/lib/python2.7/site-packages/eventlet/event.py:116 in wait
    return hubs.get_hub().switch()
  /home/berrange/src/cloud/nova/nova/utils.py:577 in wait
    return self.done.wait()
  /home/berrange/src/cloud/nova/nova/service.py:564 in wait
    x.wait()
  /home/berrange/src/cloud/nova/nova/service.py:148 in run_server
    server.wait()
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
    result = function(*args, **kwargs)

Thread ID 1
  /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
    return self.greenlet.switch()
  /usr/lib/python2.7/site-packages/eventlet/green/select.py:70 in select
    return hub.switch()
  /usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
    ready, _, _ = select([self], [], [], timeout)
  /usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
    sw.wait(timeout)
  /usr/lib/python2.7/site-packages/qpid/concurrency.py:57 in wait
    self.condition.wait(3)
  /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
    return self._waiter.wait(predicate, timeout=timeout)
  /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
    result = self._wait(lambda: self.error or predicate(), timeout)
  /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
    result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
    result = self._ewait(lambda: self.closed or predicate(), timeout)
  /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
    if self._ecwait(lambda: self.incoming, timeout):
  <string>:6 in next_receiver
    None
  /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:426 in _consume
    nxt_receiver = self.session.next_receiver(timeout=timeout)
  /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:375 in ensure
    return method(*args, **kwargs)
  /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:435 in iterconsume
    yield self.ensure(_error_callback, _consume)
  /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:518 in consume
    it.next()
  /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:526 in _consumer_thread
    self.consume()
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
    result = function(*args, **kwargs)

Thread ID 2
  /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
    return self.greenlet.switch()
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep
    hub.switch()
  /home/berrange/src/cloud/nova/nova/utils.py:638 in _inner
    greenthread.sleep(idle)
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
    result = function(*args, **kwargs)

Thread ID 3
  /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
    return self.greenlet.switch()
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep
    hub.switch()
  /home/berrange/src/cloud/nova/nova/utils.py:596 in _inner
    greenthread.sleep(interval)
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
    result = function(*args, **kwargs)
========================================================================
====                            Package                             ====
========================================================================
Vendor: OpenStack Foundation
Product: OpenStack Nova
Version: 2013.1
========================================================================