GuruMeditationReport
Contents
Guru Meditation Reports
When things go wrong in (production) deployments of OpenStack collecting debug data is a key first step in the process of triaging & ultimately resolving the problem. Nova has extensively used logging capabilities which produce a vast amount of data. This does not, however, enable an admin to obtain an accurate view on the current live state of the system. For example, what threads are running, what config parameters are in effect, and more. The eventlet backdoor facility provides an interactive shell interface for any eventlet based process, allowing an admin to telnet to a pre-defined port and execute a variety of commands. This can be used to collect the necessary state information, but is has a number of limitations.
- Every service running on a host needs to have the backdoor running on a different TCP port and the admin has to remember which process is listening where. Get this wrong and very bad things can happen.
- The backdoor needs to have been enabled when the process was started. If this was not done before the problem arose, the admin is out of luck because restarting the service to enable the backdoor will loose the critical state that was desired.
- The backdoor shell is too powerful. By presenting an interactive python shell too much burden is placed on the admin to find the right data, without causing problems.
- The backdoor shell is an effective "root shell" available over unsecured TCP port, so no sensible admin will ever enable it on production systems.
Error Report Framework
To address the issues described above, this page outlines the design of a general purpose error report generation framework, known as the "guru meditation report" (cf http://en.wikipedia.org/wiki/Guru_Meditation).
- Models: These classes define structured data for a variety of interesting pieces of state. For example, stack traces, threads, config parameters, package version info, etc. They are capable of being serialized to XML / JSON or a plain text representation
- Generators: These classes are used to populate the model classes with the current runtime state of the system
There will be a number of standard models / generators available for all OpenStack services
- StackTraceModel: a base class for any model which includes a stack trace
- ThreadModel: a class for information about a thread
- ExceptionModel: a class for information about a caught exception
- ConfigModel: a class for information about configuration file settings
- PackageModel: a class for information about vendor/product/version/package information
Each OpenStack project will have the ability to register further generator classes to provide custom project specific data.
Integration with apps
Every long running service process should have a call to install a signal handler which will trigger the guru meditation framework upon receipt of SIGUSR1. This will result in the process dumping a complete report of its current state to stderr.
For processes which deal with RPC processes, it may also be desirable to install some kind of hook in the RPC request dispatcher that will save a guru meditation report whenever the processing of a request results in an uncaught exception. It could save these reports to a well known directory (/var/log/openstack/<project>/<service>/) for later analysis by the sysadmin or automated bug analysis tools.
Example Report
========================================================================
==== Guru meditation report ====
========================================================================
UUID: 6d4faf99-480c-4d0c-90be-b7d090ecd83e
Time: Mon, 18 Feb 2013 18:01:11 +0000
========================================================================
==== Config ====
========================================================================
DEFAULT:
allow_resize_to_same_host=True
allow_same_net_traffic=True
allowed_direct_url_schemes=[]
allowed_rpc_exception_modules=['nova.openstack.common.exception', 'nova.exception', 'cinder.exception', 'exceptions']
api_paste_config=/etc/nova/api-paste.ini
api_rate_limit=True
auth_strategy=keystone
auto_assign_floating_ip=False
backdoor_port=None
bandwidth_poll_interval=600
base_dir_name=_base
bindir=/home/berrange/src/cloud/nova/bin
block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_NON_SHARED_INC
boot_script_template=/home/berrange/src/cloud/nova/nova/cloudpipe/bootscript.template
ca_file=cacert.pem
ca_path=/home/berrange/src/cloud/data/nova/CA
cert_manager=nova.cert.manager.CertManager
checksum_base_images=False
checksum_interval_seconds=3600
cinder_api_insecure=False
cinder_catalog_info=volume:cinder:publicURL
cinder_cross_az_attach=True
cinder_endpoint_template=None
cinder_http_retries=3
cnt_vpn_clients=0
compute_api_class=nova.compute.api.API
compute_driver=libvirt.LibvirtDriver
compute_manager=nova.compute.manager.ComputeManager
compute_stats_class=nova.compute.stats.Stats
compute_topic=compute
config_dir=None
config_drive_format=iso9660
config_drive_skip_versions=1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01
config_drive_tempdir=None
config_file=['/etc/nova/nova.conf']
console_host=mustard.gsslab.fab.redhat.com
console_manager=nova.console.manager.ConsoleProxyManager
console_topic=console
consoleauth_topic=consoleauth
control_exchange=nova
create_unique_mac_address_attempts=5
crl_file=crl.pem
db_backend=sqlalchemy
db_driver=nova.db
debug=True
default_access_ip_network_name=None
default_availability_zone=nova
default_ephemeral_format=None
default_floating_pool=nova
default_instance_type=m1.small
default_log_levels=['amqplib=WARN', 'sqlalchemy=WARN', 'boto=WARN', 'suds=INFO', 'keystone=INFO', 'eventlet.wsgi.server=WARN']
default_notification_level=INFO
default_publisher_id=mustard.gsslab.fab.redhat.com
default_schedule_zone=None
defer_iptables_apply=False
dhcp_domain=novalocal
dhcp_lease_time=120
dhcpbridge=/home/berrange/src/cloud/nova/bin/nova-dhcpbridge
dhcpbridge_flagfile=/etc/nova/nova.conf
disable_process_locking=False
dmz_cidr=[]
dmz_mask=255.255.255.0
dmz_net=10.0.0.0
dns_server=[]
dns_update_periodic_interval=-1
dnsmasq_config_file=
ec2_listen=0.0.0.0
ec2_listen_port=8773
ec2_private_dns_show_ip=False
ec2_strict_validation=True
ec2_timestamp_expiry=300
ec2_workers=None
enable_new_services=True
enabled_apis=['ec2', 'osapi_compute', 'metadata']
enabled_ssl_apis=[]
fake_call=False
fake_network=False
fake_rabbit=False
fatal_deprecations=False
fatal_exception_format_errors=False
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
fixed_ip_disassociate_timeout=600
fixed_range=10.0.0.0/24
fixed_range_v6=fd00::/48
flat_injected=False
flat_interface=eth0
flat_network_bridge=br100
flat_network_dns=8.8.4.4
floating_ip_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver
force_config_drive=None
force_dhcp_release=True
force_raw_images=True
forward_bridge_interface=['all']
gateway=None
gateway_v6=None
glance_api_insecure=False
glance_api_servers=['10.33.8.112:9292']
glance_host=10.33.8.112
glance_num_retries=0
glance_port=9292
glance_protocol=http
glusterfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt
heal_instance_info_cache_interval=60
host=mustard.gsslab.fab.redhat.com
host_state_interval=120
image_cache_manager_interval=2400
image_info_filename_pattern=/home/berrange/src/cloud/data/nova/instances/_base/%(image)s.info
injected_network_template=/home/berrange/src/cloud/nova/nova/virt/interfaces.template
instance_build_timeout=0
instance_dns_domain=
instance_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver
instance_format=[instance: %(uuid)s]
instance_name_template=instance-%08x
instance_usage_audit=False
instance_usage_audit_period=month
instance_uuid_format=[instance: %(uuid)s]
instances_path=/home/berrange/src/cloud/data/nova/instances
internal_service_availability_zone=internal
iptables_bottom_regex=
iptables_top_regex=
ipv6_backend=rfc2462
key_file=private/cakey.pem
keys_path=/home/berrange/src/cloud/data/nova/keys
keystone_ec2_url=http://localhost:5000/v2.0/ec2tokens
l3_lib=nova.network.l3.LinuxNetL3
libvirt_cpu_mode=none
libvirt_cpu_model=None
libvirt_disk_prefix=None
libvirt_images_type=default
libvirt_images_volume_group=None
libvirt_inject_key=True
libvirt_inject_partition=1
libvirt_inject_password=False
libvirt_lvm_snapshot_size=1000
libvirt_nonblocking=True
libvirt_ovs_bridge=br-int
libvirt_snapshot_compression=False
libvirt_snapshots_directory=/home/berrange/src/cloud/data/nova/instances/snapshots
libvirt_sparse_logical_volumes=False
libvirt_type=kvm
libvirt_uri=
libvirt_use_virtio_for_bridges=True
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
libvirt_volume_drivers=['iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver', 'local=nova.virt.libvirt.volume.LibvirtVolumeDriver', 'fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver', 'rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'nfs=nova.virt.libvirt.volume.LibvirtNFSVolumeDriver', 'aoe=nova.virt.libvirt.volume.LibvirtAOEVolumeDriver', 'glusterfs=nova.virt.libvirt.volume.LibvirtGlusterfsVolumeDriver']
libvirt_wait_soft_reboot_seconds=120
linuxnet_interface_driver=nova.network.linux_net.LinuxBridgeInterfaceDriver
linuxnet_ovs_integration_bridge=br-int
live_migration_bandwidth=0
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER
live_migration_retry_count=30
live_migration_uri=qemu+tcp://%s/system
lock_path=/home/berrange/src/cloud/data/nova
lockout_attempts=5
lockout_minutes=15
lockout_window=15
log_config=None
log_date_format=%Y-%m-%d %H:%M:%S
log_dir=None
log_file=None
log_format=%(asctime)s %(levelname)8s [%(name)s] %(message)s
logfile_mode=0644
logging_context_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s%(color)s] %(instance)s%(color)s%(message)s
logging_debug_format_suffix=from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d
logging_default_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s
logging_exception_prefix=%(color)s%(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s
max_age=0
memcached_servers=None
metadata_host=10.33.8.112
metadata_listen=0.0.0.0
metadata_listen_port=8775
metadata_manager=nova.api.manager.MetadataManager
metadata_port=8775
metadata_workers=None
mkisofs_cmd=genisoimage
monkey_patch=False
monkey_patch_modules=['nova.api.ec2.cloud:nova.openstack.common.notifier.api.notify_decorator', 'nova.compute.api:nova.openstack.common.notifier.api.notify_decorator']
multi_host=False
my_ip=10.33.8.112
network_api_class=nova.network.api.API
network_device_mtu=None
network_driver=nova.network.linux_net
network_manager=nova.network.manager.FlatDHCPManager
network_size=256
network_topic=network
networks_path=/home/berrange/src/cloud/data/nova/networks
nfs_mount_options=None
nfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt
non_inheritable_image_properties=['cache_in_nova', 'bittorrent']
notification_driver=[]
notify_api_faults=False
notify_on_any_change=False
notify_on_state_change=None
novncproxy_base_url=http://10.33.8.112:6080/vnc_auto.html
null_kernel=nokernel
num_aoe_discover_tries=3
num_iscsi_scan_tries=3
num_networks=1
os_region_name=None
osapi_compute_listen=0.0.0.0
osapi_compute_listen_port=8774
osapi_compute_workers=None
password_length=12
periodic_enable=True
periodic_fuzzy_delay=60
policy_default_rule=default
policy_file=policy.json
project_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=project-ca-%.16s-%s
public_interface=br100
publish_errors=False
pybasedir=/home/berrange/src/cloud/nova
qpid_heartbeat=60
qpid_hostname=localhost
qpid_hosts=['localhost:5672']
qpid_password=
qpid_port=5672
qpid_protocol=tcp
qpid_sasl_mechanisms=
qpid_tcp_nodelay=True
qpid_username=
quota_cores=20
quota_driver=nova.quota.DbQuotaDriver
quota_floating_ips=10
quota_injected_file_content_bytes=10240
quota_injected_file_path_bytes=255
quota_injected_files=5
quota_instances=10
quota_key_pairs=100
quota_metadata_items=128
quota_ram=51200
quota_security_group_rules=20
quota_security_groups=10
rbd_secret_uuid=None
rbd_user=None
reboot_timeout=0
reclaim_instance_interval=0
remove_unused_base_images=True
remove_unused_original_minimum_age_seconds=86400
remove_unused_resized_minimum_age_seconds=3600
report_interval=10
rescue_image_id=None
rescue_kernel_id=None
rescue_ramdisk_id=None
rescue_timeout=0
reservation_expire=86400
reserved_host_disk_mb=0
reserved_host_memory_mb=512
resize_confirm_window=0
resume_guests_state_on_host_boot=False
rootwrap_config=/etc/nova/rootwrap.conf
routing_source_ip=10.33.8.112
rpc_backend=nova.openstack.common.rpc.impl_qpid
rpc_cast_timeout=30
rpc_conn_pool_size=30
rpc_response_timeout=60
rpc_thread_pool_size=64
run_external_periodic_tasks=True
running_deleted_instance_action=log
running_deleted_instance_poll_interval=1800
running_deleted_instance_timeout=0
scheduler_manager=nova.scheduler.manager.SchedulerManager
scheduler_topic=scheduler
security_group_api=nova.compute.api.SecurityGroupAPI
security_group_handler=nova.network.sg.NullSecurityGroupHandler
send_arp_for_ha=False
send_arp_for_ha_count=3
service_down_time=60
servicegroup_driver=db
share_dhcp_address=False
snapshot_image_format=None
snapshot_name_template=snapshot-%s
sql_connection=mysql://root:123456@localhost/nova?charset=utf8
sql_connection_debug=0
sql_connection_trace=False
sql_dbpool_enable=False
sql_idle_timeout=3600
sql_max_overflow=None
sql_max_pool_size=5
sql_max_retries=10
sql_min_pool_size=1
sql_retry_interval=10
sqlite_db=nova.sqlite
sqlite_synchronous=True
ssl_ca_file=None
ssl_cert_file=None
ssl_key_file=None
state_path=/home/berrange/src/cloud/data/nova
syslog_log_facility=LOG_USER
tcp_keepidle=600
teardown_unused_network_gateway=False
tempdir=None
until_refresh=0
update_dns_entries=False
use_cow_images=True
use_forwarded_for=False
use_ipv6=False
use_network_dns_servers=False
use_project_ca=False
use_single_default_gateway=False
use_stderr=True
use_syslog=False
use_usb_tablet=True
user_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=%.16s-%.16s-%s
verbose=True
virt_mkfs=['default=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'linux=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'windows=mkfs.ntfs --force --fast --label %(fs_label)s %(target)s']
vlan_interface=eth0
vlan_start=100
vnc_enabled=True
vnc_keymap=en-us
vncserver_listen=127.0.0.1
vncserver_proxyclient_address=127.0.0.1
volume_api_class=nova.volume.cinder.API
volume_usage_poll_interval=0
vpn_image_id=0
vpn_instance_type=m1.tiny
vpn_ip=10.33.8.112
vpn_key_suffix=-vpn
vpn_start=1000
wsgi_log_format=%(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f
xen_hvmloader_path=/usr/lib/xen/boot/hvmloader
xvpvncproxy_base_url=http://10.33.8.112:6081/console
cells:
call_timeout=60
capabilities=['hypervisor=xenserver;kvm', 'os=linux;windows']
enable=False
manager=nova.cells.manager.CellsManager
name=nova
topic=cells
conductor:
manager=nova.conductor.manager.ConductorManager
topic=conductor
use_local=False
spice:
agent_enabled=True
enabled=False
html5proxy_base_url=http://10.33.8.112:6082/spice_auto.html
keymap=en-us
server_listen=127.0.0.1
server_proxyclient_address=127.0.0.1
========================================================================
==== Native threads ====
========================================================================
Thread ID 139989669103360
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139989677496064
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990096934656
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990667343616
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990063363840
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990692521728
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990088541952
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990071756544
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990675736320
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990650558208
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990080149248
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990700914432
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990781155072
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139989685888768
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139989694281472
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990684129024
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139991167133504
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:81 in get_model
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:52 in create_report
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:42 in create
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:78 in dump
None
/home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:85 in __handle_signal
None
/usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py:61 in do_poll
return self.poll.poll(seconds)
/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py:84 in wait
presult = self.do_poll(seconds)
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:226 in run
self.wait(sleep_time)
Thread ID 139990658950912
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990789547776
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990046578432
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
Thread ID 139990054971136
/usr/lib64/python2.7/threading.py:243 in wait
waiter.acquire()
/usr/lib64/python2.7/Queue.py:168 in get
self.not_empty.wait()
/usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker
msg = reqq.get()
/usr/lib64/python2.7/threading.py:504 in run
self.__target(*self.__args, **self.__kwargs)
/usr/lib64/python2.7/threading.py:551 in __bootstrap_inner
self.run()
/usr/lib64/python2.7/threading.py:524 in __bootstrap
self.__bootstrap_inner()
========================================================================
==== Green threads ====
========================================================================
Thread ID 0
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/event.py:116 in wait
return hubs.get_hub().switch()
/home/berrange/src/cloud/nova/nova/utils.py:577 in wait
return self.done.wait()
/home/berrange/src/cloud/nova/nova/service.py:564 in wait
x.wait()
/home/berrange/src/cloud/nova/nova/service.py:148 in run_server
server.wait()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
Thread ID 1
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/green/select.py:70 in select
return hub.switch()
/usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
ready, _, _ = select([self], [], [], timeout)
/usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
sw.wait(timeout)
/usr/lib/python2.7/site-packages/qpid/concurrency.py:57 in wait
self.condition.wait(3)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
return self._waiter.wait(predicate, timeout=timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
result = self._wait(lambda: self.error or predicate(), timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
result = self.connection._ewait(lambda: self.error or predicate(), timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
result = self._ewait(lambda: self.closed or predicate(), timeout)
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver
if self._ecwait(lambda: self.incoming, timeout):
<string>:6 in next_receiver
None
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:426 in _consume
nxt_receiver = self.session.next_receiver(timeout=timeout)
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:375 in ensure
return method(*args, **kwargs)
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:435 in iterconsume
yield self.ensure(_error_callback, _consume)
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:518 in consume
it.next()
/home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:526 in _consumer_thread
self.consume()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
Thread ID 2
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep
hub.switch()
/home/berrange/src/cloud/nova/nova/utils.py:638 in _inner
greenthread.sleep(idle)
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
Thread ID 3
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch
return self.greenlet.switch()
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep
hub.switch()
/home/berrange/src/cloud/nova/nova/utils.py:596 in _inner
greenthread.sleep(interval)
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main
result = function(*args, **kwargs)
========================================================================
==== Package ====
========================================================================
Vendor: OpenStack Foundation
Product: OpenStack Nova
Version: 2013.1
========================================================================
