GuruMeditationReport
Contents
Guru Meditation Reports
When things go wrong in (production) deployments of OpenStack collecting debug data is a key first step in the process of triaging & ultimately resolving the problem. Nova has extensively used logging capabilities which produce a vast amount of data. This does not, however, enable an admin to obtain an accurate view on the current live state of the system. For example, what threads are running, what config parameters are in effect, and more. The eventlet backdoor facility provides an interactive shell interface for any eventlet based process, allowing an admin to telnet to a pre-defined port and execute a variety of commands. This can be used to collect the necessary state information, but is has a number of limitations.
- Every service running on a host needs to have the backdoor running on a different TCP port and the admin has to remember which process is listening where. Get this wrong and very bad things can happen.
- The backdoor needs to have been enabled when the process was started. If this was not done before the problem arose, the admin is out of luck because restarting the service to enable the backdoor will loose the critical state that was desired.
- The backdoor shell is too powerful. By presenting an interactive python shell too much burden is placed on the admin to find the right data, without causing problems.
- The backdoor shell is an effective "root shell" available over unsecured TCP port, so no sensible admin will ever enable it on production systems.
Error Report Framework
To address the issues described above, this page outlines the design of a general purpose error report generation framework, known as the "guru meditation report" (cf http://en.wikipedia.org/wiki/Guru_Meditation).
- Models: These classes define structured data for a variety of interesting pieces of state. For example, stack traces, threads, config parameters, package version info, etc. They are capable of being serialized to XML / JSON or a plain text representation
- Generators: These classes are used to populate the model classes with the current runtime state of the system
There will be a number of standard models / generators available for all OpenStack services
- StackTraceModel: a base class for any model which includes a stack trace
- ThreadModel: a class for information about a thread
- ExceptionModel: a class for information about a caught exception
- ConfigModel: a class for information about configuration file settings
- PackageModel: a class for information about vendor/product/version/package information
Each OpenStack project will have the ability to register further generator classes to provide custom project specific data.
Integration with apps
Every long running service process should have a call to install a signal handler which will trigger the guru meditation framework upon receipt of SIGUSR1. This will result in the process dumping a complete report of its current state to stderr.
For processes which deal with RPC processes, it may also be desirable to install some kind of hook in the RPC request dispatcher that will save a guru meditation report whenever the processing of a request results in an uncaught exception. It could save these reports to a well known directory (/var/log/openstack/<project>/<service>/) for later analysis by the sysadmin or automated bug analysis tools.
Example Report
======================================================================== ==== Guru meditation report ==== ======================================================================== UUID: 6d4faf99-480c-4d0c-90be-b7d090ecd83e Time: Mon, 18 Feb 2013 18:01:11 +0000 ======================================================================== ==== Config ==== ======================================================================== DEFAULT: allow_resize_to_same_host=True allow_same_net_traffic=True allowed_direct_url_schemes=[] allowed_rpc_exception_modules=['nova.openstack.common.exception', 'nova.exception', 'cinder.exception', 'exceptions'] api_paste_config=/etc/nova/api-paste.ini api_rate_limit=True auth_strategy=keystone auto_assign_floating_ip=False backdoor_port=None bandwidth_poll_interval=600 base_dir_name=_base bindir=/home/berrange/src/cloud/nova/bin block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_NON_SHARED_INC boot_script_template=/home/berrange/src/cloud/nova/nova/cloudpipe/bootscript.template ca_file=cacert.pem ca_path=/home/berrange/src/cloud/data/nova/CA cert_manager=nova.cert.manager.CertManager checksum_base_images=False checksum_interval_seconds=3600 cinder_api_insecure=False cinder_catalog_info=volume:cinder:publicURL cinder_cross_az_attach=True cinder_endpoint_template=None cinder_http_retries=3 cnt_vpn_clients=0 compute_api_class=nova.compute.api.API compute_driver=libvirt.LibvirtDriver compute_manager=nova.compute.manager.ComputeManager compute_stats_class=nova.compute.stats.Stats compute_topic=compute config_dir=None config_drive_format=iso9660 config_drive_skip_versions=1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01 config_drive_tempdir=None config_file=['/etc/nova/nova.conf'] console_host=mustard.gsslab.fab.redhat.com console_manager=nova.console.manager.ConsoleProxyManager console_topic=console consoleauth_topic=consoleauth control_exchange=nova create_unique_mac_address_attempts=5 crl_file=crl.pem db_backend=sqlalchemy db_driver=nova.db debug=True default_access_ip_network_name=None default_availability_zone=nova default_ephemeral_format=None default_floating_pool=nova default_instance_type=m1.small default_log_levels=['amqplib=WARN', 'sqlalchemy=WARN', 'boto=WARN', 'suds=INFO', 'keystone=INFO', 'eventlet.wsgi.server=WARN'] default_notification_level=INFO default_publisher_id=mustard.gsslab.fab.redhat.com default_schedule_zone=None defer_iptables_apply=False dhcp_domain=novalocal dhcp_lease_time=120 dhcpbridge=/home/berrange/src/cloud/nova/bin/nova-dhcpbridge dhcpbridge_flagfile=/etc/nova/nova.conf disable_process_locking=False dmz_cidr=[] dmz_mask=255.255.255.0 dmz_net=10.0.0.0 dns_server=[] dns_update_periodic_interval=-1 dnsmasq_config_file= ec2_listen=0.0.0.0 ec2_listen_port=8773 ec2_private_dns_show_ip=False ec2_strict_validation=True ec2_timestamp_expiry=300 ec2_workers=None enable_new_services=True enabled_apis=['ec2', 'osapi_compute', 'metadata'] enabled_ssl_apis=[] fake_call=False fake_network=False fake_rabbit=False fatal_deprecations=False fatal_exception_format_errors=False firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver fixed_ip_disassociate_timeout=600 fixed_range=10.0.0.0/24 fixed_range_v6=fd00::/48 flat_injected=False flat_interface=eth0 flat_network_bridge=br100 flat_network_dns=8.8.4.4 floating_ip_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver force_config_drive=None force_dhcp_release=True force_raw_images=True forward_bridge_interface=['all'] gateway=None gateway_v6=None glance_api_insecure=False glance_api_servers=['10.33.8.112:9292'] glance_host=10.33.8.112 glance_num_retries=0 glance_port=9292 glance_protocol=http glusterfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt heal_instance_info_cache_interval=60 host=mustard.gsslab.fab.redhat.com host_state_interval=120 image_cache_manager_interval=2400 image_info_filename_pattern=/home/berrange/src/cloud/data/nova/instances/_base/%(image)s.info injected_network_template=/home/berrange/src/cloud/nova/nova/virt/interfaces.template instance_build_timeout=0 instance_dns_domain= instance_dns_manager=nova.network.noop_dns_driver.NoopDNSDriver instance_format=[instance: %(uuid)s] instance_name_template=instance-%08x instance_usage_audit=False instance_usage_audit_period=month instance_uuid_format=[instance: %(uuid)s] instances_path=/home/berrange/src/cloud/data/nova/instances internal_service_availability_zone=internal iptables_bottom_regex= iptables_top_regex= ipv6_backend=rfc2462 key_file=private/cakey.pem keys_path=/home/berrange/src/cloud/data/nova/keys keystone_ec2_url=http://localhost:5000/v2.0/ec2tokens l3_lib=nova.network.l3.LinuxNetL3 libvirt_cpu_mode=none libvirt_cpu_model=None libvirt_disk_prefix=None libvirt_images_type=default libvirt_images_volume_group=None libvirt_inject_key=True libvirt_inject_partition=1 libvirt_inject_password=False libvirt_lvm_snapshot_size=1000 libvirt_nonblocking=True libvirt_ovs_bridge=br-int libvirt_snapshot_compression=False libvirt_snapshots_directory=/home/berrange/src/cloud/data/nova/instances/snapshots libvirt_sparse_logical_volumes=False libvirt_type=kvm libvirt_uri= libvirt_use_virtio_for_bridges=True libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver libvirt_volume_drivers=['iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver', 'local=nova.virt.libvirt.volume.LibvirtVolumeDriver', 'fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver', 'rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver', 'nfs=nova.virt.libvirt.volume.LibvirtNFSVolumeDriver', 'aoe=nova.virt.libvirt.volume.LibvirtAOEVolumeDriver', 'glusterfs=nova.virt.libvirt.volume.LibvirtGlusterfsVolumeDriver'] libvirt_wait_soft_reboot_seconds=120 linuxnet_interface_driver=nova.network.linux_net.LinuxBridgeInterfaceDriver linuxnet_ovs_integration_bridge=br-int live_migration_bandwidth=0 live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER live_migration_retry_count=30 live_migration_uri=qemu+tcp://%s/system lock_path=/home/berrange/src/cloud/data/nova lockout_attempts=5 lockout_minutes=15 lockout_window=15 log_config=None log_date_format=%Y-%m-%d %H:%M:%S log_dir=None log_file=None log_format=%(asctime)s %(levelname)8s [%(name)s] %(message)s logfile_mode=0644 logging_context_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [%(request_id)s %(user_name)s %(project_name)s%(color)s] %(instance)s%(color)s%(message)s logging_debug_format_suffix=from (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d logging_default_format_string=%(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s logging_exception_prefix=%(color)s%(asctime)s.%(msecs)03d TRACE %(name)s %(instance)s max_age=0 memcached_servers=None metadata_host=10.33.8.112 metadata_listen=0.0.0.0 metadata_listen_port=8775 metadata_manager=nova.api.manager.MetadataManager metadata_port=8775 metadata_workers=None mkisofs_cmd=genisoimage monkey_patch=False monkey_patch_modules=['nova.api.ec2.cloud:nova.openstack.common.notifier.api.notify_decorator', 'nova.compute.api:nova.openstack.common.notifier.api.notify_decorator'] multi_host=False my_ip=10.33.8.112 network_api_class=nova.network.api.API network_device_mtu=None network_driver=nova.network.linux_net network_manager=nova.network.manager.FlatDHCPManager network_size=256 network_topic=network networks_path=/home/berrange/src/cloud/data/nova/networks nfs_mount_options=None nfs_mount_point_base=/home/berrange/src/cloud/data/nova/mnt non_inheritable_image_properties=['cache_in_nova', 'bittorrent'] notification_driver=[] notify_api_faults=False notify_on_any_change=False notify_on_state_change=None novncproxy_base_url=http://10.33.8.112:6080/vnc_auto.html null_kernel=nokernel num_aoe_discover_tries=3 num_iscsi_scan_tries=3 num_networks=1 os_region_name=None osapi_compute_listen=0.0.0.0 osapi_compute_listen_port=8774 osapi_compute_workers=None password_length=12 periodic_enable=True periodic_fuzzy_delay=60 policy_default_rule=default policy_file=policy.json project_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=project-ca-%.16s-%s public_interface=br100 publish_errors=False pybasedir=/home/berrange/src/cloud/nova qpid_heartbeat=60 qpid_hostname=localhost qpid_hosts=['localhost:5672'] qpid_password= qpid_port=5672 qpid_protocol=tcp qpid_sasl_mechanisms= qpid_tcp_nodelay=True qpid_username= quota_cores=20 quota_driver=nova.quota.DbQuotaDriver quota_floating_ips=10 quota_injected_file_content_bytes=10240 quota_injected_file_path_bytes=255 quota_injected_files=5 quota_instances=10 quota_key_pairs=100 quota_metadata_items=128 quota_ram=51200 quota_security_group_rules=20 quota_security_groups=10 rbd_secret_uuid=None rbd_user=None reboot_timeout=0 reclaim_instance_interval=0 remove_unused_base_images=True remove_unused_original_minimum_age_seconds=86400 remove_unused_resized_minimum_age_seconds=3600 report_interval=10 rescue_image_id=None rescue_kernel_id=None rescue_ramdisk_id=None rescue_timeout=0 reservation_expire=86400 reserved_host_disk_mb=0 reserved_host_memory_mb=512 resize_confirm_window=0 resume_guests_state_on_host_boot=False rootwrap_config=/etc/nova/rootwrap.conf routing_source_ip=10.33.8.112 rpc_backend=nova.openstack.common.rpc.impl_qpid rpc_cast_timeout=30 rpc_conn_pool_size=30 rpc_response_timeout=60 rpc_thread_pool_size=64 run_external_periodic_tasks=True running_deleted_instance_action=log running_deleted_instance_poll_interval=1800 running_deleted_instance_timeout=0 scheduler_manager=nova.scheduler.manager.SchedulerManager scheduler_topic=scheduler security_group_api=nova.compute.api.SecurityGroupAPI security_group_handler=nova.network.sg.NullSecurityGroupHandler send_arp_for_ha=False send_arp_for_ha_count=3 service_down_time=60 servicegroup_driver=db share_dhcp_address=False snapshot_image_format=None snapshot_name_template=snapshot-%s sql_connection=mysql://root:123456@localhost/nova?charset=utf8 sql_connection_debug=0 sql_connection_trace=False sql_dbpool_enable=False sql_idle_timeout=3600 sql_max_overflow=None sql_max_pool_size=5 sql_max_retries=10 sql_min_pool_size=1 sql_retry_interval=10 sqlite_db=nova.sqlite sqlite_synchronous=True ssl_ca_file=None ssl_cert_file=None ssl_key_file=None state_path=/home/berrange/src/cloud/data/nova syslog_log_facility=LOG_USER tcp_keepidle=600 teardown_unused_network_gateway=False tempdir=None until_refresh=0 update_dns_entries=False use_cow_images=True use_forwarded_for=False use_ipv6=False use_network_dns_servers=False use_project_ca=False use_single_default_gateway=False use_stderr=True use_syslog=False use_usb_tablet=True user_cert_subject=/C=US/ST=California/O=OpenStack/OU=NovaDev/CN=%.16s-%.16s-%s verbose=True virt_mkfs=['default=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'linux=mkfs.ext3 -L %(fs_label)s -F %(target)s', 'windows=mkfs.ntfs --force --fast --label %(fs_label)s %(target)s'] vlan_interface=eth0 vlan_start=100 vnc_enabled=True vnc_keymap=en-us vncserver_listen=127.0.0.1 vncserver_proxyclient_address=127.0.0.1 volume_api_class=nova.volume.cinder.API volume_usage_poll_interval=0 vpn_image_id=0 vpn_instance_type=m1.tiny vpn_ip=10.33.8.112 vpn_key_suffix=-vpn vpn_start=1000 wsgi_log_format=%(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f xen_hvmloader_path=/usr/lib/xen/boot/hvmloader xvpvncproxy_base_url=http://10.33.8.112:6081/console cells: call_timeout=60 capabilities=['hypervisor=xenserver;kvm', 'os=linux;windows'] enable=False manager=nova.cells.manager.CellsManager name=nova topic=cells conductor: manager=nova.conductor.manager.ConductorManager topic=conductor use_local=False spice: agent_enabled=True enabled=False html5proxy_base_url=http://10.33.8.112:6082/spice_auto.html keymap=en-us server_listen=127.0.0.1 server_proxyclient_address=127.0.0.1 ======================================================================== ==== Native threads ==== ======================================================================== Thread ID 139989669103360 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139989677496064 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990096934656 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990667343616 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990063363840 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990692521728 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990088541952 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990071756544 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990675736320 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990650558208 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990080149248 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990700914432 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990781155072 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139989685888768 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139989694281472 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990684129024 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139991167133504 /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:81 in get_model None /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/generator.py:52 in create_report None /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:42 in create None /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:78 in dump None /home/berrange/src/cloud/nova/nova/openstack/common/gurumed/report.py:85 in __handle_signal None /usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py:61 in do_poll return self.poll.poll(seconds) /usr/lib/python2.7/site-packages/eventlet/hubs/poll.py:84 in wait presult = self.do_poll(seconds) /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:226 in run self.wait(sleep_time) Thread ID 139990658950912 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990789547776 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990046578432 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() Thread ID 139990054971136 /usr/lib64/python2.7/threading.py:243 in wait waiter.acquire() /usr/lib64/python2.7/Queue.py:168 in get self.not_empty.wait() /usr/lib/python2.7/site-packages/eventlet/tpool.py:68 in tworker msg = reqq.get() /usr/lib64/python2.7/threading.py:504 in run self.__target(*self.__args, **self.__kwargs) /usr/lib64/python2.7/threading.py:551 in __bootstrap_inner self.run() /usr/lib64/python2.7/threading.py:524 in __bootstrap self.__bootstrap_inner() ======================================================================== ==== Green threads ==== ======================================================================== Thread ID 0 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch return self.greenlet.switch() /usr/lib/python2.7/site-packages/eventlet/event.py:116 in wait return hubs.get_hub().switch() /home/berrange/src/cloud/nova/nova/utils.py:577 in wait return self.done.wait() /home/berrange/src/cloud/nova/nova/service.py:564 in wait x.wait() /home/berrange/src/cloud/nova/nova/service.py:148 in run_server server.wait() /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main result = function(*args, **kwargs) Thread ID 1 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch return self.greenlet.switch() /usr/lib/python2.7/site-packages/eventlet/green/select.py:70 in select return hub.switch() /usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait ready, _, _ = select([self], [], [], timeout) /usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait sw.wait(timeout) /usr/lib/python2.7/site-packages/qpid/concurrency.py:57 in wait self.condition.wait(3) /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait return self._waiter.wait(predicate, timeout=timeout) /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait result = self._wait(lambda: self.error or predicate(), timeout) /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait result = self.connection._ewait(lambda: self.error or predicate(), timeout) /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait result = self._ewait(lambda: self.closed or predicate(), timeout) /usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in next_receiver if self._ecwait(lambda: self.incoming, timeout): <string>:6 in next_receiver None /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:426 in _consume nxt_receiver = self.session.next_receiver(timeout=timeout) /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:375 in ensure return method(*args, **kwargs) /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:435 in iterconsume yield self.ensure(_error_callback, _consume) /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:518 in consume it.next() /home/berrange/src/cloud/nova/nova/openstack/common/rpc/impl_qpid.py:526 in _consumer_thread self.consume() /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main result = function(*args, **kwargs) Thread ID 2 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch return self.greenlet.switch() /usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep hub.switch() /home/berrange/src/cloud/nova/nova/utils.py:638 in _inner greenthread.sleep(idle) /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main result = function(*args, **kwargs) Thread ID 3 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:177 in switch return self.greenlet.switch() /usr/lib/python2.7/site-packages/eventlet/greenthread.py:30 in sleep hub.switch() /home/berrange/src/cloud/nova/nova/utils.py:596 in _inner greenthread.sleep(interval) /usr/lib/python2.7/site-packages/eventlet/greenthread.py:192 in main result = function(*args, **kwargs) ======================================================================== ==== Package ==== ======================================================================== Vendor: OpenStack Foundation Product: OpenStack Nova Version: 2013.1 ========================================================================