[Hampshire] Xen hanging on shutdown/reboot

Top Page

Reply to this message
Author: paul@stimpsonfamily.co.uk
Date:  
To: hampshire@mailman.lug.org.uk
Subject: [Hampshire] Xen hanging on shutdown/reboot
Hi,

I'm having an issue with Xen 4.2 running on Ubuntu 12.04.

The install is an Ubuntu 12.04-x64 host with two Ubuntu 12.04-x64 guests and one
Windows server 2008 R2 guest with the signed PV drivers. I'm using the XL
toolkit.

I'm suffering an issue where the machine gets stuck at the "Will now reboot"
line when the host is rebooted while the guests are still running. If I shut all
the guests down using "xl shutdown" the guests do terminate properly (after
about 30 seconds). If I then do a reboot on the host, it completes cleanly.

If I don't shutdown the guests, I have observed some possibly strange behaviour
during the host shutdown before it hangs. In the shutdown progress I see:

> Stopping xenconsoled
> WARNING not stopping xenstored as it cannot be restarted
> Shutting down Xen domains: Linuxguest1 (shut) Linuxguest1 (shut) Linuxguest2
> (shut) Linuxguest2 (shut) Windowsguest (shut) SHUTDOWN_ALL * [done]


The things I think are strange are that the two Linux guests are listed twice in
the shutdown line despite each only being running once and that it blows
straight through the line saying "[done]" without waiting for the guests to
terminate. It then appears to unmount the discs before all the guests have
wrapped up so they never do terminate. If I leave it, I get repeated kernel
warnings that the reboot process has been delayed for more than 120 seconds. I
waited over 300 seconds (the time specified in XENDOMAINS_STOP_MAXWAIT) but it
doesn't time out and recover.

I looked in the Xen logs and they say that all the guests are waiting to die but
then nothing. I'm not sure if this is because the guests aren't terminating or
if it's because the wait didn't happen and the discs are getting unmounted
before the log gets written. If I do a manual "xl shutdown -w" on each of the
guests, it does indeed wait and report successful shutdown, just not when it's
happening automatically on a host reboot or shutdown.

My xendomains file contains:

XENDOMAINS_SYSRQ=""
XENDOMAINS_USLEEP=100000
XENDOMAINS_CREATE_USLEEP=5000000
XENDOMAINS_MIGRATE=""
XENDOMAINS_SAVE=
XENDOMAINS_SHUTDOWN="--halt --wait"
XENDOMAINS_SHUTDOWN_ALL="--all --halt --wait"
XENDOMAINS_RESTORE=false
XENDOMAINS_AUTO=/etc/xen/auto
XENDOMAINS_AUTO_ONLY=false
XENDOMAINS_STOP_MAXWAIT=300


I disabled the save and restore options because I am using PCI passthrough and
trying to restore a saved VM caused the machine to crash at boot.

I'm wondering if these problems are something to do with me using xl as most of
the examples and references to the settings mention xm.

Do you have any suggestions why this hanging or why the "--wait" in xendomains
doesn't seem to be having any effect please? I found a Bitkeeper commit to fix a
bug where "-w" was respected by xl but "--wait" wasn't but changing the options
in xendomains to "-w" doesn't seem to make any difference. I also tried shifting
the Xen shutdown to the first item in the shutdown list but saw no improvement.

Thanks,
Paul.--
Please post to: Hampshire@???
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--------------------------------------------------------------