Debian 8.x and Ubuntu 14.04 VMs Crash After 5 Minutes on Public Cloud
Posted by Kevin Stange on 30 January 2018 05:43 PM
This article describes an issue where Debian 8.x VMs and Ubuntu 14.04 VMs running 3.x kernels will crash after running for exactly 5 minutes when running on Xen 4.8-based hypervisors.
This issue is caused by a bug in some 3.x kernel series, including 3.13 and 3.16 when handling how Xen 4.8 presents virtual CPUs to the VM. This problem causes the kernel to produce a "machine check exception" and shut off. This problem goes away if you upgrade to a newer version of Debian or Ubuntu that includes a 4.x series kernel such as 4.4 or 4.9, and does not appear to affect older kernels including version 3.2 and 2.6 series from previous versions of Debian and Ubuntu.
To prevent this problem, the boot option "nomce" should be added to the kernel parameters. This can be done by running the following commands from inside the VM:
sed -i 's/^# kopt=.*/\0 nomce/' /boot/grub/menu.lst*
Once you have applied this configuration, reboot the VM to prevent further crashes.
You can verify that your server is running the correct command line options by running the following command:
The output will resemble the following including the "nomce" option:
root=/dev/xvda1 ro nomce
In response to this issue we have corrected the latest version of the Debian 8 template available for provisioning new VMs and have discontinued offering the Ubuntu 14.04 template. Ubuntu 18.04 is expected to be available soon, so this change follows closely to the typical schedule for template removal.