Steadfast Public Cloud Meltdown & Spectre Information
Posted by Kevin Stange, Last modified by Kevin Stange on 24 September 2018 05:35 PM
This article provides current support information for Spectre and Meltdown on the Steadfast Public Cloud.
Here is a summary of the currently known issues:
The information contained in this article makes assumptions specific to Steadfast's Cloud environment and they are not necessarily applicable to other clouds
Current System Status
The following table describes the current issues and the fix status for each:
The status information indicated above may vary for customers with dedicated hypervisors based on maintenance arrangements. We are continuously working to keep dedicated hypervisors in line with the public cloud.
The current public cloud platform has been patched against Meltdown and Spectre variant 2 attacks. The platform is based on CentOS 7 and Xen 4.8. Prior to February 11th, the platform was based on CentOS 6 and Xen 4.4. Xen 4.4 will not receive updates to fix either Meltdown or Spectre. On January 17th, the first patches that mostly mitigate Meltdown were made available for Xen 4.8, but no patches are yet available to mitigate Spectre. Our migration to Xen 4.8 was the culmination of several months of testing and planning, which we started prior to disclosure of the security issues. We did not modify our plans as a result, other than to increase the urgency of our roll-out.
On February 23rd, the first available Xen patches which mitigate variant 2 of Spectre became available, however the solution required a compiler update and microcode updates that were not yet available on CentOS. We deployed the update with initial mitigation once the compiler update was available in April.
In early June, we deployed CPU microcode updates once they were made available by Intel to mitigate spectre variant 2 using CPU assistance.
In early August, we deployed a version of Xen containing the Lazy FP fix as well as performance improvements related to the Meltdown fix. The fix for Speculative Store Bypass is included, but not effective yet. We deployed the required microcode update in late September.
Available information available suggests no fix is required in Xen for Spectre variants 1.1 and 1.2.
There are two different fixes for L1TF depending on the VM operating mode. A complete fix for Windows VMs requires a microcode update (the same as the SSB microcode update) and to disable hyperthreading features of the CPUs. A complete fix for Linux VMs requires only a Xen software update. We are planning not to disable hyperthreading at the hypervisor level because we have evaluated the risk to be minimal under normal operating conditions and the performance impact to be severe. The necessary code updates for L1TF were deployed in late September.
Current Risks to VMs
Due to the way Xen is designed, Linux VMs do not need Meltdown kernel patches inside the VM. They already cannot exploit the Meltdown issue using the normal Linux method. However, Xen must be fixed to prevent users on Linux VMs from exploiting Xen directly to read the memory of other VMs on the same physical server.
Windows VMs cannot exploit the Xen variant of the Meltdown vulnerability, but Meltdown can be exploited inside a Windows VM to read its own memory from an unprivileged account unless the correct Windows Update package has been installed in the VM. Linux VMs may be able to exploit the Xen issue to read the memory of unsuspecting Windows and Linux VMs even if they are patched.
Both types of VMs are vulnerable to Spectre, however the amount of exposure caused by Spectre is more limited and thus less of a danger overall than Meltdown. Some of the system updates available may mitigate part of Spectre, but the availability of patches varies wildly by operating system. Xen patches along with CPU microcode are required to provide complete mitigation. Some of these fixes have been applied and some are coming soon.
Customers with dedicated hypervisors are protected from other customers, but the risks within VMs still apply as described above. If you trust all the VMs on a single physical server, along with all the users and applications hosted on them, the risks associated with Spectre and Meltdown may be relatively low and you may not need to urgently upgrade. If you have concerns about Meltdown on your private hypervisors, please contact our support team and we will plan out an upgrade to the patched Xen 4.8 version right away. Otherwise, we will be contacting customers with dedicated hypervisors soon to work on upgrade planning soon.
Current Mitigation Options
The most complete mitigation available is running on patched hypervisors. All public cloud VMs are currently on these hypervisors since February 11th and are being kept up to date with patches are soon as they are ready.
If you are running Linux, you should also apply kernel updates that provide fixes to Spectre. Updates fixing Meltdown are not required. Please note that working kernels are now available for all CentOS VMs that fix all spectre variants and Lazy FP, up to and including L1TF:
If you are running Windows, you should also apply Windows updates:
If you trust your VM users, you may also want to consider a dedicated hypervisor, which guarantees all the VMs on the physical server are under your control. This option does not itself prevent Meltdown or Spectre from being exploited, but limits the number of people who may have access to the server memory. Keep in mind that if you operate applications that face untrusted users, an exploitable vulnerability in such an application could allow a user to run their own code and subsequently exploit Meltdown or Spectre. If you'd like to explore the option of switching to private hypervisors, please contact our sales team for further information.
If you need any help or have any questions regarding any of this information, please contact our support team.
September 24, 2018 @ 5:15 PM
September 14, 2018 @ 2:30 PM
August 30, 2018 @ 3:55 PM
August 30, 2018 @ 11:30 AM
August 1, 2018 @ 11:05 AM
July 17, 2018 @ 5:15 PM
March 13, 2018 @ 3:00 PM
February 12, 2018 @ 10:45 AM
February 7, 2018 @ 6:35 PM
January 26, 2018 @ 3:35 PM
January 22, 2018 @ 1:45 PM
January 18, 2018 @ 6:30 PM