We have scheduled a software upgrade of our SAN infrastructure in Chicago, IL. This is purely a backend upgrade and does not change interactions with client environments.

Date: Sunday, March 11th, 2018
Start Time: 1:00 AM CST (GMT -6)
End Time: 4:00 AM CDT (GMT -5)

Maintenance Scope: Hybrid Performance SAN (350 E. Cermak Chicago)

Customer Impact:

There is no expected impact during this maintenance as our SAN infrastructure is fully redundant. While we don’t expect any issues we recommend taking a backup prior to the start of this window. This maintenance does not impact our NetApp SAN (Public Cloud Infrastructure)

If you have any questions or concerns please feel free to reach out to our support team at support@steadfast.net.

Update: 2:15am CST (3:15am CDT) This maintenance has completed successfully. 

Note: On May 21st, 2018 an additional vulnerability identified as variant 4, called Speculative Store Bypass, was disclosed as CVE-2018-3639.  Mitigations for variant 4 are not discussed in this advisory.

We are aware of a new series of issues affecting all modern CPU models which can allow for applications to potentially access information and execute code that should not be allowed.  Some reports of these problems began to surface in the media starting around January 1st, but the reports were incomplete.  This class of vulnerabilities is reported to affect all operating systems and CPU types, including servers, home computers, and likely mobile and embedded devices.  This situation is serious and affects everyone, so we are making it a top priority to gather and communicate information as soon as we have it, and we are committed to ensuring mitigations are in place as soon as possible.

The vulnerabilities have been labeled CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754 in the Common Vulnerabilities and Exposures database.  They have also been given the nicknames Meltdown and Spectre.

Due to mistakes in the responsible disclosure process, some information about the issues was unintentionally disclosed early.  Consequently the reporters decided to move up the formal disclosure date to January 3rd from the following week.  As a result of the shortened preparation window by major vendors, most vendors did not have mitigation patches available by the disclosure date.

Patching

The following information describes patching processes for various operating systems.  Please find the section below matching your operating system to learn how to apply the relevant fixes for it.

Note that in addition to the fixes provided in your operating system, hardware changes may also be needed.  At this time, very little information is available as to which hardware will be fixed.  See the hardware fixes section for information about progress.

Important! Due to the nature of the issue, some of the patches may negatively impact the performance of your server.  This happens because the vulnerable behavior is part of a performance optimization feature.  Removing the optimization helps mitigate the problem but also eliminates the performance benefit of the optimization.  If you experience significant performance issues with your applications after applying updates, please check with your application vendor for advice or contact our support team so we can work with you to look for ways to counter the performance loss.

Windows Server

Microsoft has produced updates for the following supported Windows server versions effective January 3rd:

• Windows Server 2016: Build 14393.2007 (KB4056890)
• Windows Server 2012 R2: KB4056898
• Windows Server 2008 R2: KB4056897 or KB4056894

Windows Server 2008 and 2012 (non-R2 variants) do not currently have available updates.  Windows Server 2003 is End of Life and will not be updated.

To protect computer systems running Windows desktop or server operating systems, open the Windows Update application for your version of Windows.  Select the option "Check for Updates" and make sure that an update matching the appropriate KB number above is included in the list of available updates.  Make sure the matching updates are selected, then choose to install the updates.  After installing updates you must reboot your computer.

Due to the performance impact of Windows updates, Microsoft has not fully mitigated the vulnerabilities by default.  Additional actions may be required depending upon the types of services running on your server.  Please see the guidance article from Microsoft Support for further steps to protect your server.

In order to fully patch Spectre variant 2, you also need to apply microcode updates.  These are available for some systems inside of Windows Server 2016, but the hotfix is not available via Windows update.  Please see this article for information on how to obtain these hotfixes, please see this article.

CentOS

Package updates for CentOS 7 and CentOS 6 are available as of January 4th.  CentOS 5 and older are End of Life and no fixes will be provided for these versions.  You should keep applying newer kernel updates as they become available to ensure you have the latest versions of Meltdown and Spectre fixes.

Currently, two packages need to be updated.  You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions.  Effective in May 2018, Red Hat published updated microcode_ctl packages which include Intel processor microcode updates which should be applied to enhance mitigation of Spectre variant 2.

• CentOS 7
  • kernel: 3.10.0-693.21.1.el7
    
    • IS NOT safe to run on the Steadfast public cloud
    • IS safe to run on other types of servers
  • kernel-plus: 3.10.0-693.21.1.el7.centos.plus
  • • IS safe to run on the Steadfast public cloud
    • IS safe to run on other types of servers
  • microcode_ctl: 2.1-29.2.el7_5.x86_64
• CentOS 6
  • kernel: 2.6.32-696.20.1.el6
    • IS safe to run on the Steadfast public cloud
    • IS safe to run on other types of servers
  • microcode_ctl: 1.17-25.6.el6_9.x86_64

To check the current versions of the packages run the command:

rpm -q kernel kernel-plus microcode_ctl

To apply all available updates to your server (recommended), run the following commands:

yum clean metadata
yum update

If you would rather limit the update to the specific packages relevant to this issue, run these commands instead:

yum clean metadata
yum update kernel\* microcode_ctl

If you are running CentOS 7 in a VM, make sure an update for the "kernel-plus" package is available and matches the safe version number indicated above.  if it does not, please run the following command first, then repeat the two commands above:

yum -y install http://mirror.steadfast.net/centos-steadfast/7/x86_64/steadfast-release-1-2.noarch.rpm

Make sure that the version numbers of the packages to be installed are greater than or equal to the numbers listed above.  After the update process completes, you must reboot your server by running the reboot command for the changes to take effect.  After rebooting, run the command uname -r to verify that your running kernel version matches the one in the table above.

If you are running KVM-based virtual machines on your server, you should also update the libvirt and QEMU packages.  The updated versions are as follows:

• CentOS 7
  • libvirt: 3.2.0-14.el7_4.7
  • qemu: 1.5.3-141.el7_4.6
• CentOS 6
  • libvirt: 0.10.2-62.el6_9.1
  • qemu: 0.12.1.2-2.503.el6_9.4

To update these packages, run the following commands, or simply run a full system update as noted above:

yum clean metadata
yum update libvirt\* qemu\*

It isn't necessary to reboot the server after applying these particular updates, but you must restart your virtual machines for the new protections to be effective.  Don't forget to apply the full set of updates inside your virtual machines as well.

If you need any assistance with any part of this process, please contact support and we'll be happy to assist you.

CloudLinux

CloudLinux 7 and 6 patches are now available for production use. CloudLinux servers are based upon CentOS, so to apply the updates, follow the same process as with CentOS indicated above, but note that the following kernel versions should be included rather than the CentOS versions:

• CloudLinux 7: 3.10.0-714.10.2.lve1.4.79.el7
• CloudLinux 6 Hybrid: 3.10.0-714.10.2.lve1.4.79.el6h
  
• CloudLinux 6: 2.6.32-896.16.1.lve1.4.51

After applying updates, you must reboot for the changes to become effective.

Debian

Debian has released an update which only addresses CVE-2017-5754 (Meltdown).  Debian releases prior to 7 (Wheezy) are End of Life and will not be fixed.

You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions.

• Debian 9 (Stretch): 4.9.65-3+deb9u2
• Debian 8 (Jessie): 3.16.51-3+deb8u1
• Debian 7 (Wheezy): 3.2.96-3

To check the current versions of the packages run the command:

dpkg -s linux-image-`uname -r` | grep Version

To apply all available updates to your server (recommended), run the following commands:

apt-get update
apt-get upgrade

If you would rather limit the update to the specific packages relevant to this issue, run these commands instead:

apt-get update
apt-get install linux-image-`uname -r`

Make sure an updated package for linux-image is included among the updates.  After the update process completes, you must reboot your server by running the reboot command for the changes to take effect.  After rebooting, run the command uname -v to verify that your running kernel version matches the one in the table above.

If the portion of the version number of your linux-image package before the - symbol is substantially different than the version above, you may first need to make sure your are using the latest kernel series for your version of Debian.  For this case, first run these commands, filling in the number from the list below in place of the word "version", then reboot your server:

• Debian 9 (Stretch): 4.9.0-5
• Debian 8 (Jessie): 3.16.0-5
• Debian 7 (Wheezy): 3.2.0-5

apt-get update
apt-get install linux-image-version-`dpkg --print-architecture`

If you need any assistance with any part of this process, please contact support and we'll be happy to assist you.

KernelCare

KernelCare is currently providing patches for CentOS 6 and 7 kernels that fix part of Spectre (CVE-2017-5753) and fix Meltdown (CVE-2017-5754).  If you have automatic updates enabled (the default) you should receive the fixes automatically.  If you observe any stability or behavioral issues with severs that have been patched, you may want to apply security fixes using the normal process for your operating system noted above instead.  You must reboot after applying package updates.

VMware

Updated VMware ESXi releases are available for currently supported products as of January 3rd:

• ESXi 6.5: ESXi650-201712101-SG
• ESXi 6.0: ESXi600-201711101-SG
• ESXi 5.5: ESXi550-201709101-SG (This patch does not address CVE-2017-5753)

The recommended procedure for updating VMware is using the vSphere Update Manager.  If you need assistance with applying software updates for VMware products licensed through Steadfast, please contact our support team.  Don't forget to apply the relevant security updates to your virtual machine operating systems as well.

Steadfast Public Cloud

The latest information regarding our Public Cloud platform is now located in a separate article.  Please check here for further information.

Hardware Patches (BIOS and Microcode Updates)

Intel provided microcode updates for all CPUs that Steadfast currently supports.  This does not include Pentium 4, Pentium D, Celeron, Core 2 Quad, Core 2 Duo, or any Xeon lines prior to the E55xx series, which are considered EOL by Steadfast.  If you have a server with a CPU that is not supported, please contact our sales team to arrange an upgrade.

Steadfast uses exclusively Supermicro motherboards for servers.  Supermicro has disclosed a list of products for which provide CPU code fixes.  See the Supermicro advisory for status information.

Currently motherboards with code updates available include the X11, X10, and some X8 series.  X9 motherboard BIOS updates have not yet been made available.  Steadfast provides newly ordered servers running the latest available BIOS code, which will include the relevant microcode patches if they are available.

For customers running CentOS or Windows Server 2016, it is not necessary to apply the BIOS update, provided you take OS-level action instead.  OS-provided microcode updates include the latest set of Intel code for CPUs that have already been fixed, and this code will be applied each time you reboot your server after the package has been installed. A more permanent BIOS fix is optional in this situation. See the above sections for CentOS and Windows for information on how to apply the current microcode updates to your servers.  Microcode updates are not applied by other supported operating systems.

Due to the logistics and risks of updating BIOS code on all servers operating in the data centers and because most of our customers run versions of Windows or Linux with OS-level microcode patching support, we have decided not to proactively update BIOS versions on any hardware that is already provisioned.  If you are running a system which cannot apply OS-level microcode updates or would like extra peace of mind from having a BIOS update that contains the microcode, please reach out to support to request a version check and update.  Please note that BIOS updates require downtime and may be risky, so we advise avoiding the update unless it is necessary.

References

The following references may be useful reading:

• Spectre and Meltdown Information Web Site
• Red Hat Spectre and Meltdown Vulnerability Guidance
• Microsoft Spectre and Meltdown Guidance
• Microsoft Security Advisory ADV180002
• CloudLinux / KernelCare Status Report
• Xen Security Advisory XSA-254
• VMware Security Advisory VMSA-2018-0002
• Red Hat Performance Impact Information
• Red Hat Performance Tuning Information
• Debian Security Trackers
  • CVE-2017-5754
  • CVE-2017-5753
  • CVE-2017-5715
• Intel Security Advisory INTEL-SA-00088
• Supermicro Hardware Advisory
• Steadfast Public Cloud Meltdown & Spectre Information

If you have any questions, please contact our support team and we'll do our best to help you.

Document Revisions

We will update this article with further information on how to obtain and apply patches to affected systems as soon as it is available.  The following changes have been made since the original posting.

May 25, 2018 @ 5:35 PM CDT:

• Added a note about Speculative Store Bypass (CVE-2018-3639) to indicate it is not covered in this article
• Added updated information about microcode updates available for CentOS, Windows Server 2016, and Supermicro motherboards

Mar 13, 2018 @ 4:50 PM CDT:

• Updated section for CentOS indicating the main kernel-plus package is now safe on the Steadfast public cloud
• Updated the patch information for KernelCare to indicate CentOS 6 is available.

Feb 7, 2018 @ 6:30 PM CST:

• Updated section for CentOS indicating a patched CentOS 7 kernel works on the Steadfast public cloud
• Updated the patch information for KernelCare

Jan 26, 2018 @ 3:30 PM CST:

• Updated section for CentOS indicating the new CentOS 6 kernel works on the Steadfast public cloud
• Modified CentOS package list to indicate that microcode packages are not longer recommended as part of mitigation

Jan 18, 2018 @ 3:30 PM CST:

• Added link to separate Steadfast Public Cloud article
• Updated KernelCare and CloudLinux information

Jan 16, 2018 @ 4:45 PM CST:

• Added Supermicro advisory link and hardware patching section

Jan 10, 2018 @ 2:15 PM CST:

• Added Debian 7 and 8 information

Jan 5, 2018 @ 2:45 PM CST:

• Added Intel security advisory link

Jan 5, 2018 @ 2:15 PM CST:

• Added a warning about the kernel updates for CentOS being unsafe on the public cloud
• Updated CloudLinux and KernelCare sections with new information
• Added a note to CentOS section indicating that future package updates are expected
• Added Debian patch information for Stretch and security tracker links

Jan 4, 2018 @ 6:30 PM CST:

• Added Red Hat guidance links discussing performance and tuning options

Jan 4, 2018 @ 5:30 PM CST:

• Added Microsoft Windows update information and advisory links

Jan 4, 2018 @ 5:00 PM CST:

• Added CentOS 6 update information
• Added CentOS update information for QEMU and libvirtd
• Updated CloudLinux package information
• Added VMware information and advisory link

Jan 4, 2018 @ 11:30 AM CST:

• Added a Patching section with a note about performance impact
• Added CentOS 7 update information
• Added Windows update information for Windows 10
• Added additional details for all supported Linux distributions
• Added a statement about KernelCare patch expectations
• Added a link to the Red Hat vulnerability summary page

We have scheduled maintenance to our NetApp SAN storage platform to replace a faulty controller card.  The NetApp SAN is used to power the storage for our Xen Public Cloud and customer accessible SAN services hosted out of chi18. This maintenance does not affect the Datera SAN hosted in chi02.

Date: Sunday, December 17th, 2017
Start Time: 12:00 AM CST (GMT -6) 
End Time: 3:00 AM CST (GMT -6)
Maintenance Scope: NetApp SAN storage controllers
Customer Impact: Customers may notice brief slow performance or IO hangs. If a failure occurs, mounted filesystems could require remounting or filesystem checking.

Due to the redundant nature of the SAN platform, the entire load will be transferred to a single controller box while the other is upgraded. This process will be performed for each controller and performance and stability will be tested before and after each change before continuing. When the load is transferred using a normal "takeover," there should be no interruption to service, though periods of slow response times may occur. While we expect the transfer of load to be seamless, if a failure condition should occur that may impact filesystem integrity, we will stop maintenance and proactively correct issues.

If you have any questions or need assistance during the upgrade, please contact us via our helpdesk.

Update 4:50 PM CST: It appears that the Comcast issues have been largely resolved at this time.  Based on the information we were able to monitor throughout the day, it appears this issue likely affected major backbone provider Level(3) and a variety of internet service providers that use Level(3) for parts of their national networks.  Comcast was more substantially affected by the events due to the very large number of end users and apparent significant reliance on Level(3).

We are currently tracking a serious routing issue within the Comcast Xfinity network which started around 11:54 AM CST that may result in poor performance or complete inability to reach some services on Steadfast's network as well as many other points around the Internet.  Our network engineers are investigating options to mitigate or work around this issue.  However, the best chance of full resolution lies with Comcast's network engineering team.  If it is possible to use a backup or alternate service provider to reach your services, such as a mobile network, we recommend doing this temporarily until Comcast services are repaired.  If you are a Comcast customer, please reach out to Comcast customer support to report the issue and obtain information on expected resolution times.

You can view ongoing Comcast user reports of this issue on Down Detector's Comcast status page.

We have scheduled an upgrade of our cloud hosting control panel server (https://vm.steadfast.net).  This maintenance will upgrade the software to a new major version which fixes many bugs, adds new features and improves performance.

Date: Sunday, November 12th, 2017
Start Time: 12:00 AM CST (GMT -6)
End Time: 4:00 AM CST (GMT -6)
Maintenance Scope: Cloud Hosting Control Panel at https://vm.steadfast.net

Customer Impact:

Access to information about and control of VMs will be unavailable for part of this maintenance.  Both customers and staff will be unable to provision, control, or view system details of VMs and related resources.  No impact is expected to running VMs and support and billing systems will remain available for the entire window.

Some notable user-visible changes will occur in this upgrade:

• The dashboard layout has been redesigned to move some stats to the right side of the screen.
• When viewing lists of items, such as VMs, you can now customize which columns are shown.
• A new "Service Catalog" option on the left menu provides buttons for creating supported resource types.
• The template selection screen during VM creation has been redesigned.
• The hostname and domain name are now separate fields during VM creation.
• XFS is now supported as a file system type for Linux VM disks.
• It is now possible to temporarily reassign a disk from one VM to another for data transfer or recovery purposes.

There are some new features added in this release which will not be enabled immediately upon upgrade.  We will provide additional information to customers if and when we decide to enable any of these in the future.

If you have any questions or need assistance during the upgrade, please visit our help desk or email us.

We have scheduled an upgrade to our management portal to a new major version.

Date: Monday, October 9th, 2017
Start Time: 11:00 AM CDT (GMT -5) 
End Time: 2:00 PM CDT (GMT -5) 
Maintenance Scope: Management Portal at https://manage.steadfast.net

Customer Impact:

No customer services or systems will be impacted. Access to billing information, bandwidth graphs, remote reboot control and account management will be unavailable at points during the maintenance period. We do not expect this upgrade to require the entire maintenance period.

This upgrade will include minor UI changes, as well as significant changes to back-end functionality. These changes will allow us to manage customer accounts and services more effectively.

Significant user-visible changes include:

• A new permissions model allows more granular control of what individual contacts on an account may do.
• Two-factor authenticationis now supported via mobile apps such as Google Authenticator (TOTP).
• The DNS Zone manager now appears under the Devices section, rather than under Domains.
• Many general optimizations to the application and database should bring improved performance.

If you have any questions, please feel free to contact us via our helpdesk or by email.