We have scheduled an emergency maintenance to restore a redundant link on an aggregation device that was previously causing performance issues. The optics for this link have been replaced and link stability tested to ensure stability.

Date: Friday, April 27th, 2018
Start Time: 1:00 AM CDT (GMT -5)
End Time: 2:00 AM CDT (GMT -5)

Maintenance Scope: chi01/02 Network Resources

Customer Impact:

Customers may experience a short 30 second loss of connectivity while spanning-tree reconverges. Once reconvergence is complete the link will be monitored for stability. Any additional instability will lead to us taking this link back out of service.

If you have any questions regarding this maintenance please feel free to contact us via our helpdesk or by email.

Update 1:20am CDT:  Maintenance Complete. No Issues were seen with the link post reintroduction. 

We have scheduled an upgrade of our R1Soft Server Backup (formerly Idera Server Backup or R1Soft CDP) platform software on cdp01.steadfast.net to version 6.6.1.  This upgrade fixes a number of bugs.

This upgrade is being performed during the daytime, mid-week, because it is the lowest utilization period.

Date: Tuesday, March 20th, 2018
Start Time: 1:00 PM CDT (GMT -5)
End Time: 2:00 PM CDT (GMT -5)
Maintenance Scope: cdp01.steadfast.net Backup Server

Customer Impact:

During the maintenance period, backups will not be performed and restoration services will be unavailable. This maintenance will not impact customer equipment or services other than backup tasks and does not affect private backup servers or customers using Veeam.

It is safe to upgrade the agent to version 6.6.1 before the manager has been upgraded.

The full official release notes may be viewed here.

If you have any questions regarding this maintenance, or for assistance in upgrading the agent on your server(s), please feel free to contact us via our helpdesk or by email.

We have scheduled a software upgrade of our SAN infrastructure in Chicago, IL. This is purely a backend upgrade and does not change interactions with client environments.

Date: Sunday, March 11th, 2018
Start Time: 1:00 AM CST (GMT -6)
End Time: 4:00 AM CDT (GMT -5)

Maintenance Scope: Hybrid Performance SAN (350 E. Cermak Chicago)

Customer Impact:

There is no expected impact during this maintenance as our SAN infrastructure is fully redundant. While we don’t expect any issues we recommend taking a backup prior to the start of this window. This maintenance does not impact our NetApp SAN (Public Cloud Infrastructure)

If you have any questions or concerns please feel free to reach out to our support team at support@steadfast.net.

Update: 2:15am CST (3:15am CDT) This maintenance has completed successfully. 

We are aware of a new series of issues affecting all modern CPU models which can allow for applications to potentially access information and execute code that should not be allowed.  Some reports of these problems began to surface in the media starting around January 1st, but the reports were incomplete.  This class of vulnerabilities is reported to affect all operating systems and CPU types, including servers, home computers, and likely mobile and embedded devices.  This situation is serious and affects everyone, so we are making it a top priority to gather and communicate information as soon as we have it, and we are committed to ensuring mitigations are in place as soon as possible.

The vulnerabilities have been labeled CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754 in the Common Vulnerabilities and Exposures database.  They have also been given the nicknames Meltdown and Spectre.

Due to mistakes in the responsible disclosure process, some information about the issues was unintentionally disclosed early.  Consequently the reporters decided to move up the formal disclosure date to January 3rd from the following week.  As a result of the shortened preparation window by major vendors, most vendors did not have mitigation patches available by the disclosure date.

Patching

The following information describes patching processes for various operating systems.  Please find the section below matching your operating system to learn how to apply the relevant fixes for it.

Note that in addition to the fixes provided in your operating system, hardware changes may also be needed.  At this time, very little information is available as to which hardware will be fixed.  See the hardware fixes section for information about progress.

Important! Due to the nature of the issue, some of the patches may negatively impact the performance of your server.  This happens because the vulnerable behavior is part of a performance optimization feature.  Removing the optimization helps mitigate the problem but also eliminates the performance benefit of the optimization.  If you experience significant performance issues with your applications after applying updates, please check with your application vendor for advice or contact our support team so we can work with you to look for ways to counter the performance loss.

Windows Server

Microsoft has produced updates for the following supported Windows server versions effective January 3rd:

• Windows Server 2016: Build 14393.2007 (KB4056890)
• Windows Server 2012 R2: KB4056898
• Windows Server 2008 R2: KB4056897 or KB4056894

Windows Server 2008 and 2012 (non-R2 variants) do not currently have available updates.  Windows Server 2003 is End of Life and will not be updated.

To protect computer systems running Windows desktop or server operating systems, open the Windows Update application for your version of Windows.  Select the option "Check for Updates" and make sure that an update matching the appropriate KB number above is included in the list of available updates.  Make sure the matching updates are selected, then choose to install the updates.  After installing updates you must reboot your computer.

Due to the performance impact of Windows updates, Microsoft has not fully mitigated the vulnerabilities by default.  Additional actions may be required depending upon the types of services running on your server.  Please see the guidance article from Microsoft Support for further steps to protect your server.

CentOS

Package updates for CentOS 7 and CentOS 6 are available as of January 4th.  CentOS 5 and older are End of Life and no fixes will be provided for these versions.  You should keep applying newer kernel updates as they become available to ensure you have the latest versions of Meltdown and Spectre fixes.

Currently, one package needs to be updated.  You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions. Previous versions of this document mentioned additional packages to update, but these packages contained firmware that has since been reverted due to stability concerns.  Updating to the latest microcode_ctl and linux-firmware (CentOS 7) packages will revert the microcode for Intel CPUs, which you should do if you are experiencing unexpected reboots.

• CentOS 7
  • kernel: 3.10.0-693.21.1.el7
    
    • IS NOT safe to run on the Steadfast public cloud
    • IS safe to run on other types of servers
  • kernel-plus: 3.10.0-693.21.1.el7.centos.plus
  • • IS safe to run on the Steadfast public cloud
    • IS safe to run on other types of servers
• CentOS 6
  • kernel: 2.6.32-696.20.1.el6
    • IS safe to run on the Steadfast public cloud
    • IS safe to run on other types of servers

To check the current versions of the packages run the command:

rpm -q kernel kernel-plus

To apply all available updates to your server (recommended), run the following commands:

yum clean metadata
yum update

If you would rather limit the update to the specific packages relevant to this issue, run these commands instead:

yum clean metadata
yum update kernel\*

If you are running CentOS 7 in a VM, make sure an update for the "kernel-plus" package is available and matches the safe version number indicated above.  if it does not, please run the following command first, then repeat the two commands above:

yum -y install http://mirror.steadfast.net/centos-steadfast/7/x86_64/steadfast-release-1-2.noarch.rpm

Make sure that the version numbers of the packages to be installed are greater than or equal to the numbers listed above.  After the update process completes, you must reboot your server by running the reboot command for the changes to take effect.  After rebooting, run the command uname -r to verify that your running kernel version matches the one in the table above.

If you are running KVM-based virtual machines on your server, you should also update the libvirt and QEMU packages.  The updated versions are as follows:

• CentOS 7
  • libvirt: 3.2.0-14.el7_4.7
  • qemu: 1.5.3-141.el7_4.6
• CentOS 6
  • libvirt: 0.10.2-62.el6_9.1
  • qemu: 0.12.1.2-2.503.el6_9.4

To update these packages, run the following commands, or simply run a full system update as noted above:

yum clean metadata
yum update libvirt\* qemu\*

It isn't necessary to reboot the server after applying these particular updates, but you must restart your virtual machines for the new protections to be effective.  Don't forget to apply the full set of updates inside your virtual machines as well.

If you need any assistance with any part of this process, please contact support and we'll be happy to assist you.

CloudLinux

CloudLinux 7 and 6 patches are now available for production use. CloudLinux servers are based upon CentOS, so to apply the updates, follow the same process as with CentOS indicated above, but note that the following kernel versions should be included rather than the CentOS versions:

• CloudLinux 7: 3.10.0-714.10.2.lve1.4.79.el7
• CloudLinux 6 Hybrid: 3.10.0-714.10.2.lve1.4.79.el6h
  
• CloudLinux 6: 2.6.32-896.16.1.lve1.4.51

After applying updates, you must reboot for the changes to become effective.

Debian

Debian has released an update which only addresses CVE-2017-5754 (Meltdown).  Debian releases prior to 7 (Wheezy) are End of Life and will not be fixed.

You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions.

• Debian 9 (Stretch): 4.9.65-3+deb9u2
• Debian 8 (Jessie): 3.16.51-3+deb8u1
• Debian 7 (Wheezy): 3.2.96-3

To check the current versions of the packages run the command:

dpkg -s linux-image-`uname -r` | grep Version

To apply all available updates to your server (recommended), run the following commands:

apt-get update
apt-get upgrade

If you would rather limit the update to the specific packages relevant to this issue, run these commands instead:

apt-get update
apt-get install linux-image-`uname -r`

Make sure an updated package for linux-image is included among the updates.  After the update process completes, you must reboot your server by running the reboot command for the changes to take effect.  After rebooting, run the command uname -v to verify that your running kernel version matches the one in the table above.

If the portion of the version number of your linux-image package before the - symbol is substantially different than the version above, you may first need to make sure your are using the latest kernel series for your version of Debian.  For this case, first run these commands, filling in the number from the list below in place of the word "version", then reboot your server:

• Debian 9 (Stretch): 4.9.0-5
• Debian 8 (Jessie): 3.16.0-5
• Debian 7 (Wheezy): 3.2.0-5

apt-get update
apt-get install linux-image-version-`dpkg --print-architecture`

If you need any assistance with any part of this process, please contact support and we'll be happy to assist you.

KernelCare

KernelCare is currently providing patches for CentOS 6 and 7 kernels that fix part of Spectre (CVE-2017-5753) and fix Meltdown (CVE-2017-5754).  If you have automatic updates enabled (the default) you should receive the fixes automatically.  If you cannot afford to wait for patches that are still pending, you should apply security fixes using the normal process for your operating system noted above in order to obtain the fixes more quickly.  You must reboot after applying package updates.

VMware

Updated VMware ESXi releases are available for currently supported products as of January 3rd:

• ESXi 6.5: ESXi650-201712101-SG
• ESXi 6.0: ESXi600-201711101-SG
• ESXi 5.5: ESXi550-201709101-SG (This patch does not address CVE-2017-5753)

The recommended procedure for updating VMware is using the vSphere Update Manager.  If you need assistance with applying software updates for VMware products licensed through Steadfast, please contact our support team.  Don't forget to apply the relevant security updates to your virtual machine operating systems as well.

Steadfast Public Cloud

The latest information regarding our Public Cloud platform is now located in a separate article.  Please check here for further information.

Hardware Patches (BIOS and Microcode Updates)

Steadfast uses exclusively Supermicro motherboards for servers.  Supermicro has disclosed a list of products for which they hope to provide CPU code fixes.  See the Supermicro advisory for status information.

Currently the only motherboards with code updates available are from the X11 series, which Steadfast has not yet widely deployed.  Until fixes are available for boards that are from the X10, X9, or X8 series, no BIOS updates are available.  It is unclear which boards will ultimately be fixed because Intel has not published the list of CPUs for which code updates will be provided.

For customers running CentOS, the microcode package (microcode_ctl) includes the latest set of Intel code for CPUs that have already been fixed, and this code will be applied each time you reboot your server after the package update has been installed.  A more permanent BIOS fix is optional in this situation.  Microcode updates are not automatically applied by other supported operating systems.

Due to the logistics and risks of updating BIOS code on all servers operating in the data centers and the lack of information as to which boards will be fixed, we have not yet produced a plan for how we will be making these fixes available to customers.  This section will be updated when more information is available.

Other Platforms

We are presently assessing the state of fixes with other supported platform vendors.

References

The following references may be useful reading:

• Spectre and Meltdown Information Web Site
• Red Hat Spectre and Meltdown Vulnerability Guidance
• Microsoft Spectre and Meltdown Guidance
• Microsoft Security Advisory ADV180002
• CloudLinux / KernelCare Status Report
• Xen Security Advisory XSA-254
• VMware Security Advisory VMSA-2018-0002
• Red Hat Performance Impact Information
• Red Hat Performance Tuning Information
• Debian Security Trackers
  • CVE-2017-5754
  • CVE-2017-5753
  • CVE-2017-5715
• Intel Security Advisory INTEL-SA-00088
• Supermicro Hardware Advisory
• Steadfast Public Cloud Meltdown & Spectre Information

If you have any questions, please contact our support team and we'll do our best to help you.

Document Revisions

We will update this article with further information on how to obtain and apply patches to affected systems as soon as it is available.  The following changes have been made since the original posting.

Mar 13, 2018 @ 4:50 PM CDT:

• Updated section for CentOS indicating the main kernel-plus package is now safe on the Steadfast public cloud
• Updated the patch information for KernelCare to indicate CentOS 6 is available.

Feb 7, 2018 @ 6:30 PM CST:

• Updated section for CentOS indicating a patched CentOS 7 kernel works on the Steadfast public cloud
• Updated the patch information for KernelCare

Jan 26, 2018 @ 3:30 PM CST:

• Updated section for CentOS indicating the new CentOS 6 kernel works on the Steadfast public cloud
• Modified CentOS package list to indicate that microcode packages are not longer recommended as part of mitigation

Jan 18, 2018 @ 3:30 PM CST:

• Added link to separate Steadfast Public Cloud article
• Updated KernelCare and CloudLinux information

Jan 16, 2018 @ 4:45 PM CST:

• Added Supermicro advisory link and hardware patching section

Jan 10, 2018 @ 2:15 PM CST:

• Added Debian 7 and 8 information

Jan 5, 2018 @ 2:45 PM CST:

• Added Intel security advisory link

Jan 5, 2018 @ 2:15 PM CST:

• Added a warning about the kernel updates for CentOS being unsafe on the public cloud
• Updated CloudLinux and KernelCare sections with new information
• Added a note to CentOS section indicating that future package updates are expected
• Added Debian patch information for Stretch and security tracker links

Jan 4, 2018 @ 6:30 PM CST:

• Added Red Hat guidance links discussing performance and tuning options

Jan 4, 2018 @ 5:30 PM CST:

• Added Microsoft Windows update information and advisory links

Jan 4, 2018 @ 5:00 PM CST:

• Added CentOS 6 update information
• Added CentOS update information for QEMU and libvirtd
• Updated CloudLinux package information
• Added VMware information and advisory link

Jan 4, 2018 @ 11:30 AM CST:

• Added a Patching section with a note about performance impact
• Added CentOS 7 update information
• Added Windows update information for Windows 10
• Added additional details for all supported Linux distributions
• Added a statement about KernelCare patch expectations
• Added a link to the Red Hat vulnerability summary page

We have scheduled maintenance to our NetApp SAN storage platform to replace a faulty controller card.  The NetApp SAN is used to power the storage for our Xen Public Cloud and customer accessible SAN services hosted out of chi18. This maintenance does not affect the Datera SAN hosted in chi02.

Date: Sunday, December 17th, 2017
Start Time: 12:00 AM CST (GMT -6) 
End Time: 3:00 AM CST (GMT -6)
Maintenance Scope: NetApp SAN storage controllers
Customer Impact: Customers may notice brief slow performance or IO hangs. If a failure occurs, mounted filesystems could require remounting or filesystem checking.

Due to the redundant nature of the SAN platform, the entire load will be transferred to a single controller box while the other is upgraded. This process will be performed for each controller and performance and stability will be tested before and after each change before continuing. When the load is transferred using a normal "takeover," there should be no interruption to service, though periods of slow response times may occur. While we expect the transfer of load to be seamless, if a failure condition should occur that may impact filesystem integrity, we will stop maintenance and proactively correct issues.

If you have any questions or need assistance during the upgrade, please contact us via our helpdesk.

Update 4:50 PM CST: It appears that the Comcast issues have been largely resolved at this time.  Based on the information we were able to monitor throughout the day, it appears this issue likely affected major backbone provider Level(3) and a variety of internet service providers that use Level(3) for parts of their national networks.  Comcast was more substantially affected by the events due to the very large number of end users and apparent significant reliance on Level(3).

We are currently tracking a serious routing issue within the Comcast Xfinity network which started around 11:54 AM CST that may result in poor performance or complete inability to reach some services on Steadfast's network as well as many other points around the Internet.  Our network engineers are investigating options to mitigate or work around this issue.  However, the best chance of full resolution lies with Comcast's network engineering team.  If it is possible to use a backup or alternate service provider to reach your services, such as a mobile network, we recommend doing this temporarily until Comcast services are repaired.  If you are a Comcast customer, please reach out to Comcast customer support to report the issue and obtain information on expected resolution times.

You can view ongoing Comcast user reports of this issue on Down Detector's Comcast status page.