This form logs you into your management portal account. To access your help desk account, click here and use the form to the right of the news.
This form logs you into your management portal account. To access your help desk account, click here and use the form to the right of the news.
|
Jan 3 |
Security Advisory: Meltdown and Spectre CPU Defects
Posted by Kevin Stange on 03 January 2018 06:04 PM |
|
Note: On May 21st, 2018 an additional vulnerability identified as variant 4, called Speculative Store Bypass, was disclosed as CVE-2018-3639. Mitigations for variant 4 are not discussed in this advisory. We are aware of a new series of issues affecting all modern CPU models which can allow for applications to potentially access information and execute code that should not be allowed. Some reports of these problems began to surface in the media starting around January 1st, but the reports were incomplete. This class of vulnerabilities is reported to affect all operating systems and CPU types, including servers, home computers, and likely mobile and embedded devices. This situation is serious and affects everyone, so we are making it a top priority to gather and communicate information as soon as we have it, and we are committed to ensuring mitigations are in place as soon as possible. The vulnerabilities have been labeled CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754 in the Common Vulnerabilities and Exposures database. They have also been given the nicknames Meltdown and Spectre. Due to mistakes in the responsible disclosure process, some information about the issues was unintentionally disclosed early. Consequently the reporters decided to move up the formal disclosure date to January 3rd from the following week. As a result of the shortened preparation window by major vendors, most vendors did not have mitigation patches available by the disclosure date. PatchingThe following information describes patching processes for various operating systems. Please find the section below matching your operating system to learn how to apply the relevant fixes for it. Note that in addition to the fixes provided in your operating system, hardware changes may also be needed. At this time, very little information is available as to which hardware will be fixed. See the hardware fixes section for information about progress. Important! Due to the nature of the issue, some of the patches may negatively impact the performance of your server. This happens because the vulnerable behavior is part of a performance optimization feature. Removing the optimization helps mitigate the problem but also eliminates the performance benefit of the optimization. If you experience significant performance issues with your applications after applying updates, please check with your application vendor for advice or contact our support team so we can work with you to look for ways to counter the performance loss. Windows ServerMicrosoft has produced updates for the following supported Windows server versions effective January 3rd:
Windows Server 2008 and 2012 (non-R2 variants) do not currently have available updates. Windows Server 2003 is End of Life and will not be updated. To protect computer systems running Windows desktop or server operating systems, open the Windows Update application for your version of Windows. Select the option "Check for Updates" and make sure that an update matching the appropriate KB number above is included in the list of available updates. Make sure the matching updates are selected, then choose to install the updates. After installing updates you must reboot your computer. Due to the performance impact of Windows updates, Microsoft has not fully mitigated the vulnerabilities by default. Additional actions may be required depending upon the types of services running on your server. Please see the guidance article from Microsoft Support for further steps to protect your server. In order to fully patch Spectre variant 2, you also need to apply microcode updates. These are available for some systems inside of Windows Server 2016, but the hotfix is not available via Windows update. Please see this article for information on how to obtain these hotfixes, please see this article. CentOSPackage updates for CentOS 7 and CentOS 6 are available as of January 4th. CentOS 5 and older are End of Life and no fixes will be provided for these versions. You should keep applying newer kernel updates as they become available to ensure you have the latest versions of Meltdown and Spectre fixes. Currently, two packages need to be updated. You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions. Effective in May 2018, Red Hat published updated microcode_ctl packages which include Intel processor microcode updates which should be applied to enhance mitigation of Spectre variant 2.
To check the current versions of the packages run the command: rpm -q kernel kernel-plus microcode_ctl To apply all available updates to your server (recommended), run the following commands: yum clean metadata If you would rather limit the update to the specific packages relevant to this issue, run these commands instead: yum clean metadata If you are running CentOS 7 in a VM, make sure an update for the "kernel-plus" package is available and matches the safe version number indicated above. if it does not, please run the following command first, then repeat the two commands above: yum -y install http://mirror.steadfast.net/centos-steadfast/7/x86_64/steadfast-release-1-2.noarch.rpm Make sure that the version numbers of the packages to be installed are greater than or equal to the numbers listed above. After the update process completes, you must reboot your server by running the If you are running KVM-based virtual machines on your server, you should also update the libvirt and QEMU packages. The updated versions are as follows:
To update these packages, run the following commands, or simply run a full system update as noted above: yum clean metadata It isn't necessary to reboot the server after applying these particular updates, but you must restart your virtual machines for the new protections to be effective. Don't forget to apply the full set of updates inside your virtual machines as well. If you need any assistance with any part of this process, please contact support and we'll be happy to assist you. CloudLinuxCloudLinux 7 and 6 patches are now available for production use. CloudLinux servers are based upon CentOS, so to apply the updates, follow the same process as with CentOS indicated above, but note that the following kernel versions should be included rather than the CentOS versions:
After applying updates, you must reboot for the changes to become effective. DebianDebian has released an update which only addresses CVE-2017-5754 (Meltdown). Debian releases prior to 7 (Wheezy) are End of Life and will not be fixed. You can check the version numbers in this list to verify they are up to date. The highlighted numbers are the ones that will have changed from previous package versions.
To check the current versions of the packages run the command: dpkg -s linux-image-`uname -r` | grep Version To apply all available updates to your server (recommended), run the following commands: apt-get update If you would rather limit the update to the specific packages relevant to this issue, run these commands instead: apt-get update Make sure an updated package for linux-image is included among the updates. After the update process completes, you must reboot your server by running the If the portion of the version number of your linux-image package before the - symbol is substantially different than the version above, you may first need to make sure your are using the latest kernel series for your version of Debian. For this case, first run these commands, filling in the number from the list below in place of the word "version", then reboot your server:
apt-get update
If you need any assistance with any part of this process, please contact support and we'll be happy to assist you. KernelCareKernelCare is currently providing patches for CentOS 6 and 7 kernels that fix part of Spectre (CVE-2017-5753) and fix Meltdown (CVE-2017-5754). If you have automatic updates enabled (the default) you should receive the fixes automatically. If you observe any stability or behavioral issues with severs that have been patched, you may want to apply security fixes using the normal process for your operating system noted above instead. You must reboot after applying package updates. VMwareUpdated VMware ESXi releases are available for currently supported products as of January 3rd:
The recommended procedure for updating VMware is using the vSphere Update Manager. If you need assistance with applying software updates for VMware products licensed through Steadfast, please contact our support team. Don't forget to apply the relevant security updates to your virtual machine operating systems as well. Steadfast Public CloudThe latest information regarding our Public Cloud platform is now located in a separate article. Please check here for further information. Hardware Patches (BIOS and Microcode Updates)Intel provided microcode updates for all CPUs that Steadfast currently supports. This does not include Pentium 4, Pentium D, Celeron, Core 2 Quad, Core 2 Duo, or any Xeon lines prior to the E55xx series, which are considered EOL by Steadfast. If you have a server with a CPU that is not supported, please contact our sales team to arrange an upgrade. Steadfast uses exclusively Supermicro motherboards for servers. Supermicro has disclosed a list of products for which provide CPU code fixes. See the Supermicro advisory for status information. Currently motherboards with code updates available include the X11, X10, and some X8 series. X9 motherboard BIOS updates have not yet been made available. Steadfast provides newly ordered servers running the latest available BIOS code, which will include the relevant microcode patches if they are available. For customers running CentOS or Windows Server 2016, it is not necessary to apply the BIOS update, provided you take OS-level action instead. OS-provided microcode updates include the latest set of Intel code for CPUs that have already been fixed, and this code will be applied each time you reboot your server after the package has been installed. A more permanent BIOS fix is optional in this situation. See the above sections for CentOS and Windows for information on how to apply the current microcode updates to your servers. Microcode updates are not applied by other supported operating systems. Due to the logistics and risks of updating BIOS code on all servers operating in the data centers and because most of our customers run versions of Windows or Linux with OS-level microcode patching support, we have decided not to proactively update BIOS versions on any hardware that is already provisioned. If you are running a system which cannot apply OS-level microcode updates or would like extra peace of mind from having a BIOS update that contains the microcode, please reach out to support to request a version check and update. Please note that BIOS updates require downtime and may be risky, so we advise avoiding the update unless it is necessary. ReferencesThe following references may be useful reading:
If you have any questions, please contact our support team and we'll do our best to help you. Document RevisionsWe will update this article with further information on how to obtain and apply patches to affected systems as soon as it is available. The following changes have been made since the original posting. May 25, 2018 @ 5:35 PM CDT:
Mar 13, 2018 @ 4:50 PM CDT:
Feb 7, 2018 @ 6:30 PM CST:
Jan 26, 2018 @ 3:30 PM CST:
Jan 18, 2018 @ 3:30 PM CST:
Jan 16, 2018 @ 4:45 PM CST:
Jan 10, 2018 @ 2:15 PM CST:
Jan 5, 2018 @ 2:45 PM CST:
Jan 5, 2018 @ 2:15 PM CST:
Jan 4, 2018 @ 6:30 PM CST:
Jan 4, 2018 @ 5:30 PM CST:
Jan 4, 2018 @ 5:00 PM CST:
Jan 4, 2018 @ 11:30 AM CST:
Read more » | |
|
Dec 4 |
NetApp SAN & Public Cloud Storage Maintenance
Posted by Kevin Stange on 04 December 2017 01:01 PM |
|
We have scheduled maintenance to our NetApp SAN storage platform to replace a faulty controller card. The NetApp SAN is used to power the storage for our Xen Public Cloud and customer accessible SAN services hosted out of chi18. This maintenance does not affect the Datera SAN hosted in chi02. Date: Sunday, December 17th, 2017 Due to the redundant nature of the SAN platform, the entire load will be transferred to a single controller box while the other is upgraded. This process will be performed for each controller and performance and stability will be tested before and after each change before continuing. When the load is transferred using a normal "takeover," there should be no interruption to service, though periods of slow response times may occur. While we expect the transfer of load to be seamless, if a failure condition should occur that may impact filesystem integrity, we will stop maintenance and proactively correct issues. If you have any questions or need assistance during the upgrade, please contact us via our helpdesk. Read more » | |
|
Nov 6 |
Resolved: Comcast Connectivity Issues - November 6th, 2017
Posted by Kevin Stange on 06 November 2017 12:44 PM |
|
Update 4:50 PM CST: It appears that the Comcast issues have been largely resolved at this time. Based on the information we were able to monitor throughout the day, it appears this issue likely affected major backbone provider Level(3) and a variety of internet service providers that use Level(3) for parts of their national networks. Comcast was more substantially affected by the events due to the very large number of end users and apparent significant reliance on Level(3). We are currently tracking a serious routing issue within the Comcast Xfinity network which started around 11:54 AM CST that may result in poor performance or complete inability to reach some services on Steadfast's network as well as many other points around the Internet. Our network engineers are investigating options to mitigate or work around this issue. However, the best chance of full resolution lies with Comcast's network engineering team. If it is possible to use a backup or alternate service provider to reach your services, such as a mobile network, we recommend doing this temporarily until Comcast services are repaired. If you are a Comcast customer, please reach out to Comcast customer support to report the issue and obtain information on expected resolution times. You can view ongoing Comcast user reports of this issue on Down Detector's Comcast status page. Read more » | |
|
Oct 31 |
Cloud Hosting: Control Panel Upgrade
Posted by Kevin Stange on 31 October 2017 03:17 PM |
|
We have scheduled an upgrade of our cloud hosting control panel server (https://vm.steadfast.net). This maintenance will upgrade the software to a new major version which fixes many bugs, adds new features and improves performance. Date: Sunday, November 12th, 2017 Customer Impact: Access to information about and control of VMs will be unavailable for part of this maintenance. Both customers and staff will be unable to provision, control, or view system details of VMs and related resources. No impact is expected to running VMs and support and billing systems will remain available for the entire window. Some notable user-visible changes will occur in this upgrade:
There are some new features added in this release which will not be enabled immediately upon upgrade. We will provide additional information to customers if and when we decide to enable any of these in the future. If you have any questions or need assistance during the upgrade, please visit our help desk or email us. Read more » | |
|
Sep 27 |
Management Portal Upgrade
Posted by Kevin Stange on 27 September 2017 04:56 PM |
|
We have scheduled an upgrade to our management portal to a new major version. Date: Monday, October 9th, 2017 Customer Impact: This upgrade will include minor UI changes, as well as significant changes to back-end functionality. These changes will allow us to manage customer accounts and services more effectively. Significant user-visible changes include:
If you have any questions, please feel free to contact us via our helpdesk or by email. Read more » | |
|
Sep 15 |
Hybrid Performance SAN Maintenance (Chicago, IL)
Posted by Marc Schulz on 15 September 2017 04:45 PM |
|
We have scheduled a software upgrade of our SAN infrastructure in Chicago, IL. This is purely a backend upgrade and does not change interactions with client environments. Date: Sunday, September 24th, 2017 Maintenance Scope: Hybrid Performance SAN (350 E. Cermak Chicago) Customer Impact: There is no expected impact during this maintenance as our SAN infrastructure is fully redundant. While we don’t expect any issues we recommend taking a backup prior to the start of this window. This maintenance does not impact our NetApp SAN (Public Cloud Infrastructure) If you have any questions or concerns please feel free to reach out to our support team at support@steadfast.net. 09/24/17 1:49am CST: This maintenance completed successfully. Read more » | |