Spectre/Meltdown VM Remediation
Posted: 2018-03-29 10:41:42 Expiration: 2018-07-02 10:41:42
The Campus Computing Infrastructure – Virtualization Team has been working to mitigate the risks associated with the Spectre/Meltdown vulnerabilities in accordance with Campus Cyber Security directives. While the technical staff has recently applied the appropriate patches to the virtualization hosts, each of the Virtual Machines hosted within the environment must be restarted. The virtual machine must be power cycled, powered off and powered back on again, for the remediation to be successful. A reboot of the virtual machine is not sufficient. Customers may refer to the following KB for details: https://kb.wisc.edu/cci/52026. A power cycle will be referred to as a restart throughout the remainder of this document.
Background
Public cloud providers, such as Amazon, Google, and Microsoft, are scheduling and forcing restarts on customer virtual machines to complete the Spectre/Meltdown remediation. Service leadership has determined that CCI Virtualization customers should have the opportunity to control the restarts themselves as a way to limit the risk of service interruptions due to forced restarts. The Customer Remediation Period, lasting a little more than one month, will be provided for customers to plan and execute the restart of their virtual machines. Virtual machines that have already been remediated will not be required to perform an additional restart. The CCI Virtualization Team will provide a regularly updated list of virtual machines that still require remediation.
Performance Impact
Spectre/Meltdown Performance Impact - https://kb.wisc.edu/cci/news.php?id=9258
Implementation
The CCI Virtualization Team will schedule the forced restart of all un-remediated virtual machines at a time following the Customer Remediation Period. Virtual machines that have a maintenance window specified in WiscIT will have their restarts scheduled during those times. Any virtual machines that do not have a maintenance window will be restarted during the late evening or early morning hours as needed. Any department that has not established maintenance windows for their virtual machines may do so by contacting the CCI Virtualization Team via email (cci-virtualization@wisc.edu). Virtual machines that have been scheduled for a forced restart that is remediated by the customer will be removed from the schedule and will not be required to undergo an additional restart. Customers may also request that the scheduled restart of a particular virtual machine be changed to a different data or time by following the instructions in KB <kb number>.
Timeline
The following schedule has been established by the CCI Virtualization Team in consultation with Service Sponsorship, Campus Cyber Security, and Campus Leadership:
- 28 March – 04 May: Customer Remediation Period – This time is being provided for CCI Virtualization Customers to plan and execute the restart of all virtual machines that have not already been remediated
- 06 April: Initial Customer Forced Restart Schedule published – Virtual machines that have not been remediated will be scheduled for a forced restart following the end of the Customer Remediation Period. Scheduled restarts occur during the virtual machine’s maintenance window as indicated in WiscIT. Virtual machines that are remediated by the customer prior to the scheduled restart will be removed from the schedule.
- 08 May: Finalization of the Forced Restart Schedule – No more changes to the Forced Restart Schedule will be allowed. However, hosts that are remediated by the customer after this date will still be removed from the schedule and not required to undergo another restart.
- 12 May – 01 June: Forced Restart Execution Period – Customer virtual machines that have not been remediated will be forced to restart during the time indicated on the Customer Forced Restart Schedule. Any virtual machine that is seen as been remediated will not be forced to undergo a restart.
If you have questions about these actions please contact the CCI Virtualization Service Team via email (cci-virtualization@wisc.edu).
References: CCI Virtualization - Powering a Virtual Machine On and Off - https://kb.wisc.edu/cci/52026
-- CCI: Drew Denson