Virtualizing servers and applications is nothing new, and is often considered common practice these days. More and more Tier 1 applications – Domain Controllers, Microsoft Exchange Server, Microsoft SQL Server, and VDI Desktops – are provisioned as Virtual Machines, due to their ability to be agile and highly available. With continued advancements at the hypervisor and storage levels – including VM high availability, distributed/dynamic scheduling, and scale-out, shared-nothing architectures, infrastructure is much more resilient than ever. However, human error is still a VERY common factor in many unplanned outages. Out of the box, Microsoft Windows desktop and server OSes allow any user the ability to eject various HotPlug devices, which could lead to a server or desktop being immediately disconnected and unavailable.
The Windows feature to ‘Safely Remove Hardware and Eject Media’ sounds good in theory, say to eject a USB drive from the operating system. Thanks to Windows Search Index, the OS will often prevent you doing something really bad, such as ejecting a virtualized SCSI attached disk. Nonetheless, ejecting a virtualized network adapter of a production VM seems like a great way to ruin your day.
Now, yes, you can set a Microsoft Group Policy Object to limit the users and groups who are allowed to eject removable devices. https://technet.microsoft.com/en-us/library/jj852171(v=ws.11).aspx
GPO_Name\Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options
This STILL does not prevent an administrator (you know, the ones often working on servers) from accidentally clicking the wrong icon.
Well-respect consultant and developer Helge Klein released a great (and frequently referenced!) post in 2012, detailing the ability to disable HotPlug functionality in VMware ESX – both at the hypervisor level, along within the Windows Guest OS.
Nutanix
Since the publishing of Helge’s article, additional hypervisors have been released and adopted in the marketplace. Nutanix may have started the explosion on the scene as a storage disruptor, while making traditional three-tier architecture obsolete. In doing so, Nutanix still allowed a choice of industry standard hypervisors, VMware ESX or Microsoft Hyper-V, to run on top of their commodity hardware. But Nutanix did not stop at simply disrupting the storage market, they wanted to ensure a practice known as ‘Invisible Infrastructure for Enterprise Computing.’ In theory – the storage shouldn’t matter, the hypervisor shouldn’t matter – IT should be a transparent and turnkey solution that focuses on applications and on the business. At .NEXT, Nutanix’s first user conference in June 2015, another disruptive announcement was dropped – Nutanix would release their own hypervisor entitled Acropolis (AHV), based on Linux KVM. Only 1 year later at .NEXT 2016, Nutanix is now reporting 15% of all deployed clusters are running AHV for client workloads.
Placing key infrastructure workloads on Acropolis, such as Microsoft Exchange, SQL, and Citrix XenDesktop, and VMware View are vendor supported (and recommended, if I must say so myself) AND yield an incredible ROI to the business. However, proper precautions need to be addressed to ensure users or administrators cannot accidentally eject virtualized hardware.
Windows VMs running on AHV will have 3 HotPlug devices within the guest OS
- Nutanix VirtIO Ethernet Adapter
- Nutanix VirtIO Balloon Driver
- Nutanix VirtIO SCSI pass-through controller
Currently on Nutanix Acropolis, there is no supported way to disable HotPlug functionality at the hypervisor level. I have confirmed with Nutanix support and engineering that this is not something that is currently publicly exposed.
Currently, the best approach to disabling HotPlug devices within the guest OS would be to disable functionality from within the registry by changing flags on the ‘Capabilities’ key, based on each device within HKLM\SYSTEM\CCS\Enum\PCI.
A few notes regarding these changes:
- Security permissions for all keys under ‘HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI’ are protected so that only the SYSTEM account has Full Control.
- You could create a workaround by launching Regedit.exe as the SYSTEM account, leveraging a tool like Systernals PSExec (Owned by Microsoft)
- Command Prompt – cmd.exe – Run as Administrator
> psexec -i -d -s c:\windows\regedit.exe
- Command Prompt – cmd.exe – Run as Administrator
However, due to these keys being reverted upon machine restart, the best way to achieve this functionality is either via a computer startup script or Group Policy Preference registry keys – both which run under the SYSTEM account – and applied at computer boot.
Deployment – GPP
The simple and repeatable way to deploy these registry is using Group Policy Preference registry items
Update the REG_DWORD value of the Capabilities key to decimal value of (2) for the following keys:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1AF4&DEV_1000&SUBSYS_00011AF4&REV_00\3&13c0b0c5&0&18 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1AF4&DEV_1002&SUBSYS_00051AF4&REV_00\3&13c0b0c5&0&28 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1AF4&DEV_1004&SUBSYS_00081AF4&REV_00\3&13c0b0c5&0&20
Deployment – Startup Script
Alternatively, the following script can be saved as a batch file and dropped into a GPO applied to your VMs.
:: Disable Eject Nutanix VirtIO Ethernet Adapter (Win 10) :: Disable Eject Midfin Systems eFabric Network Adapter (W2K12 R2) :: Original value: 6 reg.exe add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1AF4&DEV_1000&SUBSYS_00011AF4&REV_00\3&13c0b0c5&0&18" /v Capabilities /t REG_DWORD /d 2 /f :: Disable Eject Nutanix VirtIO Balloon Driver (Win 10) :: Disable Eject PCI Device (W2K12 R2) :: Original value: 6 reg.exe add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1AF4&DEV_1002&SUBSYS_00051AF4&REV_00\3&13c0b0c5&0&28" /v Capabilities /t REG_DWORD /d 2 /f :: Disable Eject Nutanix VirtIO SCSI pass-through controller (Win 10 & W2K12 R2) :: Original value: 6 reg.exe add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1AF4&DEV_1004&SUBSYS_00081AF4&REV_00\3&13c0b0c5&0&20" /v Capabilities /t REG_DWORD /d 2 /f
Whichever methodology you choose – following this change, the HotPlug functionality of the Nutanix VirtIO devices should be removed – making users and admins sleep a little easier at night!