TL;DR: You can’t upgrade. Do a clean install with EFI ( it will do bios by default) and enable secure boot for efi. Boot from the Windows 11 ISO and use the BypassSecureBootCheck and BypassTPMCheck registry settings. Then install as normal.

I have a Windows 10 VM on my Xserve 3,1 running ESXi 6.5 that I wanted to upgrade to Windows 11. There are 2 big issues to point out here though. The first is that the Nehalem CPU family, mine being a W5590, were dropped from the Win 11 support list. The second being that the Xserve lacks TPM 2.0 and ESXi didn’t start support for vTPM till ESXi 6.7.

I spent longer than I would like to admit trying to get Windows 10 to upgrade to Windows 11. I tried everything from modifying the registry and using the installer to getting an ISO and modding it. Try as a might I was unable to get it to update. I also tried doing it on VM Fusion, but I keep getting crazy errors. The closest I came to success was getting the install to pass pre-checks using the Windows11Upgrade project on GitHub. In the end though that resulted in a, “Windows Server Setup: there was a problem running preinstall.cmd scripts” error though.

As a last ditch effort I decided to do a clean install and finally found success. Below are the steps I followed:

  1. Go the Microsoft site and download the Windows 11 ISO. You don’t want the installer or the media creation tool, just download the ISO itself.
  2. Setup your virtual machine. Windows 11 requires a minimum of 2 virtual cores and at least 8GB of ram. With my install I used 2 vCores and 16GB for good measure.
  3. Set your machine to use EFI and “secure boot” in the “VM Options” tab. It will be set to Bios by default so don’t skip this step.
  4. Mount your ISO and boot it up.
  5. At boot like always hit any key to boot to the Windows installer.
  6. As soon as the installer environment boots press Shift + F10 to get the command prompt and type “regedit” followed by the return key.
  7. Once the Registry editor comes up navigate or jump to “HKEY_LOCAL_MACHINE\SYSTEM\Setup”
  8. Create a new folder here called “LabConfig”
  9. You will need to add 2 DWORD (32-bit) entries “BypassTPMCheck” and “BypassSecureBootCheck”
  10. Once you add these entries set them to “1”
  11. Close both regedit and the command prompt
  12. You should now be able to install the VM.

When I did my install I used Win 11 Pro so depending on what you are installing your mileage may vary, however this should get any Win 11 VM installed. Like with any Win VM once you have it up and running Install VMware Tools and run windows update. Even though I downloaded the ISO today it still had a small stack of updates it downloaded and updated.