Installing Windows 11 on ESXi 6.5

TL;DR: You can’t upgrade. Do a clean install with EFI ( it will do bios by default) and enable secure boot for efi. Boot from the Windows 11 ISO and use the BypassSecureBootCheck and BypassTPMCheck registry settings. Then install as normal.

I have a Windows 10 VM on my Xserve 3,1 running ESXi 6.5 that I wanted to upgrade to Windows 11. There are 2 big issues to point out here though. The first is that the Nehalem CPU family, mine being a W5590, were dropped from the Win 11 support list. The second being that the Xserve lacks TPM 2.0 and ESXi didn’t start support for vTPM till ESXi 6.7.

I spent longer than I would like to admit trying to get Windows 10 to upgrade to Windows 11. I tried everything from modifying the registry and using the installer to getting an ISO and modding it. Try as a might I was unable to get it to update. I also tried doing it on VM Fusion, but I keep getting crazy errors. The closest I came to success was getting the install to pass pre-checks using the Windows11Upgrade project on GitHub. In the end though that resulted in a, “Windows Server Setup: there was a problem running preinstall.cmd scripts” error though.

As a last ditch effort I decided to do a clean install and finally found success. Below are the steps I followed:

  1. Go the Microsoft site and download the Windows 11 ISO. You don’t want the installer or the media creation tool, just download the ISO itself.
  2. Setup your virtual machine. Windows 11 requires a minimum of 2 virtual cores and at least 8GB of ram. With my install I used 2 vCores and 16GB for good measure.
  3. Set your machine to use EFI and “secure boot” in the “VM Options” tab. It will be set to Bios by default so don’t skip this step.
  4. Mount your ISO and boot it up.
  5. At boot like always hit any key to boot to the Windows installer.
  6. As soon as the installer environment boots press Shift + F10 to get the command prompt and type “regedit” followed by the return key.
  7. Once the Registry editor comes up navigate or jump to HKEY_LOCAL_MACHINE\SYSTEM\Setup
  8. Create a new folder here called “LabConfig”
  9. You will need to add a DWORD (32-bit) entry BypassTPMCheck
  10. Once you add these entries set them to 1
  11. Close both regedit and the command prompt
  12. You should now be able to install the VM.

When I did my install I used Win 11 Pro so depending on what you are installing your mileage may vary, however this should get any Win 11 VM installed. Like with any Win VM once you have it up and running Install VMware Tools and run windows update. Even though I downloaded the ISO today it still had a small stack of updates it downloaded and updated.

16 thoughts on “Installing Windows 11 on ESXi 6.5”

  1. Hello!

    Unfortunately since WU 22H2 (10/2022) the TPM-hack isn’t working anymore, although the hacks for diskspace, CPU,… are working fine. During windows-update on my VM (21H2) a pop-up “Windows 11 Update” will apear and notes “Thsi PC doesn’t currently meet Windows 11 system requirements” and rolls back the update-process ans quits with error 0x8024000b

    Does anybody have a new hack for the TPM issue im combination with vSphere6.5 ?

    Every hint is very appreciated!

    Michael

    1. It sounds like you are trying to do an upgrade and not a clean install. Everything I have experienced, read, seen and heard indicates you MUST do a clean install of Win 11 NOT an upgrade from Win 10. What you will instead have to do is a fresh install of Win 11 on ESXi 6.5 using this process and then transfer all the data from the old Win 10 VM to the new Win 11 VM.

      I can also confirm that as of today my Win 11 VM is running the latest version of Win 11 without any issues on ESXi 6.5. If you experience issues with your clean install of Win 11 along these lines I would try to get an older ISO and then use windows update to upgrade it once you have windows installed.

  2. Hello James, thanks for your quick response!!
    In my case, I did an unsual windows-update, the installation succeeded months ago. Updating from Windows 11 – 21H1 to 22H2 has caused those symptoms, I’ve reported yesterday.

    But nevertheless I’ll go your way in installing a new VM but already version 22H2, maybe this will succeed.

    1. It should work fine with a fresh installation and then doing a migration over of the information from one VM to the other. If you run into issues just use an older ISO of Win 11 and later update it with Windows Update. I can confirm that is working as I fired up my Win 11 VM on my ESXi host yesterday and all current updates install just fine.

      1. I did the installation with an older ISO and ran windows-update yesterday, finally quiting with a.m. issues. But I hope, that a new installation with its build from 22H2 will skip the issues. I’ll let you know!

Leave a Reply