Category: VMWare

How to Remove Inaccessible vSAN Objects in vSphere: Step-by-Step Guide

This post is about how to remove such an inaccessible object within vSAN.

Afbeelding met tekst, schermopname, nummer, Lettertype Automatisch gegenereerde beschrijving

Open an SSH session to the vCenter and enter the command rvc localhost in the command line.

Navigate to the destinated vSAN cluster where you want to remove the inaccessible objects using cd and utilize ls to list in each step like this one:

Verify the state of vSAN objects using the command vsan.check_state -r . This check involves three steps:

  • Checking for inaccessible vSAN objects
  • Verifying invalid or inaccessible VMs, and
  • Checking for VMs for which VC/hostd/vmx are out of sync

During this check, as you can see in the next screenshot, there are four inaccessible objects with the same UUID as those listed in Virtual Objects within the vSphere Client.

Afbeelding met tekst, schermopname, Lettertype Automatisch gegenereerde beschrijving

To remove them, open an SSH session to any ESXi in the cluster and use the following command /usr/lib/vmware/osfs/bin/objtool delete -u <UUID> -f replacing UUID with the one you want to remove. Afbeelding met tekst, schermopname, Lettertype Automatisch gegenereerde beschrijving

After you remove all inaccessible objects and run the (vsan.checkstate -r .) once again, you should no longer see any inaccessible objects. Afbeelding met tekst, schermopname, software, Lettertype Automatisch gegenereerde beschrijving

‘Ineligible for use by VSAN’ can’t be added to VSAN disk groups

I had the opportunity to test a Dell vSAN node. I had a older unattend install esxi iso.
This installed the ESXi OS on the wrong disk. After a correct install vSAN did not see this this disk ready for use for vSAN. Combining the following articles Dell VXRai vSAN Drives ineligible and identify-and-solve-ineligible-disk-problems-in-virtual-san/
I solved this problem with the following steps:

Step 1: Identify the Disk with vdq -qH

Step 2: Use partedUtil get “/dev/disks/<DISK>” to list all partitions:

partedUtil get “/dev/disks/t10.NVMe____Dell_Ent_NVMe_CM6_MU_3.2TB______________017D7D23E28EE38C”

Step 3: Use This disk has 2 partitions. Use the partedUtil delete “/dev/disks/<DISK>” <PARTITION> command to delete all partitions:

Afbeelding met tekst, Lettertype, schermopname Automatisch gegenereerde beschrijving

Step 4:

When all partitions are removed, do a rescan:

~ # esxcli storage core adapter rescan –all

Afbeelding met tekst, schermopname, software, Computerpictogram Automatisch gegenereerde beschrijving

Step 5: Claim Unused Disks

Afbeelding met tekst, software, Lettertype, nummer Automatisch gegenereerde beschrijving

ESXi Unattend Install on Dell BOSS controller

I had the opportunity to test a Dell vSAN node. I had a older unattend install esxi iso.
This installed the ESXi OS on the wrong disk.

I hate to type a very complex password twice.
So automation is the key.
I love de ks.cfg install option

Sow following the following guide did not the trik:
https://www.dell.com/support/kbdoc/en-us/000177584/automating-operating-system-deployment-to-dell-boss-techniques-for-different-operating-systems

VMware ESXi Automated Install

This did not work:
install –overwritevmfs –firstdisk=”DELLBOSS VD”

After doing a manual install:
Afbeelding met tekst, schermopname Automatisch gegenereerde beschrijving

What works:

# For Dell Boss Controller “Dell BOSS-N1″

install –overwritevmfs –firstdisk=”Dell BOSS-N1”

Deploy Windows Core Server 2022 with Server Core App Compatibility Feature on Demand with Packer

I while ago I started with parker to create simple templates for use in my homelab.

It take some time to find the rights scripts and learning en understanding the HCL2 coding

But in related to Security reasons I want to use a Windows Core Server the smaller footprint.

What is Server Core App Compatibility Feature on Demand: https://learn.microsoft.com/en-us/windows-server/get-started/server-core-app-compatibility-feature-on-demand

Installing Features on Demand through Powerschell contains a bug. You may see “failure to download files”, “cannot download”, or errors like “0x800F0954” or file not found.

To Solve that I created I powerschell script to run the install twice: featuresondemand.ps1

You can find al the needed files on my Public Github Packer repository: https://github.com/WardVissers/Packer-Public

When running is showing like this:

A blue screen with white squares Description automatically generated

It works for now, but there is one thing that would the hole thing a quiet nicer.
Passwords encrypted in a separate file.

VCF 5.0 running inside Nested ESXi server with only 64GB Memory

So I interested to trying to deploy latest release of VMware Cloud Foundation (VCF) 5.0 on my Windows 11 Home PC witch have 128GB and 16 core intel cpu.

William Lee wrote a nice artikel about VMware Cloud Foundation 5.0 running on Intel NUC

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Requirements:

  • VMware Cloud Builder 5.0 OVA (Build 21822418)
  • VCF 5.0 Licenses Through VMUG ADVANTAGE
  • Home PC (Not Special Hardware)
    – 128GB Memory
    – Intel 12600 CPU
    – 4TB of NVME Storage
  • Windows 11 with VMware Workstation 17

Setup

Virtual Machines

  • DC02 (Domain Controller, DNS Server) (4GB 2vcpu)
  • VCF-M01-ESX01 (ESXi 8.0 Update 1a) (64GBGB 1x140GB 2x600NVME 2x NIC) (Every Thin Provisiond)
  • VCF-M01-CB01 (4GB and 4CPU) Only needed through First Deploment

Network settings on my PC

  • 1 IP In my home network
  • 172.16.12.1 (To Fool Cloudbuilder)
  • 172.16.13.1 (To Fool Cloudbuilder)

Procedure:

Install en Configure ESXi

Step 1 – Boot up the ESXi installer from de iso mount and then perform a standard ESXi installation.

Step 2 – Once ESXi is up and running, you will need to minimally configure networking along with an FQDN (ensure proper DNS resolution), NTP and specify which SSD should be used for the vSAN capacity drive. You can use the DCUI to setup the initial networking but recommend switching to ESXi Shell afterwards and finish the require preparations steps as demonstrated in the following ESXCLI commands:

esxcli system ntp set -e true -s pool.ntp.org
esxcli system hostname set –fqdn vcf-m01-esx01.wardvissers.nl

Note: Use vdq -q command to query for the available disks for use with vSAN and ensure there are no partitions residing on the 600GB disks.
Don’t change time server pool.ntp.org.

To ensure that the self-signed TLS certificate that ESXi generates matches that of the FQDN that you had configured, we will need to regenerate the certificate and restart hostd for the changes to go into effect by running the following commands within ESXi Shell:

/bin/generate-certificates
/etc/init.d/hostd restart

Cloudbuilder Config

Step 3 – Deploy the VMware Cloud builder in a separate environment and wait for it to be accessible over the browser. Once CB is online, download the setup_vmware_cloud_builder_for_one_node_management_domain.sh setup script and transfer that to the CB system using the admin user account (root is disabled by default).

Step 4 – Switch to the root user and set the script to have the executable permission and run the script as shown below

su –
chmod +x setup_vmware_cloud_builder_for_one_node_management_domain.sh
./setup_vmware_cloud_builder_for_one_node_management_domain.sh

The script will take some time, especially as it converts the NSX OVA->OVF->OVA and if everything was configured successfully, you should see the same output as the screenshot above.

A screenshot of a computer Description automatically generated

Step 4 – Download the example JSON deployment file vcf50-management-domain-example.json and and adjust the values based on your environment. In addition to changing the hostname/IP Addresses you will also need to replace all the FILL_ME_IN_VCF_*_LICENSE_KEY with valid VCF 5.0 license keys.

Step 5 – The VMnic in the Cloud Builder VM will acked als a 10GB NIC so I started the deployment not through powershell but normal way in Cloud Builder GUI.

Your deployment time will vary based on your physical resources but it should eventually complete with everything show success as shown in the screenshot below. (I have one retry for finish)
A screenshot of a computer Description automatically generated A screenshot of a cloud support Description automatically generated
Here are some screenshots VCF 5.0 deployment running on my home PC.

A screenshot of a computer Description automatically generated

A screenshot of a computer Description automatically generated

Problems

Check this if you have problems logging in NSX:
https://www.wardvissers.nl/2023/07/26/nsx-endless-spinning-blue-cirle-after-login/

Next Steps.

1. Reploy with use of the Holo-Router https://core.vmware.com/resource/holo-toolkit-20-deploy-router#deploy-holo-router

2. Testing if can deploy Single Host VCF Workload Domain, on same way by following this blog post HERE! 😁
A screenshot of a computer Description automatically generated

If I can start another 64GB ESXi Server.

NSX endless spinning blue cirle after login

After Upgrading NSX in my Homelab from from 3.2.0.1 to 4.0.1.1 I had weird issue. Also when I deployed my VCF5 lab
When I log in the NSX manager I had endless spinning blue circle
Tried different browsers Chrome/Firefox.

A blue and white circle Description automatically generated

Solution: Setting the Browser back to English.. Works

Chrome.

A screenshot of a computer Description automatically generated

And it works again

A screen shot of a computer Description automatically generated

Source: Link

VMware Cloud Foundation 5.0 is Here

The VMware Cloud Foundation team is pleased to announce the availability of VMware Cloud Foundation 5.0

Waiting for VMUG Advantive to be updatet & Finding the time to test it out.

Powercli Delete Orphaned Edge Transport Nodes

I made a little mistake on my Lab Environment. There fore it was required to clean up my last unremoveable edge server.

Sow i made a little powercli script

RemoveOrphanedEdge.ps1

# https://kb.vmware.com/s/article/892835
# https://virtuallyjason.blogspot.com/2020/11/using-nsx-t-api.html

$nsxfqdn = Read-Host “Give the IP of the FQDN from your NSX Server”
if (!($creds)){

$creds = Get-Credential -Message "NSX-T Administrative Credentials:"

}
$base64Creds = [Convert]::toBase64String([System.Text.Encoding]::UTF8.GetBytes(“$($creds.username):$($creds.GetNetworkCredential().password)”))
$header = @{Authorization = “Basic $base64Creds”}
$uri = “https://$nsxserver”

# Check API Connection

invoke-webrequest -uri “$uri/api/v1/spec/openapi/nsx_api.json” -headers $header -SkipCertificateCheck

# Clean Stale Entries

invoke-webrequest -uri “$uri/api/v1/transport-nodes?action=clean_stale_entries” -headers $header -Method ‘POST’ -SkipCertificateCheck

Enable Virtualization-based Security on a Virtual Machine on Nested ESXi Server in VMware Workstation

First Step Shutdown ESXi Server enable Encryption
Graphical user interface, application Description automatically generated

Second Add vTPM

Graphical user interface, application, Word Description automatically generated

Boot ESXi Server(s)

Configure Key Providers (Add Native Key Provider)

Graphical user interface, text, application, website Description automatically generated

A screenshot of a computer Description automatically generated

Now you can add vTPM to you VM
Don’t forget to enable VBS

Graphical user interface, application Description automatically generated

Create GPO SRV 2022 – Virtualization Based Security and I did Apply only to my Server 2022 Lab Environment
Graphical user interface, application Description automatically generated

System Information on my Server 2022 Lab Server
A screenshot of a computer Description automatically generated with medium confidence

Virtual Machine with Windows Server 2022 with KB5022842 (Feb 2023) installed and configured with secure boot will not boot up on vSphere 7 unless updated to 7.0u3k (vSphere 8 not affected)

Virtual Machine with Windows Server 2022 with KB5022842 (Feb 2023) installed en configured with secure boot enabled will not boot up on vSphere 7 unless updated to 7.0u3k (vSphere 8 not affected)

VMware Seciroty Violation message

In VM vmware.log, there is ‘Image DENIED’ info like the below:
2023-02-15T05:34:31.379Z In(05) vcpu-0 – SECUREBOOT: Signature: 0 in db, 0 in dbx, 1 unrecognized, 0 unsupported alg.
2023-02-15T05:34:31.379Z In(05) vcpu-0 – Hash: 0 in db, 0 in dbx.
2023-02-15T05:34:31.379Z In(05) vcpu-0 – SECUREBOOT: Image DENIED.

To identify the location of vmware.log files:

  1. Establish an SSH session to your host. For ESXi hosts
  2. Log in to the ESXi Host CLI using root account.
  3. To list the locations of the configuration files for the virtual machines registered on the host, run the below command:

#vim-cmd vmsvc/getallvms | grep -i “VM_Name”

  1. The vmware.log file is located in virtual machine folder along with the vmx file.
  2. Record the location of the .vmx configuration file for the virtual machine you are troubleshooting. For example:

/vmfs/volumes/xxxxxxxx-xxxxxxx-c1d2-111122223333/vm1/vm1.vmx
/vmfs/volumes/xxxxxxxx-xxxxxxx-c1d2-111122223333/vm1/vmware.log

Resolution

This issue is resolved in VMware ESXi 7.0 U3k, released on February 21st 2023. Build 21313628
A screenshot of a computer Description automatically generated with medium confidence
Notes:

  • Virtual machines running on any version of vSphere ESXi 8.0.x are not impacted by this issue
  • vSphere ESXi 6.7 is End of general Support. For more information, see The End of General Support for vSphere 6.5 and vSphere 6.7 is October 15, 2022.
  • If you already face the issue, after patching the host to ESXi 7.0 Update 3k, just power on the affected Windows Server 2022 VMs. After you patch a host to ESXi 7.0 Update 3k, you can migrate a running Windows Server 2022 VM from a host of version earlier than ESXi 7.0 Update 3k, install KB5022842, and the VM boots properly without any additional steps required.

Translate »