With stateless VMware Auto deploy hosts, or hosts with SD or USB Stick installation, these hosts typically do not have local storage attached. There will be no space assigned to the scratch partition for dumps and logs. Defaulted they will be stored in RAM (/tmp/scratch). After a reboot the /tmp will be cleared.
We need to redirect the scratch partition to a shared VMFS and a central log / dump location. Scratch redirection is performed with the following steps:
- Decide on a shared data store where preferably all hosts in a physical location connect, beware of physical location Bias. The volume is like /vmfs/volumes/UUID
- Create a locker directory on that data store, for example .locker-Hostname
- Connect to the ESXi host with Connect-ViServer -Server <Hostname>
- Set the scratch partition with Set-VMHostAdvancedConfiguration -Name “ScratchConfig.ConfiguredScratchLocation” -Value “/vmfs/volumes/UUID/.locker-Hostname”
- Maintenance mode (Set-VMHost -VMHost <host> -State ‘Maintenance’) and reboot (Restart-VMHost <host>).
With central management of an environment, a central log/dump location is recommended (also for stateful deployments this should be a standard). For a central log we need some services added to the vCenter:
– dump collector; collecting ESXi dumps of the infamous PSOD.
– syslog collector; collecting the host logs or syslogs.
On Windows we use the vCenter installer to install both services. With the vCenter Server Appliance (VCSA) these service are bundled with the appliance.
Dump Collector installation
Installation of the dump collector is straightforward. Determine in advance where your dumps will be saved, how much space (2GB default is sufficient for most environments) and to do a standalone or integrate with vCenter (last option is preferred for single management).
(if you got a data disk, change the default C: parameters)
And fill in the vCenter and account details on the next screen. The account will be used to register the extension.
Syslog Collector installation
Just like the dump collector the installation is straightforward. Determine the syslog rotation and log location. And start the installation to the VMware vCenter Server.
(if you got a data disk change the default C:).
Change the ports when needed for your environment, standard default is sufficient for most environments.
And finish the installation.
If you still use the vSphere client, then install both service add-ins.
Configuring the hosts
The ESXi hosts need to be configured to use the remote syslog and network dump location. You can use ESXi CLI and the advanced system settings or you can use PowerCLI. If we use PowerCLI for syslog we can use the Set-VMHostAdvancedConfiguration cmdlet and for the dump collector we need the Get-EsxCli cmdlet to use esxcli.
When we do this for a list of hosts we can use the following script. Just replace the vcenter servername at $vCenterFQDN and vCenter IP at the $esxcli.system.coredump.network.set line.
#set vCenter IP/FQDN
$vCenterFQDN = “<vcenter server>”
#connect to vCenter
$viServer = Connect-VIServer -Server $vCenterFQDN -Credential (Get-Credential)
#get all hosts in vCenter so we can cycle thru them
$Hosts = Get-VMHost
#cycle through each host
ForEach ($VMHost in $Hosts)
#Add Syslog to Syslog Collector
Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.global.logHost -Value $vCenterFQDN
# Add DUmp to collector
$esxcli = Get-EsxCli -vmhost $VMHost
$esxcli.system.coredump.network.set($null, “vmk0”, “<vcenter ip>”, “6500”)
#write out host name that was changed:
write “Updated host: $VMHost”
#disconnect from ESX server
Disconnect-VIServer -Server $viServer -Confirm:$false
Checking the configuration
A host can be checked by going to the dump directories and checking the contents, just like the following for a host’s syslog directory.
I don’t have a crashed ESXi host at my disposal, so it is easier to check on the hosts via esxcli:
esxcli system coredump network get.
The IP and port should be in the output at:
Network Server IP:
Network Server Port: 6500