ESXi - 503 Service Unavailable /tmp/.vsanDiskOpLock.lock
 

ESXi - 503 Service Unavailable



Error interacting with configuration file /tmp/.vsanDiskOpLock.lock: Filesystem reported Full while trying to lock /tmp/.vsanDiskOpLock.lock



https://internet-lab.ru/sites/intern...?itok=3OUBzsvH

A host connected to vCenter is hanging. Virtual machines work, but the host is not managed. After rebooting the services, it is not possible to enter the hypervisor: the error "503 service unavailable".
I must say right away that the cause of the error is the overflow of the / tmp directory. The amsd service from Hewlett Packard causes the error tat , which is described here:


Error "The ramdisk 'tmp' is full" on HPE ProLiant servers with ESXi custom image


Due to the / tmp overflow, even the error "The ramdisk is full" is not displayed. At the same time, virtual machines continue, fortunately, to work.
Check the status of the hostd service.
/etc/init.d/hostd status The service is not running, try to start:
/etc/init.d/hostd start The service starts, but after a while it stops again.
List the errors from the hostd log:


cat /var/log/hostd.log | grep error

https://internet-lab.ru/sites/intern...esxi_tmp_2.png


We read the error:

2020-04-08T07:51:56.935Z error hostd[FFC54B20] [Originator@6876 sub=Hostsvc] Dvs manager not found
2020-04-08T07:51:57.842Z error hostd[FFC96B70] [Originator@6876 sub=VsanPluginMgr] Mount diskgroup failed with exception Failed to get VSAN disk operation lock, a disk management operation is currently in progress(locked pid: 0), Error interacting with configuration file /tmp/.vsanDiskOpLock.lock: Filesystem reported Full while trying to lock /tmp/.vsanDiskOpLock.lock Please clear disk space and try again..
2020-04-08T07:52:05.406Z panic hostd[FFC54B20] [Originator@6876 sub=Default] error: N7Hostsvc21HaPlatformConfigFault9ExceptionE(vim.fa ult.PlatformConfigFault) The most important is the "Filesystem reported Full". We conclude that it is not possible to create a file in the / tmp directory . But didn’t the place end there?


vdf https://internet-lab.ru/sites/intern...esxi_tmp_1.png


Indeed, / tmp is 100% full . We calculate what takes up the most space:
cd /tmp

ls -FlS

https://internet-lab.ru/sites/intern...esxi_tmp_3.png
The ams-bbUsg.txt file took up all the space. Error in HPE Agentless Management (AMS) package version 11.4.0:


Advisory: VMware - VMware AMS Data File Filling Up Tmp May Cause VUM Updates to Fail On HPE Servers Running VMware ESXi 6.0 / 6.5 / 6.7 with AMS Version 11.4.0


The esxcli command will not execute (localcli may work in this case), for example:


https://internet-lab.ru/sites/intern...esxi_tmp_4.png




Just nothing is output.
First, free up space:
cd /tmp
rm ams-bbUsg.txt Run hostd:
/etc/init.d/hostd restart


Nguồn: https://internet-lab.ru/esxi_503


:battay: