VMware v Lunatic
I seem to have a mental aberration every month or so, something to do with full moons I think, when I do something rather rash, often preceded by doing too many things at once and time constraints.
During the last one I decided to set the wrong drive offline. This caused a couple of servers to disappear, no VMware is really quite resilient but losing a drive seemed to cause it a few problems. I quickly realised my error and set the drive online again, however, the two ESXi host servers that had running guest servers on the offline volume had locked up and were unresponsive, though other guest servers on the two host servers will still up and running though the VC was showing the ESXi servers as disconnected and the other servers as unavailable. I started shutting down the servers using RDP to connect with the intention of forcing a reboot of the ESXi servers.
I cleared one off and restarted it in the meantime the second one had managed to recover its composure. All the servers were back and the one that had been on the missing volume looked like nothing had happened, I checked the event logs and apart from a few missing disks logs you wouldn't have known it had lost its drives or stopped working, it never crashed. I can only assume VMware put the servers in a suspended state then restarted it once the drive returned.
If I had been a bit more patient it would have recovered the server on the other host as well and no one would have noticed too much. It would have been nicer if it had all happened a bit quicker and if the host hadn't stopped talking to the VC but at the end of the day it did a pretty good job when faced by a rather foolish action.
During the last one I decided to set the wrong drive offline. This caused a couple of servers to disappear, no VMware is really quite resilient but losing a drive seemed to cause it a few problems. I quickly realised my error and set the drive online again, however, the two ESXi host servers that had running guest servers on the offline volume had locked up and were unresponsive, though other guest servers on the two host servers will still up and running though the VC was showing the ESXi servers as disconnected and the other servers as unavailable. I started shutting down the servers using RDP to connect with the intention of forcing a reboot of the ESXi servers.
I cleared one off and restarted it in the meantime the second one had managed to recover its composure. All the servers were back and the one that had been on the missing volume looked like nothing had happened, I checked the event logs and apart from a few missing disks logs you wouldn't have known it had lost its drives or stopped working, it never crashed. I can only assume VMware put the servers in a suspended state then restarted it once the drive returned.
If I had been a bit more patient it would have recovered the server on the other host as well and no one would have noticed too much. It would have been nicer if it had all happened a bit quicker and if the host hadn't stopped talking to the VC but at the end of the day it did a pretty good job when faced by a rather foolish action.
Comments
Post a Comment