Nothing like a bit of Hyperbole, but some days are better than others! One thing I have learned over the years is to always expect the unexpected. Yesterday’s ‘unexpected‘ was one of these;
That’s cool, HA kicked in and the stranded guests began to restart as expected. Not great, but it had done it’s job.
When the host returned, I began to migrate guests back to it (this is a Essentials + env, so no DRS). Then guess what ? Boom, on the other host;
… couple minutes and the first hosts goes again, and again and a lovely little PSOD loop was in full swing !
What made this particulalry ‘fun’ was the fact that this was a 2 node cluster, and it was in a different country. What topped it off was JAVA being a PITA and not allowing me to load an iDRAC viewer to do some triage.
A few deep breaths and I managed to harness my JAVA woes and a 2 minute analysis showed the cause to be a bug around E1000. (kb 2059053)
The workaround is simple, remove all E1000 nics from guests. Problem was, VC is virtual and the 2 hosts are yo-yo ing and HA was going crazy. I couldn’t get access to a console or anything to be able to remediate.
I powered down both hosts cold, and brought one up. When I could get access I could see all the guests came up powered off, all 70 of them. The phones were starting to run hpt, and people were getting very concerned.
This is where you learn to truly appreciate the power and efficiency of PowerCLI.
This command showed me all the E1000’s that needed changing;
|
1 |
ForEach($VM in(Get-VM)){$VM | Where{$VM | Get-NetworkAdapter| Where{$_.ExtensionData -like "*e1000*"}}} |
I cobbled together a quick script to go ahead and change them all, leaving all properties intact. Since the guests were already powered off, once powered back on , we were good to go.
|
1 2 3 4 5 |
$targetvms = "c:\temp\targetvms.txt" $(ForEach($vm in(Get-VM)){$VM | Where{$vm | Get-NetworkAdapter| Where{$_.ExtensionData -like "*e1000*"}}})|select-object Name |out-file $targetvms (gc $targetvms|Select-Object -skip 3) |sc $targetvms ;(gc $targetvms|foreach-object {$_.trim()}) | where {$_ -match "\w"} | sc $targetvms ForEach ($targetvm in (gc $targetvms)) { get-vm $targetvm | Get-NetworkAdapter | set-networkadapter -type vmxnet3 -confirm:$false } |
Not overly complicated, simply finds the affected guests, uses a sub-expression and some input string cleanup so the output of the for each loop hits the pipeline, then change the nic type without confirmation. This saved a lot of time manually going through and checking, removing, adding for each affected guest, and got people back to work quickly.
A painful hour, and the hosts were upgraded the next day. Lessons learnt, but all good….
4. List snapshot older than x days, and their size;
|
1 |
Get-Snapshot * |Where {$_.Created -lt ((Get-Date).AddDays(-3))} |sort-object VM |ft VM, Name, Description, @{Label="Size";Expression={"{0:N2} GB" –f ($_.SizeGB)}} |
Old snapshots…Someone has been naughty…
All of the inputs/scopes can be filtered by Cluster, Datacenter, Resource Pool etc.
Add these to your Powershell cookbooks, you never know when they might come in useful.
As always, test them, and use caution in your environments.