Category Archives: VMware

Mismatched MTUs on iSCSI uplinks cause VMware ESXi server to hang

Over the last few days I’ve been having an issue with a pair of ESXi 5.0 servers in a clustered pair. All appeared to be working normally then one day vMotion stopped working and the progress would hang at 9%. Furthermore ESXi itself would then stop responding to vCenter requests.

Eventually the vMotion would timeout and the host would come back online (or you can kick it via ssh and restart the hostd service). Looking at the events I could see that the system has lost contact with the iSCSI storage during the hang.

The problem was intermittent, occasionally it would work then break again which made it harder to diagnose.

The problem in the end was caused by a mismatch in the MTU sizes on the iSCSI uplinks.

I had created a vSwitch for iSCSI and added two port groups then assigned one uplink per port group. Unfortunately one of the port groups was set at MTU = 1500 and the other at MTU = 9000. This configuration was on both ESXi servers.

Once I set the MTUs all to 9000 the problem went away.

Tagged , , ,

Device or resource busy when trying to delete vmdk on vmfs

When you try and delete an orphaned vmdk file you get the error:

rm: cannot remove `xxxxxxx-flat.vmdk': Device or resource busy

or

Cannot delete file [xxxxxx] xxxx/xxxx-flat.vmdk

This may be because

  • The file is being used and locked open by a running VM
  • The file is an orphan but is still locked by ESX

To check if the file is an orphan (zombie) download RVtools from

http://www.robware.net/

and click on the vHealth tab. All files that are potentially orphans are listed here.

If the vmdk is an orphan then one of the ESX hosts will be locking the file. Most likely you’ll have quite a few ESX hosts and rebooting each one until the file unlocks would be painful. Instead you can make use of vmkfstools:

SSH to each ESX host in turn and run the following command:

vmkfstools -L release /vmfs/volumes/xxxxxxxx/xxxxxxx/xxxxxx-flat.vmdk

There are two potential errors:

Could not open /vmfs/volumes/xxxxxxxx/xxxxxxx/xxxxxx-flat.vmdk
Command release failed
Error: Device or resource busy

or

Command release failed
Error: Inappropriate ioctl for device

The host that gives you this second error is the one that has locked the file.

Migrate off the VMs from the host and place it into maintenance mode and reboot it. Now try and delete the file that was locked. You should find that the file can now be deleted.

Tagged ,

Call “PropertyCollector.RetrieveContents” for object “propertyCollector” on vCenter Server “vcenter.mydomain.com” failed

When you convert a template back to a virtual machine and try and edit the settings you may get an error message like this one:

Call "PropertyCollector.RetrieveContents" for object "propertyCollector" on 
vCenter Server "vcenter.mydomain.com" failed

To fix just shutdown the VM, remove the VM from the inventory and re-add it.

1. Shutdown the VM

2. Right click the VM in the vCenter client -> Remove from inventory

3. Go to Inventory -> Datastores -> Browse the datastore which the vmdk/vmx files for the VM are on

4. Double click the VM’s folder -> Right click the vmx file -> Add to inventory

Now if you try and edit it again you’ll find it works OK.

Tagged , , ,

VMware guest customizations fail to be applied on sysprep of XP

Recently I experienced a problem where a previously working XP template deployment suddenly stopped working. The sysprep kicked in on first boot and the VM restarted back to the sysprep dialogue boxes. Usually the customization specifications would fill in the necessary details as an answer file would and the VM would boot up fully customized.

After a bit of rooting around I discovered an old post in VMware communities that provided a solution. It turns out that the Altiris Client installed within the template in some way blocks vCenter from deploying the answer file to the guest.

The fix was as follows:

1. Boot up the template guest

2. Open up the services list (Start -> Run -> Services.msc)

3. Right click the Altiris Client service (aclient) and select Properties. Set the start mode to disabled.

4. Shutdown the template and revert it back to a template.

5. Now open up your customization preferences for this operating system and click through until you get to the option to run programmes on first boot after sysprepping.

6. Type in

cmd /c sc config aclient start= auto

(note the space between = and auto)

and click add

7. Now add another line

cmd /c sc start aclient

and add that.

 

8. Save your changed customization specification

9. Deploy a new VM using this customization specification

For more information check out the original communities post.

Tagged , , , ,

Upgrading SCSI Controller from LSI to PVSCSI (VMware Paravirtual)

Here’s a workaround method to change the type of SCSI controller for your windows VM without falling prey to the BSOD on boot. The method involves adding a new disk to the VM to install the PVSCSI driver into windows before changing the SCSI controller type for the system disk:

1. Shutdown the VM
2. VM -> Edit Settings -> Add -> Hard Disk -> Create a new virtual disk -> Choose 1MB as the size -> Tick Allocate space on demand -> Change the virtual device node to a new SCSI channel that hasn’t been used before (e.g. 3:15) -> Finish
3. Click on the New SCSI Controller -> Change Type -> VMware Paravirtual -> OK
4. Click OK
5. Power on the VM
6. Open console
7. Logon to windows
8. Wait a minute or so for the hardware to install. You can check it in device manager -> SCSI and raid controllers. Look for VMware PVSCSI Controller.
9. Shutdown the VM
10. VM -> Edit Settings -> Click on the 1MB disk you created -> Remove -> Select “Remove from virtual machine and delete files from disk”
11. Click on the other controller (SCSI controller 0) -> Change Type -> Select VMware Paravirtual -> Click OK -> Click OK
11. Power on VM

The VM should now boot using the Paravirtual SCSI driver.

Tagged , , , , ,

Network interface disappears from VMware virtual machine

I recently had an issue where an interface on a VM disappeared from the VM configuration seemingly without warning. Checking the events showed that there had been no changes to the VM configuration by any other administrators. In the vmware.log for the VM I found the following lines:

Nov xx 11:02:18.831: vcpu-1| Ethernet0 MAC Address: 00:50:56:xx:xx:xx
Nov xx 11:02:19.395: vmx| Powering off Ethernet0
Nov xx 11:02:19.396: vmx| Hot removal done.

This indicated that the interface had been hotplug removed. vSphere now supports more hotpluggable devices. While this makes it easier to add devices while the VM is running it also means that the devices can be removed. This means they show up in windows’ “Safely Remove Hardware” dialogue in the system tray:

Safely Remove Hardware Screen

So one mis-double click on the system tray icon causes the interface to be ejected from windows and removed from the VM configuration.

There is two ways to fix this:

Disable hotplug for the whole VM

Edit the registry on the VM to disable being able to eject the interface in windows

The former will of course stop you adding or removing any other hardware, not just the interface, while the VM is still running. This is probably the best option for VMware View VMs.

The latter will require a bit of hacking and isn’t really desired either.

Both methods are discussed in the VMware links below:

VMware blog: NIC is missing in my machine

Virtual machines in the VMware View environment lose network connectivity

Disabling the HotPlug capability in ESX 4.x virtual machines

Tagged , , ,

Backing up ESXi images using ghettoVCB and FTP

Update: mksbackup can also do this: http://www.magikmon.com/mksbackup/ghettovcb.en.html

ghettoFTP currently doesn’t support backing up VMs to FTP so I’ve written a simple wrapper shell script that ftps the backups that ghettoVCB creates.

You’ll need the ghettoVCB scripts which you can grab from the VMware communities forum here.

The script does the following:

1. Queries the local host for a list of VMs (esxcli vms vm list)

2. For each VM it runs the ghettoVCB shell script

3. At the end of each VM backup it uploads the backup to an FTP server using ESXi’s built-in ftpput command.

ESXi doesn’t have any cron built in which means you’ll either have to run the script manually (a pain!) or schedule it from another box.

I use plink which you can use in a scheduled task to kick off the script via SSH. You’ll obviously need to enable the remote SSH on the ESXi box.

#!/bin/sh
###############################################################################################
# Name: runbackup.sh
# Desc: This backup uploads the gzipped VM files produced by ghettoVCB.sh
# History:
#
# Date          Author          Version         Details
# 28/02/2011    Stuart Jordan   1.0             Script creation
# 01/02/2011    Stuart Jordan   1.1             Amended to upload directories rather than gzips
#                                               Added tidying up of directories post backup
#

ftpUsername="ftpusername"
ftpPassword="ftppassword"
ftpServer="ftpserver"
backupPath="/vmfs/volumes/localvmfsfolder/backups"
scriptPath="/path/to/the/ghettoVCB/script/folder"

esxcli vms vm list  | grep "Display Name:" | awk '{print $3}' | while read currVM
do
        echo $currVM > $scriptPath/curr_backup
        $scriptPath/ghettoVCB.sh -f $scriptPath/curr_backup
        rm -f $scriptPath/curr_backup

        if [ -d "$backupPath/$currVM" ]
        then
                find $backupPath/$currVM -type f |
                while read currBackup
                do
                        echo Processing $currVM
                        filename=$(basename $currBackup)
                        echo File is named $filename
                        filePath=$(dirname $currBackup)
                        echo Uploading file $currBackup to $ftpServer using account $ftpUsername
                        ftpput -u $ftpUsername -p $ftpPassword $ftpServer /VMware/$currVM/$filename $currBackup
                        retval=$?
                        if [ $retval -gt 0 ]
                        then
                                echo Error $retval connecting to FTP server $ftpServer and uploading file $filename
                                else
                                        echo FTP completed OK. Deleting $currBackup file.
                                        rm -f $currBackup
                                fi
                        done

                        find $backupPath/$currVM/* -type d |
                        while read currBackupDirs
                        do
                                if [ -d "$currBackupDirs" ]
                                then
                                        echo Removing directory $currBackupDirs
                                        rmdir $currBackupDirs
                                fi
                        done

                        if [ -d "$backupPath/$currVM" ]
                        then
                                echo Removing $backupPath/$currVM
                                rmdir $backupPath/$currVM
                        fi
                retval=$?
                        if [ $retval -gt 0 ]
                        then
                                echo Error $retval cleaning up directory $filePath. It probably still has some files in there.
                        fi
        else
                echo $backupPath/$currVM is missing
        fi
        echo Completed $currVM
done

Download – rename the file to .sh

Tagged , , , ,

Uninstalling plugins from vCenter

Some of the plugins for vCenter don’t offer a way to uninstall which leaves unused plugins lying around in the list.

To unregister a plugin from vcenter you can follow this method:

1. Open your web browser and browse to http://vcenter_ip_address/mob

2. Click on Content on the second line down

3. Click on ExtensionManager on the right about half way down the page

This will show you a list of plugins installed and some methods at the bottom.

4. Click on the plugin you wish to unregister

5. At the top you’ll see something like

Property Path: extensionList[“com.emc.EMCStoragePlugin”]

6. Select the part enclosed in the quotes (e.g.com.emc.EMCStoragePlugin) and copy it

7. Now hit back in your browser to get the list of plugins again

8. Click on UnregisterExtension at the bottom

9. In the text field paste the text into the box and click InvokeMethod

10. Refresh your browser and the plugin should be gone from the list

Now reconnect to vCenter using the client and the plugin should be gone from Plug-ins-> Manage Plugins list.

There’s some more info at the VMware KB here.

Tagged , , ,

The virtual machine has virtual disk in link-cloned mode that prevents migration

Sometimes vCenter loses track of the snapshots on a virtual machine and may show no snapshots when in fact there is one (or more) in place. If you look at the actual vmdk files on disk you will see “delta” files for each vmdk.

Also when you try to storage vMotion the VM you get an error like this one

The fix is quite simple:

1. Right click on the VM -> Snapshot -> Take Snapshot

2. Exit window and reopen it again. You’ll see a consolidation-helper snapshot like this one

3. Click on Delete All and click on close

Depending on how large the snapshot is it may take a while. Once done check the vmdk files again, the delta files should now be gone. Then try and storage vMotion the VM again.

Tagged , , , , , , ,

Slow mouse in Windows 7/2008 on vSphere 4.x

Poor mouse performance on Windows 2008 due to the display driver (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011709) is fairly well known. Just replace the SVGA/VGA display driver detected in Device Manager with the WDDM one as described in the article. However today I had a similar issue on a Windows 7 VM that showed the same problem. Reinstalling the WDDM did not fix the issue.

In the end the problem was relating to the mouse driver installed.

Automatically updating the  “PS2 compatible mouse” driver didn’t find anything. The fix was to manually select the “VMware pointing device” from the list of available drivers.

As soon as I’d done that the mouse was smooth as ever!

Tagged ,