Sam.McGeown.co.uk

So you’ve upgraded your ESX 3.x servers to 4.0 and you’ve upgraded your vCenter server, now you want to access the shiny new hot-add feature to upgrade some running server’s memory. Except you can’t, the feature is no-where to be seen. Something to bear in mind though, your OS needs to support hot-add, so you’ll need a Windows Enterprise or Datacenter edition.

Here’s how to enable it:

Upgrade the VM’s tools, if you haven’t already. This requires a re-boot, but don’t reboot, shut the server down down, otherwise you’ll require another reboot to add the feature. The tools can be upgraded from the system tray icon.

 

 

Once the VM is shut down, upgrade the Virtual Hardware (again, assuming you haven’t already!)

 

 

 

 

 

 

 

Once your virtual machine is fully up to date with VMTools and Hardware, you can edit the Advanced options (again, while the machine is shut down) and enable the feature.

 

 

 

 

 

 

 

 

Now with the server booted, you can add Memory and CPU without the need for a reboot. In my tests changing the Memory from 1GB to 2GB caused a single dropped ping, and changing from 1 vCPU to 2 vCPUs caused a ping to drop.

VMware vSphere 4.0, VMware vCenter 4.0, VMWare, VMware ESX 4.0 vmware, esx 4.0, vsphere 4, hot add, memory, cpu, vcpu

I'm currently in the process of migrating a 2-host High Availability cluster of ESX 3.5u4 servers to vSphere 4. This is going to come in 3 distinct stages: Stage 1 is to upgrade VirtualCenter Server 2.5 to vCenter 4, which I am going to cover today. Stage 2 is to upgrade each host, and will be covered as I do it. Stage 3 is the upgrade of the Virtual Machines to the latest VMware Tools and then the new VM hardware.

So to start, I'll outline the process:

• Download the vSphere vCenter 4 installer from VMware (~1.8GB).
• Download your updated licensing for vSphere.
• Back up your VirtualCenter server.
• Run the installation.

I'm not going to run through the download of the installer or licensing, if you're not sure how to do that, probably best not to do the rest.

VMWare, VMware Converter, VMware Update Manager, VMware vCenter 4.0, VMware vSphere 4.0 vmware, vsphere, vcenter, upgrade, virtualcenter, vsphere 4, vi3

I'm migrating some hosts off of an older storage LUN, but when I drag the disk to the new Datastore with the SVMotion plug-in the job fails with the following error:

The error occurs because the virtual disk cannot be moved without moving the source files, the .vmx, .vswap etc. Simply drag the entire VM, rather than the virtual disk to the new Datastore.

If you're trying to move a 2nd, 3rd or nth disk and you get this error, drag the entire VM as per above over to the new Datastore, once that's completed, go back in to SVMotion and drag the whole VM across again, only this time before you apply, drag the nth disk back to the new Datastore.

VMWare, VMware ESX 3.5, VMware Storage vMotion svmotion, vmware, storage vmotion

A.K.A Why not to use snapshots

I ran into a slightly confusing problem today - our SQL servers are all created with 4 disks on 4 separate LUNs (System, Swap, SQL Data and SQL Logs). When viewing the server through Virtual Center I couldn't see all of the LUNs, just the System LUN. It's not a major problem as the VM can see the storage, but a little annoying when you have to remember what LUN the disks are on.

Slightly more distressing was the fact that the System-LUN was running out of space - fast. A LUN that should have had about 150GB free was running dangerously low. On investigation I found various snapshot files were being stored in with the System-LUN, which is where the VM's VMX, vswap etc are situated. These were the snapshot delta files of the additional disks, which were on other storage! This isn't first apparent at first as the disk snapshots have been named sequentially by ESX, so a VM with 4 disks on separate LUNs will in fact create 4 snapshot files on the SYSTEM-LUN named VM01-00001.vmdk, VM01-00002.vmdk, VM01-00003.vmdk and VM01-00004.vmdk. 00001 is for the System disk, 00002 is for the Swap disk etc etc. This means that the IO on that LUN has been multiplied, and the storage space is shrinking very rapidly.

A little more digging and it seems that this is by design - snapshots are not meant to be kept for very long, and I think VMware made a deliberate decision to make it difficult to do so. Any virtual disks created for a VM, lets call it VM01, were named VM01.vmdk. When additional virtual disks were created through vCenter on a different LUN, they were still named VM01.vmdk - there's no conflict because they're in different locations. However, when vCenter takes a snapshot it places them with the original disk, and because it's got the same name as the existing disk it starts to enumerate them.

This is bad for a number of reasons - most prominent of which is that if the snapshot file grows large, vCenter does not handle the commit well. In fact, neither does ESX, but I'll get to that. vCenter will time out on any operation that takes more than 15 minutes, so a commit of a 10GB snapshot will look for all intents and purposes in vCenter like it's failed. On top of that, the enumeration of snapshot delta files can cause confusion as to which disk it actualy belongs to, and if that happens, commiting

We all know snapshots are performance killers, but the functionality they provide is not insignificant, and as with most things a balance has to be struck between the functionality and the performance.

So the headlines

• VMs created with disks on multiple LUNs in vCenter use the SAME DISK NAME (eg; for VM01 the disks were created in /vmfs/volumes/SYSTEM-LUN/VM01.vmdk, /vmfs/volumes/SWAP-LUN/VM01.vmdk etc etc).
  • Mitigate this by creating disks using the vmkfstools and adding them to the VM or renaming the existing disks (see below).
• Snapshots cause ALL disk delta files onto the "system" LUN (i.e. where your VMX file is stored.) This is bad because a) it multiplies your I/O on that disk and b) you negate the benefits of storing on multiple LUNs.
  • Mitigate this by deleting your snapshots. There's no other way*, don't try manually moving them or you will have problems.
• Commiting large snapshots takes time - LOTS of time - and can have a big performance hit on your server.
  • Mitigate this by shutting down your VM first and commiting the disk using the vmware-cmd out of business hours. You can also merge the old disk and snapshots into a "new" disk, then shut down the VM and boot with the "new disk".
• vCenter has a hard coded 15m timeout.
  • If you are doing a operation that will take longer than that, do it via the console!

* when I say there's no other way, I mean, there's no other practical way. There are methods to move the snapshot files to another LUN but they bring some serious problems with them.

Create a vmdk (virtual disk) using vmkfstools

• Log in to your server console.
• Type su - (to log in as root, enter root password, note the "-" to load the root user environment variables)
• Navigate to the storage that you wish to use. E.g. cd /vmfs/volumes/System-LUN/
• Create a new folder for the virtual disk: mkdir VM01
• Navigate to the folder: cd VM01
• Create the disk: vmkfstools -c <size> <filename> -a <buslogic|lsilogic>
• For help just type vmkfstools

Rename vmdk files using vmkfstools

• Shut down the VM in vCenter.
• Edit the VM settings and remove the disk you wish to change. Do not delete the file!
• Log in to your server console.
• type su - (to log in as root - enter root password, note the "-" to load the root user environment variables)
• use the command: vmkfstools -E /vmfs/volumes/<LUN>/<Server Name>/<Disk Name>.vmdk /vmfs/volumes/<LUN>/<Server Name>/<New Disk Name>.vmdk

• Go back to the vCenter and re-add the disk, using the new name.

Commit your snapshots using vmware-cmd

• Log in to your server console.
• Type su - (to log in as root, enter root password, note the "-" to load the root user environment variables).
• Use the vmware-cmd -l command to list your VMs. Note the path to the VM you want to deal with.
• Remove all snapshots for a VM: vmware-cmd /path/to/vm/VM01.vmx removesnapshots

VMWare, VMware ESX 3.5, VMWare ESXi 3.5 vmware, esx, snapshots, snapshot, vmware-cmd, vmkfstools

Here's the setup. We have a core switch of 2 Cisco 3750s, connected together for fault tolerance as a single logical switch; we also have several ESX 3.5 hosts with 4 Gigabit Ethernet NICs installed each. The Virtual Machines will all be on VLAN 8 (reserved for internal servers) and the VMKernel will be on VLAN 107 (reserved for VMKernel traffic like VMotion).  I want to create a load balanced, fault tolerant aggregate of these four NICs over the Core Switch.

Configure ESX server's vSwitch

Configuring the vSwitch is actually pretty simple, but there are a couple of gotchas, so don't skip this bit! First thing to note is that if you are making changes to the vSwitch and the Service Console is on that vSwitch you can quite easily lock yourself out. Make sure you configure this correctly, first time! In this setup, I am adding all 4 NICs to vSwitch0, which will be the only vSwitch. I'll then use Port Groups to assign VLANs and Active/Passive configurations to the VMKernel/Service Console.

First things first then - assign the four NICs to the vSwitch. This is done in the Configuration Tab in VMware Infrastructure Client, then the Networking page. Edit the properties of your vSwitch, then select the Network Adaptor tab. Add all the NICs you wish to team in there (they may already be in there, depending on your setup). You should end up with something that looks like this (note that I've not assigned any VLAN yet):

 

Now you need to configure the NIC teaming, so edit the vSwitch Properties and under the Ports tab select the vSwitch. Click edit, and then go to the NIC teaming tab. Configure the teaming options like this:

That's the easy part over and done with! Time to move onto the Cisco!

Configuring the Cisco Core Switch

Firstly, we need to log on to the switch and enter enable mode; I'm going to assume you know how to do this - if not, you really shouldn't be attempting this setup!

Determine the switches trunk load balancing setup by using the command "show etherchannel load-balance". It should look something like this:

If the protocol is NOT src-dst-ip, then you won't be able to establish a trunk connection with the ESX server. If your protocol is not src-dst-ip, change it with the command "port-channel load-balance src-dst-ip". This now matches the "Route based on IP hash" setting you configured in ESX. Although ESX has a setting for MAC based hashing, as does the Cisco, I was unable to get it to work.

Moving on. You need to create a Port-Channel interface for the trunk (this is a virtual interface that binds the 4 GigabitEthernet interfaces together). As i've got other Port-channels in use for connections to other switches, I'm setting up port-channel 40. Move to config mode (conf t) and then enter the setup:

interface Port-channel40
 description VMTEST01 Aggregate
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 8
 switchport mode trunk
 switchport nonegotiate
 spanning-tree portfast trunk
end

Description simply adds a description, "switchport trunk encapsulation dot1q" sets the encapsulation of the trunk to 802.1Q. "switchport trunk native vlan 8" means that any traffic without a VLAN tag will be automatically assigned to VLAN 8. "switchport mode trunk" obviously designates that we want a trunk, rather than access. "switchport nonegotiate" means that it will not attempt to negotiate the protocol, and be a static trunk, rather than LCAP or PGaP. "spanning-tree portfast trunk" causes a Layer 2 LAN interface configured as an access port to enter the forwarding state immediately, bypassing the listening and learning states (i.e. if the link goes down and then comes back up, it will do so quickly).

With the Port-channel configured, you now need to edit your GigabitEthernet ports and assign them to the Port-channel. For each port in the trunk, enter the following config (this example is port 8 on the master switch in my stack, hence 1/0/8):

interface GigabitEthernet1/0/8
 description VMTEST01 VMNIC1
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 8
 switchport mode trunk
 switchport nonegotiate
 channel-group 40 mode on
 spanning-tree portfast trunk
end

The difference between that and the Port-channel setup? "channel-group 40 mode on" is simply assigning the port-channel in static mode.

Once all four NICs are assigned you might have to wait a few minutes for every layer of the connection to settle down before the trunk comes up. To check the status of the etherchannel you can use the command "show etherchannel 40 summary", replacing the 40 for whichever number you assigned to your port-channel.

I hope this helps navigate the minefield that I found to be setting up the NIC teaming!

Cisco, Networking, VMWare, VMware ESX 3.5 cisco, vlan, teaming, nic, esx 3.5, aggregate, trunk, portfast, 802.1q, load balancing

Having just rebooted my laptop, I clicked on the link to open VMWare Infrastructure Web Access in the my browser and was slightly puzzled by the "page cannot be displayed" error. I figured it was probably the VMWare Server Web Access service not starting for some reason, so I opened services.msc and checked it out. The Web Access service was actually running, but the Host Agent service was not, so I tried to start it - it failed. Dutifully opening Event Viewer for some more information I found the error:

The VMware Host Agent service terminated with service-specific error 4294967295 (0xFFFFFFFF).

Great. Thanks for the elaboration. Time to check the VMWare logs, (c:\ProgramData\VMWare\VMWare Server\) - the most recent of which contained the following entry:

[2009-02-18 10:55:27.908 'App' 4408 info] Trying hostsvc
[2009-02-18 10:55:27.948 'App' 4408 panic] error: not well-formed (invalid token)
[2009-02-18 10:55:27.948 'App' 4408 panic] backtrace:(backtraces not supported)
[2009-02-18 10:55:27.948 'App' 4408 info] Win32 service stopped

 A bit of Googling pointed me in the right direction, "not well-formed" refers to the XML config files which are stored in c:\ProgramData\VMWare\VMWare Server\hostd, after that it was just a case of opening each one in turn until I found the malformed one. I backed it up and replaced it with a default version, and lo and behold, my services started!

VMWare vmware, server, web access, xml, not well-formed

 

Having recently installed an ESXi server, I am getting to grips with the management and administration of it, one of the things that I wanted to be able to do was connect to the remote terminal through SSH.

I downloaded my SSH client of choice, PuTTY, and set about connecting, however the server refused the connection. It seems that SSH is not enabled out of the box for ESXi and you need to go through some steps to get there - I found some helpful hints here.These are the steps that I took, based on the advice and some other research.

Go to the ESXi server console screen and hit "Alt+F1". This will switch you to a screen that looks like a log.

Type "unsupported" - you won't see it enter on the screen, just type it blind, followed by the enter key.

You'll see a nice warning, followed by a prompt for the root password. Type your root password and you'll now be at the comand prompt.

Now to edit the inetd.conf. Type "vi /etc/inetd.conf" which will open the conf file in the VI editor. If you're a Windows admin VI will seem a bit strange...just follow the instructions and you'll be ok!

Type "/ssh" to find the line responsible for the SSH config.

Move the cursor using the arrow keys over the # symbol at the beginning of the line, and then press "x" to delete it.

Now type ":wq!" to save the file and quit VI.

Back at the command prompt you can type "/sbin/services.sh restart" to restart the management services.

Next if your server is not running any VMs you can just restart it...if not you will need to identify the process ID for inetd.

Type "ps | grep inetd" which returns the process ID

Type "kill -HUP <process ID>" with the process ID you've just identified.

Type "exit" to log out of that console, and do another "Alt+F1" to switch back to the ESXi home screen.

Back on your SSH client you should be able to log in successfully.

VMWare ssh, remote, vmware, esxi

We recently needed to upgrade one of our applications, and the new version requires an addition server instead of the application and SQL it requires a back end search, a front end web server and a SQL server. The specifications of the new server which are "required" to qualify for support are pretty high. The problem is that the actual processor usage is very light, and it is very hard to justify buying a whole new server that I know is going to be barely used.

The alternative plan was to virtualise the servers, make use of the existing physical hardware, upgrade the RAM and add a couple of drives to the RAID array, which we opted for because it would cost less than £300, instead of £3000.

I forgot, however, the 4GB limitations of Windows Server 2003. 32 bit processors cannot address more than 4GB of RAM, so to get round that you can use Physical Address Extensions (using the /PAE switch in the boot.ini) which enables you to utilise more than the standard 4GB.

Typically a 32 bit system with 4GB RAM will allow 2GB for the kernel, and 2GB for the Applications to use. This means that each application can virtually address up to 2GB of RAM. You can change this balance using the /3GB option in the boot.ini to allow 3GB for applications. Think carefully before doing this!

To enable PAE:

1. Right click "My Computer", select "Properties"
2. Select the "Advanced" tab and click the "Startup and Recovery" button
3. Under "System startup" you can click "Edit" to open the boot.ini file.
4. BE CAREFUL! You can render you OS unbootable! Add the /PAE and /3GB options to the startup (see below for an example) Save, OK and reboot.

multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Windows Server 2003, Standard" /PAE /3GB /fastdetect

It's worth noting that if you have DEP (Data Execution Protection) turned on then PAE will be turned on by default. DEP is on automatically in Windows Server 2003 SP1 - you'll see the /noexecute=[policy level] in the boot.ini

VMWare, Windows Server 2003 windows, server, 2003, /3gb, /pae, 4gb, ram, boot.ini, vmware

I've just upgraded my VMWare Server install to the latest version of the VMWare Server beta, which I have to say, is looking pretty good! One slight niggle that I had was that when I opened the VMWare Remote Console it would tell me that I could not connect, with the error:

Error opening the remote virtual machine SM-00109:8333\16:  The host name could not be resolved.

Clearly THAT's a pain in the proverbial. A quick trawl through VMWare's forums reveals two solutions, one that is clunky and stupid, the other that works. For brevity, the one that works is...

Use the fully qualified domain name of the computer to connect to the VMWare console, so for me, that mean using:

https://SM-00109.domain.dom:8333

instead of

https://SM-00109:8333

Simple!

VMWare vmware beta server 2 host resolved

VMWare, Windows Server 2003 resizing, vmware, vmdk, disk, partition