The case of the disappearing LUNs!

The case of the disappearing LUNs!

We ran into an interesting situation at work this week. For one cluster of ESXi hosts, LUNs were disappearing seemingly at random. Now, we had been using this cluster as a 'Jump' cluster to facilitate moving from one environment to another, so it had lots and lots of LUNs attached. (162, in fact).

We found that every time we would 'Rescan' for new storage, some LUNs would be discovered and others would disappear. It seemed pretty obvious that we were running into some sort of storage maximums, but we were well below the ESXi published maxiums of 256 attached LUNs and 1024 physical paths.  We had previously run into another issue with VMFS heap, so every host in this cluster already had VMFS3.MaxHeapSizeMB set to its maximum. A real stumper.

A call to VMware and we found that we were, in fact, running into a heap issue, but it was not the VMFS heap that was filling.  The lvmdriver also has a heap setting, which by default is 42000KB (~42MB). This lvmdriver heap was completely full.  From the hosts' consoles, this was easily fixed by running an esxcli-module command.  That's great, but not going to work in an environment with 100s of hosts.  PowerCLI to the rescue!!

For today's fun, I wrote a quick function to get the lvmdriver heap size for ESXi 5.0 hosts.  Since I know the default from the VMware case, if the option in the lvmdriver is not explicitly set then the function returns the default value. 

function Get-lvmDriverHeapSize {
<#
.SYNOPSIS
 Gets the lvmDriverHeapSize configured for a 5.0 host
.DESCRIPTION
 This function will get the lvmDriverHeapSize for a 5.0 host if it
 has been configured.
.NOTES
 Author: Cheryl L. Barnett
 .PARAMETER VMhost
 The host object to check
.EXAMPLE
 PS> Get-lvmDriverHeapSize -VMhost (Get-VMHost Test01.fqdn.local)
.Example
 PS> Get-VMHost Test01.fqdn.local | Get-lvmDriverHeapSize
#>
  param(
      [parameter(valuefrompipeline = $true, mandatory = $true,
     HelpMessage = "Enter a VMHost entity")]
    [VMware.VimAutomation.ViCore.Impl.V1.Inventory.VMHostImpl] `
        $VMhost)
  process{
    #Note that if multiple options are set then this entire function breaks
    if (!($_.extensiondata.config.product.version -match "5.0"))
        {
        Throw "$($VMHost) is not an ESXi 5.0 host! Please use this function for` 5.0 hosts only!"
        }
    $maxHeapSizeString = (get-vmhost $_ | get-vmhostmodule ` lvmdriver).options.tostring()
    if($maxHeapSizeString -match "maxHeapSizeKB=")
        {
        $maxHeapSizeKB = [int]($maxHeapSizeString.split("="))[1]
        }
        else
            {
            $maxHeapSizeKB = 42000
            }
    $maxHeapSizeKB
    }
}

So, with this function you can sweep through your environment relatively quickly and get the configured lvmdriver heap size. Note the caveat in the code - if the lvmdriver has multiple options set then this code will break.  That's not an issue I want to tackle right now, since it's way beyond the scope of what I need to do to fix my problem.  Also, watch the line breaks there. I think I put all the backticks in the right place but no guarantees.

If all goes well, next week I'll have a function to set the lvmdriver heap size.

IAAS - Objective 1.1

Objective 1.1- Install vCloud components
Identify required vCloud components and pre-requisites for installation


This post by Frank Denneman hits the nail on the head and includes a very helpful diagram.
To run a VMware vCloud environment, the minimum components to run are:
• VMware vCloud Director (vCD)
• VMware vShield Manager
• VMware Chargeback
• VMware vCenter (supporting the Resource groups)
• vSphere infrastructure
• Supporting infrastructure elements (Networking, SAN)

And these might be worth printing
http://www.hypervizor.com/diags/Hypervizor-com-VMware-vCloud-Suite-Architecture-Diagram-v1.pdf
http://www.hypervizor.com/diags/Designing-your-Private-Cloud-With-vCloud-Director.pdf

Trainsignal has a free video "Everything you need to know about installing vCloud Director 5.1"

KB Article 1026339 'VMware vCloud Director components' sounds like it would cover this objective, but it doesn't really because it's more of a description of the parts of the cloud after it has been installed. It's based on vCD 1.5, but I'm including it anyways because in general you still need to know these things.

2 links I need to re-read

2 links I need to re-read:

http://wahlnetwork.com/2012/12/17/granting-ad-accounts-administrative-access-to-vcenter-sso/
http://longwhiteclouds.com/2012/02/04/the-trouble-with-ca-ssl-certificates-and-esxi-5/

Quick links

Just 2 links I came across I want to peruse more later. 

http://blog.mwpreston.net/vcp-5/vcp-5-objective-4-3-manage-virtual-machine-clones-and-templates/

http://virtuallymikebrown.com/2012/02/05/a-small-virtual-machine-for-a-test-lab/

Info on adding a datastore to multiple hosts in a cluster via PowerCLI:

http://www.vmware.com/support/developer/PowerCLI/PowerCLI41/html/New-Datastore.html

I should be able to modify the code in example 2 to add NFS storage to a group of hosts:

Get-VMHost | New-Datastore -Nfs -Name NFS1 -Path "/mnt/nfs1/nfs11/test1" -NfsHost 10.91.246.21
Looks easy, right? How hard could that be? ;)

Licensing

3 editions of vCenter Server:
VMware vCenter server for Essentials (bundled with kits)
VMware vCenter Server Foundation (up to 3 vSphere hosts)
VMware vCenter Server Standard (limited only by sizing limits; includes vCenter Orchstrator)

3 editions of vSphere
vSphere Standard Edition
vSphere Enterprise Edition
vSphere Enterprise Plus Edition

Note: version 4 had an edition called VMware vSphere Advanced.  It no longer exists, and users who purchased Advanced Edition licenses are entitled to use Enterprise Edition in vSphere 5

Physcial CPU cores and Phsycial max of 256GB of RAM limits go away in vSphere 5. However, they introduced vRAM entitlements
vRAM entitlements:
Standard: 32GB
Enterprise: 64GB
Enterprise Plus: 96 GB

These entitlements are per license.  vSphere 5 is licensed on a per-processor basis.  So, a physical server with 2 physical CPUs needs 2 licenses. There is no limit on the number of cores or the amount of RAM that can be physically installed in the server. If you use 2 licenses of Enterprise Plus, you get a vRAM entitlement of 192 GB of RAM.
vRAM entitlements can be pooled across all the hosts being managed by one vCenter server
basic idea is to help organizations move closer to usage-based cost and chargeback models that are more typical of cloud computing or IaaS models.

Memorize the table on pp 18-19 on vRAM entitlement, vCPUs per VM, and what functions are included in each product edition.

All editions of vSphere 5 include support for thin provisioning, vSphere Update Manager, and storage APIs for Data Protection.

On all editions except Essentials, VMware requires at least one year of Support to be purchased.
Essentials kits are all-in-one solutions for small environments (up to 3 hosts with 2 CPUs each and a 32GB RAM entitlement)

You can't buy Essentials kids on a per-CPU basis; they are bundled solutions for three servers

You can buy Acceleration Kits as well. (allows customers an easier way to purchase necessary licenses in one step

**Note: I am aware that VMware has ditched its vRAM licensing as of August 27, 2012.  My notes are based on the book.

High availability, Fault Tolerance, Storage APIs

High Availability (HA)

automated process for restarting VMs that were running on an ESXi host at a time of complete server failure
VMs are migrated to an ESXi host that is part of the HA-enabled cluster (does not use vMotion!!)
Intended to address unplanned downtime, not planned outages
Improved in v5

1. scalability (up to 512 VMs per host and 3000 VMs per cluster)
2. integrates more closely with DRS, giving Ha greater ability to restart VMs in the event of a host failure
3. rewrite of the architecture to Fault Domain Manager, eliminating many of the contraints in earlier versions

by default, vSphere HA does not provide failover in the event of a guest OS failure, but it can.
Remember that HA will have service interruptions. If vSphere HA restarts the VM, the applications or services provided by that VM are unavailable during the restart

vSphere Fault Tolerance (FT)

Even greater level of high availability than HA can provide.
Eliminates any downtime in the event of a host failure
uses vLockstep technology
utilizes a mirrored secondary VM on a separate physical host that is kept in lockstep with the primary VM.
If the physical host on which the primary VM is running fails, the secondary VM can immediately step in and take over without any loss of connectivity.
Also, if the physical host on which the secondary VM is running fails, the mirrored secondary VM will be automatically recreated on another host.
In the event of multiple host failures, vSphere HA will reboot the primary VM on another available server and recreate the secondary VM
In this version, vSphere FT can work in conjunction with vMotion and is integrated with DRS, but it does require EVC

vSphere Storage APIs for Data Protection and VMware Data Recovery

Two key components:

1. vSphere Storate APIs for Data Protection (VADP)
2. VMware Data Recovery (VDR)

VADP is a set of APIs that backup vendors leverage to provide enhanced backup functionality of virtualized environments
enables functionality like file-level backup and restore; support for incremental, differential, and full-image backups
supports multiple storage protocols
provides for native integration with backup software
NOT a backup solution, it just provides the framework for making backups possible.
VMware Data Recovery (VDR) is VMware's backup tool that leverages VADP to provide a full backup solution for smaller vSphere environments