Installing vCenter – CA Signed Certificates with the Certificate Automation Tool

With the installation of vCenter 5.1 components (just like vSphere hosts) the components are installed with self signed certificates. No problem when you trust your own systems, but securing your environment when you have different interactions with the services (for example the corporate users of the vSphere web clients) will lead you to implement signed certificates. Especially when the most environments have their own CA infrastructure, use it!

There are seven separate components in vCenter Server 5.1 that utilize certificates to encrypt communication. We want don’t want to spend time generating and installing certificates for each component, we want automation! And what do we have that, the VMware SSL Certificate Automation Tool. Well…..not a complete automation, but automated enough to help updating the vCenter related services (and rollbacking in case something is going wrong). That beats doing it all on yourself.

The VMware SSL Certificate Automation Tool can be used to generate certificate signing requests for the vCenter services, that can be send to the CA to be signed. The Automation Tool can then be used to implement the certificates for the services. Use the planner option to be presented with the correct order of steps. Possibly in a second screen.

Procedure

The procedure is as follow:

– Download the VMware SSL Certificate Automation Tool.
– Generate CSR’s for the vCenter Service with the automation tool.
– Sign the CSR’s in the CA.
– Creating the PEM chain files.
– Update the vCenter services in the correct order. (Replacing one service and not re-trusting the interacting services will have some errors as result)

Example

The installation starts with downloading the Automation Tool at https://my.vmware.com/group/vmware/get-download?downloadGroup=SSL-TOOL-101. Download and unzip to a preferred location. For this example I have unzipped to D:Scripts, a the folder structure starts from SSLAutomationTool1.0.1.

Predefining the default values in the tool helps prevent typing errors and save time, you can edit ssl-environment.bat. This lets the tool automatically include specific information that you have defined as the default instead of prompting you for it. Details for the serveral services we will fill in after generating the CSR (plus key’s) and PEM chain files.

For now we add the parameters to: Common parameters (SSO and VC users) and parameters that will be used to generate a CSR (fqdn, ips, country, state, organization and so on).

Now we start the tool: ssl-updater.bat.

image

We first generate the CSR by choosing action 2. At that menu we choose every service we have to update. With the common parameters we only have to take the default entries (be sure to check them). The CSR’s are saved to the requests directory. There you will also find the keys. Repeat until you have selected all options.

With all the necessary services done, we have a collection of CSR’s to be signed by the CA. The process of signing differs from what product is used as CA. From every directory the rui.csr is the signing request we have to upload or open and copy the contents.

For example Microsoft CA works with a web site you can use to request the certificate:

image

Choose the request a certificate and an advanced request. Request an BASE64-encoded web server certificate, with the option Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.

You will have to repeat the request for every service you wish to update.

We continue when we have received the signed requests. Save the .crt file in a appropriate location and open the file to verify the proper names.

After repeating this we have to create PEM files with the complete certificate chain in one file. Depending on the hierarchy used in the certificate infrastructure you will need the root and all the intermediate (if any) Base64-encoded certificates. Get them from your trusted certificate store or from the CA signing service. Also save these files to the directory. Open a text editor and create a PEM file (name including the service). Open the signed certificate and copy and paste to the PEM, open any intermediate certifcate and past the contents to the PEM (after the —-END Certificate — of the certificate), repeat for any other intermediate. Finish by copying the root certificate in the PEM. This should look like something like this (truncated):

——-BEGIN CERTIFICATE——- <——-Certificate
MIIFxTCCBK2gAwIBAgIKYaLJSgAAAAAAITANBgkqhkiG9w0BAQUFADBGMRMwEQYK
CZImiZPyLGQBGRYDbmV0MRYwFAYKCZImiZPyLGQBGRYGbW5uZXh0MRcwFQYDVQQD
Ew5tbm5leHQtQUQtMS1DQTAeFw0xMzAyMDExNjAxMDNaFw0xNTAyMDExNjExMDNa
SMhYhbv3wr7XraAnsIaBYCeg+J7fKTFgjA8bTwC+dVTaOSXQuhnZfrOVxlfJ/Ydm
NS7WBBBFd9V4FPyRDPER/QMVl+xyoaMGw0QKnslmq/JvID4FPd0/QD62RAsTntXI
ATa+CS6MjloKFgRaGnKAAFPsrEeGjb2JgMOpIfbdx4KT3WkspsK3KPwFPoYza4ih
4eT2HwhcUs4wo7X/XQd+CZjttoLsSyCk5tCmOGU6xLaE1s08R6sz9mM=
——-END CERTIFICATE——-
——-BEGIN CERTIFICATE——- <——-Intermediate Certificate
MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG
K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq
GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr
/Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC
TLqwbQm6tNyFB8c=
——-END CERTIFICATE——-
——-BEGIN CERTIFICATE——- <——-Root Certificate
MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG
K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq
GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr
/Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC
TLqwbQm6tNyFB8c=
——-END CERTIFICATE——-

Close and repeat. And we now have the PEM chain’s for all (or you choice) of vCenter services.

Now we need to implement them at the appropriate services. First edit the ssl-environment.bat for the services and include location for keys and pem files. When you are finished the menu options Update will handle the updates for you.

Here is a screenshot of the the updating of the SSO service. It handles the update and restarting of the appropriate services.

image

Type 3 to return to the main menu and repeat for the other actions the planner .

In case something goes amiss rollback and you can find the service-update-ssl.log in the logs directory of the automation tool.

One more thing, when updating the vCenter certificate one of the wanted information is “Enter vCenter server original database password”. This is a somewhat confusing line. But it is the vCenter database user password (ODBC System DSN Password for connection to VCDB) that is requested. That is either a Windows user or a SQL user.

—- Enjoy updating!

Learned Lessons – Nexus 1000V and the manual VEM installation

At an implementation project we implemented 24 ESXi hosts and used the Nexus 1000V to have a consistent network configuration, feature set and provisioning throughout the physical and the virtual infrastructure. When I tried to add the last host to the Nexus it failed on me with the InstallerApp (Cisco provided java installer that adds the VEM and adds the ESXi host to the configured DVS and groups). Another option is to use Update Manager (the Nexus is a appliance that can be updated by update manager), but that one threw an error code at me. I will describe the symptoms a little bit later, first some quick Nexus product architecture so you will have a bit understanding how the components work, where they are and how they interact.

Nexus 1000V Switch Product Architecture

Cisco Nexus 1000V Series Switches have two major components: the Virtual Ethernet Module (VEM), which runs inside the hypervisor (or with other words on the ESXi host), and the external Virtual Supervisor Module (VSM), which manages the VEMs (see figure below). The VSM can be either a virtual appliance or an appliance in the hardware device (for example in the physical Nexus switch).
The Nexus 1000V replaces the VMware virtual switches and adds a Nexus Distributed Switch (Enterprise Plus). Uplink and portgroup definitions are bound to the Cisco ethernet profile configurations. The VEM and VSM use a control and data link to exchange configuration items.

Configuration is performed through the VSM and is automatically propagated to the VEMs. Virtualization admins can pick up these configuration to select the portgroups at VM provisioning.

image

Symptoms to a failing installation 

The problem occurred as follows:

– Tried to install the VEM with the InstallerApp. The installer app finds the host and when the deployment is done, it stops when adding the hosts to the existing Nexus DVS. This happens somewhere from moving existing vSwitches to the DVS. Error presented is: got (vim.fault.PlatformConfigFault) exception.

– Checked the status of the host in update manager and this showed a green compliant Cisco Nexus Appliance. This probably delayed me a bit, because it really wasn’t.

– Tried to manually add a host to the Nexus DVS in the vSphere Webclient. This gave an error in the task. Further investigation let me to the line in the vmkernel.log: invalid Net_Create: class cisco_nexus_1000v not supported. Say what?

– With the Cisco support site and a network engineer I tried some VEM cli on the host. But hey wait vem isn’t there. An esxcli software vib list | grep cisco doesn’t show anything either (duh). While on an VEM installed ESXi host this shows the installed VEM software version. So Update Manager is screwing with me.

Manual Installation

That leaves me with trying to manually install the VEM. With the Cisco Nexus 1000 installation guides the working sollution is as follows:

  • The preferred Update Managers does not work. It fails with an error 99.
  • copy the vib file containing the VEM software from the VSM homepage using the following url: http://Your_VSM_IP_Address/. Check an ESXi host that is installed for the running version (esxcli software vib list | grep cisco). Download this file (save as).
  • Upload this file to a location where the host can access this. On the host or a datastore accessible from the host. I did the latter as the host did not have direct storage. Used WinSCP to transfer the files to a datastore directory ManualCiscoNexus.
  • On the hosts I added this vib by issuing the following command:

esxcli software vib install -v /vmfs/volumes/<datastore>/ManualCiscoNexus/Cisco_bootbank_cisco-vem-v160-esx_4.2.1.2.2.1.0-3.1.1.vib

  • vem status -v now gives output. Look for VEM Agent is running in the output of the vem status command.
  • vemcmd show port vlans only shows the standard switches. Communication with the VSM is not yet there.
  • I added the host manually to the Nexus DVS and success. When migrating the standard vmkernel management port to the DVS groups the hosts is also visible on the VSM. Communication is flowing and the host is part of the Nexus 1000v.

I hope this post will help when you experience the same problem, and also learns you a little about the Nexus 1000V Product Architecture.

vCenter Collectors – Syslog and Dump collector (with some PowerCLI)

With stateless VMware Auto deploy hosts, or hosts with SD or USB Stick installation, these hosts typically do not have local storage attached. There will be no space assigned to the scratch partition for dumps and logs. Defaulted they will be stored in RAM (/tmp/scratch). After a reboot the /tmp will be cleared.

We need to redirect the scratch partition to a shared VMFS and a central log / dump location. Scratch redirection is performed with the following steps:

  • Decide on a shared data store where preferably all hosts in a physical location connect, beware of physical location Bias. The volume is like /vmfs/volumes/UUID
  • Create a locker directory on that data store, for example .locker-Hostname
  • Connect to the ESXi host with Connect-ViServer -Server <Hostname>
  • Set the scratch partition with Set-VMHostAdvancedConfiguration -Name “ScratchConfig.ConfiguredScratchLocation” -Value “/vmfs/volumes/UUID/.locker-Hostname”
  • Maintenance mode (Set-VMHost -VMHost <host> -State ‘Maintenance’) and reboot (Restart-VMHost <host>).

With central management of an environment, a central log/dump location is recommended (also for stateful deployments this should be a standard). For a central log we need some services added to the vCenter:

– dump collector; collecting ESXi dumps of the infamous PSOD.

– syslog collector; collecting the host logs or syslogs.

On Windows we use the vCenter installer to install both services. With the vCenter Server Appliance (VCSA) these service are bundled with the appliance.

image

Dump Collector installation


Installation of the dump collector is straightforward. Determine in advance where your dumps will be saved, how much space (2GB default is sufficient for most environments) and to do a standalone or integrate with vCenter (last option is preferred for single management).

image

(if you got a data disk, change the default C: parameters)

image

And fill in the vCenter and account details on the next screen. The account will be used to register the extension.

Syslog Collector installation

Just like the dump collector the installation is straightforward. Determine the syslog rotation and log location. And start the installation to the VMware vCenter Server.

image

(if you got a data disk change the default C:).

image

Change the ports when needed for your environment, standard default is sufficient for most environments.

And finish the installation.

If you still use the vSphere client, then install both service add-ins.

Configuring the hosts

The ESXi hosts need to be configured to use the remote syslog and network dump location. You can use ESXi CLI and the advanced system settings or you can use PowerCLI. If we use PowerCLI for syslog we can use the Set-VMHostAdvancedConfiguration cmdlet and for the dump collector we need the Get-EsxCli cmdlet to use esxcli.

When we do this for a list of hosts we can use the following script. Just replace the vcenter servername at $vCenterFQDN and vCenter IP at the $esxcli.system.coredump.network.set line.

#set vCenter IP/FQDN

$vCenterFQDN = “<vcenter server>”

#connect to vCenter

$viServer = Connect-VIServer -Server $vCenterFQDN -Credential (Get-Credential)

#get all hosts in vCenter so we can cycle thru them

$Hosts = Get-VMHost

#cycle through each host

ForEach ($VMHost in $Hosts)

{

   #Add Syslog to Syslog Collector

   Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.global.logHost -Value $vCenterFQDN

    # Add DUmp to collector

    $esxcli = Get-EsxCli -vmhost $VMHost

    $esxcli.system.coredump.network.set($null, “vmk0”, “<vcenter ip>”, “6500”)

    $esxcli.system.coredump.network.set($true)

    #write out host name that was changed:

    write “Updated host: $VMHost”

}

#disconnect from ESX server

Disconnect-VIServer -Server $viServer -Confirm:$false

Checking the configuration

A host can be checked by going to the dump directories and checking the contents, just like the following for a host’s syslog directory.

image

I don’t have a crashed ESXi host at my disposal, so it is easier to check on the hosts via esxcli:

esxcli system coredump network get.

The IP and port should be in the output at:

Network Server IP:

Network Server Port: 6500

VMware IO Analyzer – lab Flings

Flings in VMware labs is a great place for (very) useful tools or applications. This time I want to blog about a fling I often use in a test phase for implementation projects or in health assessments, see what synthetic load an environment can handle and if your vSphere design is up to the right io charactics and capacity.

Important in these kinds of test is your test methodology and plan: Assess, Filter test, plan, collect, analyse and report. With several of these steps IO Analyzer can be the player.

IO analyzer can configure, schedule and run several IOmeter workloads or replay vSCSI traces.

Download at:

http://labs.vmware.com/flings/io-analyzer

Import ovf to your environment. Start with more then one, so you have some dedicated workers thoughout your environment.

After deployment change the second vmdk for the defaulted “small” configuration to approx 4GB plus (and Thick Eager). Why? Because the small amount of disk is used as disk test and fits in most storage cache. We need to get out of that and hit some real scenario’s.

One (or yes two) more things, logon to the consoles of all the appliances. Open a console, choose first option or press enter and login with root and password vmware. *ssst a very secret vmware user*. An other usage of the console is checking or monitoring the IOmeter tests i the console when they are running.

Demo:

Type down one of the ip’s or hostnames of the appliances, and will use that one as the controller.

Open a browser (chrome or firefox) and type http:// and you will reach IO analyzer in your environment.

There we have the following options

image

For this I will use the workload configuration to add two tests to two appliances and check the results. Test scheduler is not used, will run immediately.

In this screen we first add the hosts where our test machines are, use the root password to connect to the hosts. When a connection is established the VM’s on that host are visible in the Add Workload Entry. Here you can find all kinds of IOmeter tests.

image

I have created two workloads, one Exchange 2007 on our first appliance and SQL 64K blocks on the other appliance. The duration is changed from the default 120 seconds to a 5 minute (300 seconds) schedule. This configuration is saved as Demo config.

Click on Run Now to let the test run. After a initialization you can see the progress in the console of one of appliances.

image

After completion of the tests you can view the test results in View Test Results (so it is not just a clever name :)).

Here you can check the two tests and the different VM and host metrics saved from IOmeter and esxtop (if there are any, you will get as a bonus the metrics of other VM’s on the hosts). More detailed information about these metrics, see the following URL: https://communities.vmware.com/docs/DOC-9279. Duncan Epping also has a good article about esxtop metrics and more. Go see his site when waiting for the test to finish: http://www.yellow-bricks.com/esxtop/

Here see the results of our Demo tests (I’m not going over in detail in this post).

image

Enjoy stress testing.

PowerCLI collection – Changing PSP

On storage controllers in combination vSphere 5.x multi path IO, path selecting policy should be Round Robin (always check if your storage supports this). When receiving the LUN it happens that those PSP are defaulted to Fixed. You can change the PSP policies of all attached storage with this PowerCLI oneliner (Round Robin in the example):

get-cluster -Name “Cluster” | Get-VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.MultipathPolicy -ne “RoundRobin”} | Set-ScsiLun -MultipathPolicy “RoundRobin”

A little caution when you have storage from mixed storage providers (mixed storage controllers without a SVC,Vplex or such layer).

What does this one do:

– get-cluster -Name ”Cluster”; Retreives the cluster information for Cluster. This command can be replaced with get-datacenter. And pipe’s this information to:

– Get-VMHost; retreives the hosts information (one by one in from the cluster). The output is piped to:

– Get-ScsiLun – Luntype disk | Where-Object {$_.MultipathPolicy -ne “RoundRobin}

Get de Scsi Lun of type disk en checks if this object current policy is not equal with Rond Robin. The output is piped to:

– Set-ScsiLun -MultipathPolicy “RoundRobin. The Lun found with previous part is changed to a RoundRobin policy.

The policies can be set to:

RoundRobin, MostRecentlyUsed, Fixed or Unknown.

PowerCLI collection – Adding hosts to inventory

With several projects at several sites I have come along a nice list of powerCLI oneliners or scripts. I wanted a place to have these scripts in a handy catalogue online, maybe share and learn on some blogging skills (or be a little louder than only 140 characters). I know there are probably already place online, but..how can I learn the bloggin. So here goes…

This example is a script that adds 24 hosts to a vCenter inventory. ESXi is installed, vCenter and PowerCLI are setup. PoweCLI has a connection setup to the vCenter (Connect-VIServer -Server <vc> -Protocol HTTPS). No other automation or scripting is in place.

To add 24 hosts to a inventory:

1..24 | Foreach-Object { Add-VMHost hostesxi$_.example.nl -Location (Get-Datacenter DataCenter) -User root -Password “<replace>” -RunAsync -force:$true}

So what does this one do:

  •  1..24; Amount of hosts. This is piped to the command and used by the $_ variabel. Foreach-Object let’s us loop within the 1..24 range so the hosts number will change every loop.
  • Add-VMHost is actually the PowerCli cmdlet to add the host.
  • hostesxi$_.example.nl. Creates hostnames of hostesxi1_.example.nl till hostesxi24.example.nl in the datacenter DateCenter
  • -User and Password are the user and password for access to the ESXi hosts.
  • -RunAsync option is to let the tasks run parallel.