Migrating vCenter databases from SQL Express to SQL Server

When installing vCenter you will need a vCenter database, an Update Manager database and with vCenter prior to 5.5 also a SSO database (RSA). With the installation you have the option to point to an (existing) dedicated SQL Server, SQL Server on the same Windows host or use the bundled SQL Express with the installer. Depending on the requirements of the environment a design decision is made to which sort of database setup vCenter will communicate. I prefer to separate the database resources and services from the vCenter server and will often option for a dedicated SQL Server or SQL instance part of SQL mirror/cluster. But sometimes the size of the environment is small enough, and a dedicated SQL license is a constrain at the design/build phase, test/pilot grows into production (don’t!), in these situations SQL Express is `fine’. Fine and supported up to a vCenter inventory of max 5 hosts and or 50 VM’s. No.. your vCenter will not crash immediately after adding a sixth, seventh or whatever hosts but performance is degraded to a point that certain sub processes/services are not running within optimal limits. This can have unforeseen consequences to your environment, and this is not what is wanted in production.

This post will combine the migration of SSO, vCenter and Update Manager databases from SQL Express to SQL Server. Yes you are missing a database, vCenter Orchestrator should be migrated as well. Moving this database should be included when using, use the vCenter Orchestrator Configuration service to point to the destination database server (stop orchestrator server to move.) Unfortunately when using the vCenter simple installation with a bundled SQL Express this combination is not used very often or not at all (a shame).

The proces

Migrating SQL Express from the bundled to a full installation includes the following actions:

  • Assess your environment. What database names are used, what users and do you have the correct administration of users and passwords (see the SSO paragraph below for a way to find out the current RSA_DBA and RSA_USER passwords). Decide on whether you want to use the same users, or use new ones. Same goes for ports of destination SQL Server, Authentication mode used here etc. Always check the interoperability of vCenter with the database version. Do they support each other? Use SQL Authentication with the SSO database, vCenter supports Windows authentication (Best Practice) and SQL users.
  • Plan for some downtime of SSO, vCenter and Update Manager. As your moving services and databases from one server to another, the vCenter, SSO and Update Manager services must be stopped before and until the data is copied and configuration is changed, remain down. No VM workload is hit and remain running, and host can be managed via the standalone vSphere client during changes. That is, versions before 5.5 and hardware version 10.
  • Create back-ups of the to change files, registry keys and such before changing values.
  • Migrate the SSO database. This step is only for pre-5.5 versions as SSO in vCenter 5.5 does not use a database.
  • Migrate the vCenter database.
  • Migrate the Update Manager database. That is when Update Manager is installed on the same SQL Express instance. Normally you would see this combination when a SQL Express Windows vCenter server is used.

Migrating the database itself can be done via a SQL Backup and restore (overwrite), detach-copy-attach, or SQL Server Import/Database Copy Wizard. Prior to the database move the appropriate services must be stopped. When doing the copy in one task, stop the services as described in the following paragraphs. If you use the one go and backup or copy is finished stop the SQL Express (VIM_SQLEXP).

SSO

Not sure what your current RSA_USER/RSA_DBA passwords are. Go to the SSOServer\utils directory in an elevated prompt and run the following command:

ssocli manage-secrets -a listallkeys

Stop the vCenter Single Sign-On service. After the service is stopped backup and restore or copy the database files on the destination SQL Server. For SSO two SQL users are created by the installer, RSA_DBA and RSA_USER. RSA_DBA is only used when doing the installation, not needed for this migration. For operational purposes RSA_USER is used. Add the login/user to the destination SQL Server and set the password / required permissions if these where not saved.

Open an elevated command prompt and go to the SSOServer installation directory\utils directory. For example when installed on D:

SSO Directory

Run the following command: 

ssocli configure-riat -a configure-db –database-host newdbhost –database-port newdatabaseport –rsa-user RSA_USER –rsa-user-password password -m master_password

Replace the database hostname, database portnumber 1433 for standard SQL. Add the RSA-USER password and -m master password is the [email protected] password. rsa-user-password works only in combination with rsa-user option.

Open a text editor (also with elevated permissions if required). Open the following two files:

  • SSOServer\webapps\ims\web-inf\classes\jndi.properties

Check the values:

    • com.rsa.db.type=MSSQL.
    • com.rsa.db.instance=RSA. (or whatever your RSA instance dbname is)
    • com.rsa.db.msserverinstance= . Leave empty when using the default MSSQLSERVER instance on the destination server.
    • com.rsa.db.hostname=destinationSQL server.
    • com.rsa.db.port=1433.
    • Note: don’t mix up the com.rsa.instanceName. This is not the SQL Instance, but the SSO instance. This should not be changed.
  • SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties

Check the values:

    • db.url=jdbc:jtds:sqlserver://;serverName=;portNumber=1433;databaseName=RSA. PortNumber has to be added here.
    • db.user=RSA_USER
    • db.pass=yourpassword
    • db.type=mssql
    • db.host=destination SQL server fqdn.

Change the lines appropriately to you environment. Save the files.

Start the vCenter Single Sign-On service.

vCenter System

Stop the vCenter services. Include all vCenter services except Update Manager, Orchestartor or SSO. After the service is stopped backup and restore or copy the database files on the destination SQL Server. Add the login/user to the destination SQL Server and set the password / required permissions if these where not saved. The user needs db_owner on the vCenter database and dbo on MSDB system database on the destination server.

Reconfigure the System DSN to use the destination SQL Server. Use the 64-bit ODBC Manager. When using a default installation of MSSQL server, the default MSSQLSERVER instance does not have to be filled in, connection to the SQL server hostname is enough. When using Windows credentials the ODBC is editted in the service account user. Change the default database to the VIM_VCDB or what ever name you gave it. Test the connection to be sure everything works.

ODBC vCenter

Update the ODBC connection in the registry:

  • Click Start > Run, type regedit, and click OK. The Registry Editor window opens.
  • Navigate to HKEY_LOCAL_MACHINE > SOFTWARE > VMware, Inc > VMware VirtualCenter.
  • Modify the key DbInstanceName and remove the current Value data. Do not delete this key.
  • Modify the key DbServerType and change the Value data from Bundled (the SQL Express) to Custom.
  • Navigate to HKEY_LOCAL_MACHINE > SOFTWARE > VMware, Inc > VMware VirtualCenter > DB.
  • Modify Key 4 and change the ODBC driver to the new driver if not changed check this value and leave intact.
  • Modify key 2 and add the vCenter Server SQL user, for example vcuser.

The password that is set (at key 3) is needed to be change, open an elevated command promp and run the following command:

<install drive>:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe -p

Go to the SQL Server and recreate the SQL Jobs. Launch SQL Server Management Studio on the SQL Server and Navigate to SQL Server Agent. In the Jobs list create the following jobs:

Rollup Job SQL Job Filename
Event Task Cleanup vCenter Database job_cleanup_events_mssql.sql
Past Day stats rollup vCenter Database job_schedule1_mssql.sql
Past Month stats rollup vCenter Database job_schedule3_mssql.sql
Past Week stats rollup vCenter Database job_schedule2_mssql.sql
Process Performance Data vCenter Database job_dbm_performance_data_mssql.sql
Property Bulletin Daily Update vCenter Database job_property_bulletin_mssql.sql
Topn past day vCenter Database job_topn_past_day_mssql.sql
Topn past month vCenter Database job_topn_past_month_mssql.sql
Topn past week vCenter Database job_topn_past_week_mssql.sql
Topn past year vCenter Database job_topn_past_year_mssql.sql

Adding the jobs: browse to Program Files\VMware\Infrastructure\VirtualCenter Server\sql or whever your install path is. Open the corresponding SQL file (Open > file). Select the vCenter database. Execute Query. And repeat for others.

When finding existing jobs, check that the corresponding SQL user in jobs is correct. Right click the Job, properties and check the user. And the database name is correct, if not recreate them.

Check the vcdb.properties file for the corresponding values. On Windows 2008R2 and higher this file is located in C:\ProgramData\VMware\VMware VirtualCenter (check show and uncheck hide files and protected files in explorer). When using SQL authentication set the vcdb.properties file value url= ;integratedSecurity\= to False. When using Windows authentication set this value to true. the rest should match sqlserver://<destination server>\\SQL instance;databaseName=\dbname; dbtype=mssql.

It is important that the Windows user has the required permissions on the SQL database and the VMware VirtualCenter Management Webservices and VMware VirtualCenter service is set to run as this user.

Remove the depend on SQL Express in the vCenter services. Open regedit and goto HKLM\System\CurrentControlSet\Services\. Remove the SQL Express service value from the DependOnService Multi string in the vCenter services where set. Do not delete the value, just removing the VIM_SQLEXP from the data is enough. The other dependencies should remain intact. Start the vCenter services. Check the logs if everything starts up correctly.

Update Manager

Last one. Stop the vSphere Update Manager services. After the service is stopped backup and restore or copy the database files on the destination SQL Server. Add the login/user to the destination SQL Server and set the password / required permissions if these where not saved. The user needs db_owner on the Update Manager database and dbo on MSDB system database on the destination server.

Reconfigure the System DSN to use the destination SQL Server. Use the 32-bit ODBC Manager. On Windows 2008R this is located in the C:\Windows\SysWOW64\odbcad32.exe. On Windows 2012 you can select the 32-bit from the apps menu.
When using a default installation of MSSQL server, the default MSSQLSERVER instance does not have to be filled in, connection to the SQL server hostname is enough. When using Windows credentials the ODBC is editted in the service account user. Change the default database to the VIM_UMDB or what ever name you gave it. Test the connection to be sure everything works.

Next go to your installation directory of Update Manager, for example D:\Program Files (x86)\VMware\Infrastructure\Update Manager\, and launch VMwareUpdateManagerUtility.exe (sometimes takes a while to start up). Login with an admin in vCenter (should have the user that installed and registered Update Manager to the vCenter in the user screen.
Click on database settings. Check the DSN name is the same you editted and tested. Type the SQL username and password twice. Click apply.

VMware Update Manager Update Manager database Settings

Start the vSphere Update Manager service.

After all changes be sure to check the vCenter Service Status if all is in green here. Some tracking or statistics roll up need the jobs to complete. This can take some time.

– Hopefully this helps you migrating from your SQL Express.

Sources: vmware.com.

Design Choice: vCenter Windows or vCenter Server appliance?

A question that is again popping up lately is whether customers must go for the vCenter Server Appliance (VCSA) or a Windows based vCenter system. Yes I have my favorite, but I will often go for the consultant answer: it depends. But on what does this depend? I will try and sum up some of the arguments to take in to account.

But first, what is this VCSA your talking about?

You may or may not know vCenter can be deployed in two ways, the commonly known vCenter installation on a Windows based host(s) or deploying vCenter as a appliance. The appliance is a 64-bit Linux based (SUSE Linux Enterprise for the distro interest) virtual machine with an embedded database (vPostgres) or the option to use an external Oracle database. The appliance has been around since it’s introduction with version 5.0, but has not been really production worthy since the 5.5 version. With the 5.5 version the vCSA supports managing 100 hosts and 3000 VM’s with the embedded database versus the 5 host and 50 VM’s with the previous version. The 5.5 limit for the external Oracle database is even higher (100 hosts and 10000 VM’s), but to be honest I haven’t seen that combination around.

VCSA

The appliance is build with the following vCenter services: vCenter services (vCenter, Inventory, SSO, Web Client) and services such as syslog collector, dump collector, AutoDeploy (and capable of doing DHCP/TFTP).

vCSA services

What are the arguments to consider?

  •  For organizations that are having a DBMS install base of MSSQL there is currently no support of using this in combination with the vCSA where the Windows based happily works with MSSQL (via ODBC DSN). If there is no experience, no business investment or no DBA’s wanting to manage an additional DB flavor you can stop right here (or wait when a next version of VCSA will hopefully support MSSQL as an external database).
  • Windows vs Linux install base and license. You got your Windows based organizations and you got your Linux based organizations. More often they are mixed. Got Windows data center licenses sorted out for your organization, another Windows VM is often not an issue. Need to save some licensing or just don’t want to invest in new, VCSA will probably be more of your taste. Beware that you can’t install Update Manager on VCSA, this still needs to be set up on a Windows host which can be an existing. Same goes for your trusty toolset, PowerCLI needs Windows PowerShell, RVTools need .Net. These tools can be easily put on a management server or admin workstation (where the toolset are supposed to be on the first place), they connect to VCSA just as vCenter on Windows. Need remote CLI? Added by deploying the vSphere Management Assistant (vMA) appliance. For Windows only based IT departments a knowledge investment is needed to support Linux based appliances.
  • vCenter Orchestrator. Windows vCenter installers installs vCenter Orchestrator on your Windows system or you can use the Windows installer to install Orchestrator on another designated Windows server. vCenter Orchestrator is not installed on the VCSA, but there is the vCenter Orchestrator appliance you can deploy next to it.
  • Linked mode. Want a single management plane, or in other definition a Single Pane of Glass, for multiple vCenter instances linked mode is required. VCSA does not support Linked mode where vCenter on Windows does. There is an other option, leveraging the built-in SSO Replication by joining all (or some you want to share) SSO instances under a common domain. A benefit next to the automatic synchronization, is that you will be able to see the inventory of all SSO instances from whatever Web client.
  • Experience with appliances. I think appliances add simplicity for deployment of the vCenter systems, but can be a hassle to (re)configure and for example do database back-up dumps when you don’t have any experience with Linux. Don’t worry, VCSA has an admin Web interface or the vSphere Web client for configuration tasks and when you need the console commands are documented. The procedures are on Documentation Center, Knowledge Base and widely blogged about in the community. Go ahead and learn appliances as there are a lot of good ones.
  • Monitoring/Back-up. Monitoring of vCenter server services and system parameters. If you use a Windows based or agent type of monitoring, the Windows based monitoring does not work for a Linux appliance. Check if your monitoring solution supports Linux. Same goes with thinking about your back-up strategy and how to back-up the VCSA.
  • Physical vCenter. When there are reasons to install the management of your virtual infrastructure on a physical server outside the virtual infrastructure, than Windows vCenter is the only option. VCSA is a virtual appliance which needs to be deployed on virtual environment. Depending on the requirements, you could also go with an VCSA deployment in an other vSphere cluster than the one vCenter is managing.
  • IPv6. IPv6 is not supported on the VCSA and is supported on Windows. If you happen to use IP addresses in the vCenter configuration or connecting to vCenter these are IPv4 only. To connect to vCenter system in an IPv6 environment you must use the fully qualified domain name (FQDN) or host name of the vCenter Server. FQDN is preferred to be used in any case.
  • vCenter Server Heartbeat. Use this solution to protect your vCenters, you cannot use this with the VCSA. But on the other hand, you should be working on a replacement solution (you have some time) as VMware announced EOA of all vCenter Server Heartbeat versions. See details here: http://www.vmware.com/nl/products/vcenter-server-heartbeat. Leveraging vSphere HA instead for example.

Note: These arguments are not ordered by importance, that’s is up to the organization in question.

And what do you think?

I have a favorite, but do keep in mind that there is no one size fit’s all. I happen to like the appliance way of deployment, simplicity at your fingertips and no abuse from Windows admins logging on and hogging the vCenter resources (sizing) with OS / RDS footprint. Or worst destroying to system with all those I needs this and this tool for my work.

Yes you need Windows for Update Manager, which can be combined with other update management tools. And yes you need your Windows based tool set somewhere, are those supposed to be on the vCenter server in the first place?
When a design decision is made to use the Windows based vCenter, please use it as an infrastructure server. Don’t abuse it as an administrator playground (aka RDS server, aka steppingstone, aka PowerCLI test heaven). I like to remove the GUI aka use core when Windows is the design choice (PowerCLI Get-WindowsFeature *gui* | Uninstall-WindowsFeature –Restart). From Windows version 2012 switching from GUI to core does not require a reboot.

What do you think? I’m interested in your views. Let me know in the comments.

Sources: vmware.com, microsoft.com

Protecting vCenter services, what is around (comes around)

Depending on your environment there is a need to protect vCenter or some of the services included in the vCenter system. A big question to ask yourself is what kind of downtime can you have according to your service levels and what kind of options do we have or need to have in place?

What will go down if you lose a vCenter component?

Like said this depends on your environment and components using vCenter services to connect to and from. A “plain” server virtualization workload for one company is different than a VD workload in a high demanding organization. The latter probably needs the ability to provision a little more urgent then the first example. Want to deploy a vCOPS vApp or VD Desktop, well wait until your vCenter is back. Using solutions like VMware Data Protection requires an operational vCenter with a functioning vCenter Single Sign-On server to restore a virtual machine. Losing that part of your environment could impact your recovery options seriously. Manage/Edit some VM version 10? How will you do that without vSphere Web client? You can’t. Have a HA or DRS cluster? Well HA will still partially function, it will react with restarts when needed. But to add to the cluster will need vCenter to make this posible. DRS needs vCenter to function in manual or automatic mode. And these are just a few examples.
Important to keep in mind, running VM’s will keep on running and HA will keep on HA’ing, no need to panic there.

Let see which components make up vCenter, a little vCenter architecture to start with.

A “standard” vCenter is made up of the components vCenter SSO (Single Sign-on), Lookup Service, Inventory Service, vSphere Web Client and the vCenter Server (with all of it’s services) itself. Optional services are Dump collector, Syslog Collector and Auto Deploy (and optionally TFTP and PXE DHCP service, but they can be on a separate system so not included in the model as a part). vCenter is also expanded by Update Manager, vCOPS and all sorts.

image

What are your standard protecting options?

  • Do nothing
    Not advisable, but if you are sure, have a small (just a few hosts and VM’s) environment and have an insight of your environment (or use some scripting to dump your configuration), you could do nothing. You lose part of your services and (in worse case) will have to manually rebuild vCenter and your configuration. You will lose any trending information. Recovery time is typically measured in days, and requires manual intervention.
  • Back-up Restore or Replication.
    Backup and restore should be an essential part of any availability solution, exclamation mark. This provides a recovery method utilizing tape, disk, replication or snapshot technology. This also enables a recovery method when data corruption occurs, depending on the solution that is. If data is corrupt on the primary VM then a replication to the recovery VM can occur after this moment. vCenter VM replication from primary to recovery site should be well monitored (and tested with SRM plans for example). Preferably used on several layers, application and application data (for example databases, certificates, logs, dump locations etc.). Be sure to know your backup  and recovery steps (look in the VMware KB’s for backing up the vCenter Server Appliance services and embedded vPostgres database), document, practice and test them. Recovery time is typically measured in hours or days, and typically requires manual intervention.
  • MS SQL Log shipping – database only
    A simple and cost effective solution. You can use log shipping to send transaction logs from one database (the primary database) to another (the secondary database) on a constant basis. Continually backing up the transaction logs from a primary database server and then copying and restoring them to a secondary database server keeps the secondary database nearly synchronized (depending on your plan) with the primary database. The destination server acts as a cold standby or backup server. Your destination server can also act as primary database for other databases so you will have some sort of active-active instead of a cold standby. Be ware of licensing in this case, log shipping target only or serving database is a different license show! Has to be setup for every database, include your vCenter, Inventory, SSO and such. Recovery time is depending on your plan, but can be minutes or hours. Requires manual intervention to fail over from primary to secondary.
  • SQL mirror / clustering – database only
    Depending on the license of MSSQL these are a more robust solution then the previously mentioned SQL log shipping. These have data replication mechanism in place and have the ability to automatically detect failures and do there fail overs with out manual intervention. Mostly used with a Witness out side the cluster/mirror pair to act as a tie breaker to prevent split brain scenario’s in case of partial failures. Mirroring, clustering has to be setup  for every database, include your vCenter, Inventory, SSO and such. Clustering can also be done per instance with it’s included databases. Oracle will have it’s own clustering, with Oracle RAC for example. Recovery time is typically measured in minutes. No intervention to fail over.
  • Hypervisor HA.
    Hypervisor HA will start your VM after a host failure or VMtools timeout. The time it takes to recover is depending on your amount of free slots, your priority of vCenter vs the other workload and the amount of VM’s needed to restart. Depending on your environment this can take some time to start up. Hypervisor HA will not protect against service failures as it will not monitor any application components, it will also not protect against any data corruption. Hypervisor HA is to be used in conjunction with one or more other protection options. For example a vCenter system on HA and SQL databases on MSSQL Cluster. Recovery time is typically measured in minutes or hours depending on your consolidation ratio and restart settings.
  • App Aware HA.
    If you have the correct edition and have the application aware components in place. Monitors the application and if it goes down, it can be restarted. There is no app aware HA specifically for vCenter yet. But you can protect parts of the applications with app HA, for example MSSQL services. Recovery time is typically measured in minutes or hours.
  • FT
    That is currently a no no. Why did I put it up here? Because it comes up as a question once in a while. FT creates virtual machine “pairs” that run in lock step—essentially mirroring the execution state of a virtual machine. This only protects against host or VM failures. Services that go down or corruption in the application data will be mirrored to the secondary VM.
    FT in vSphere 5.5 is still limited to 1 vCPU, and with a small inventory you still need a minimum of 2vCPU. Same goes for for example a database server these also tend to have more vCPU’s. Yes this has been a issue all along for FT, and we know from following those VMworld sessions demo’s that there is work in progress on multiple vCPU FT, but unfortunately this is not yet released. But a similar technique is next up.
  • vCenter Server Heartbeat
    vCenter Server Heartbeat is a separately licensed vCenter Server plug-in that provides protection of your vCenter system, (physical or virtual). Next to protecting against host failures, heartbeat adds application-level monitoring and intelligence of all vCenter Server components. Heartbeat replicates changes to a cloned virtual machine. The cloned virtual machine can take over when a failure event is triggered.
    imageThe vCenter recovery can be accomplished by restarting the vCenter service, by restarting the entire application, or by the entire failover of the vCenter system. Use in conjuction with a data protection like SQL mirroring to protect against corruption. Recovery time is measured in minutes and requires no manual intervention.
  • Scale out / HA service pair
    Move some of your vCenter services to other components or use multiple same role servers to provide high available and load balanced services. Not all of the vCenter services can be separated this way, but for example SSO can be. Those high availability service are placed behind a third-party network load balancer (for example, Apache HTTPD, vCloud Networking and Security vShield Edge load balancer or load balance appliance like Netscaler).
    imageMove logs to a log insight server, move statistics to vCOPS. Keep vCenter lean and mean.

Conclusion

vCenter Server Heartbeat is a done package for protecting your vCenter server system, but this is at an additional cost. More often you will have some back-end services, like Oracle/MSSQL clustering and back-up restore/replication solutions, already in place or products with a similar need. A combination of protection is the preferred way to utilize those in or to be in place solutions with the need for protection and the allowed recovery/down time. But this is the main thing, know your environment, know how the components interact, know what is needed at which time and know what will be (temporary) unavailable when services are down. Protect against unavailability, corruption and please randomly test to be sure all components are working as expected (even the manual procedures).

And yes sure there will be some other great options out there like a script collection or cold standby solution et al….. but hey isn’t that what the comments section is about. Tell me yours. Share.

– Happy managing your environment!

Installing vCenter – CA Signed Certificates with the Certificate Automation Tool

With the installation of vCenter 5.1 components (just like vSphere hosts) the components are installed with self signed certificates. No problem when you trust your own systems, but securing your environment when you have different interactions with the services (for example the corporate users of the vSphere web clients) will lead you to implement signed certificates. Especially when the most environments have their own CA infrastructure, use it!

There are seven separate components in vCenter Server 5.1 that utilize certificates to encrypt communication. We want don’t want to spend time generating and installing certificates for each component, we want automation! And what do we have that, the VMware SSL Certificate Automation Tool. Well…..not a complete automation, but automated enough to help updating the vCenter related services (and rollbacking in case something is going wrong). That beats doing it all on yourself.

The VMware SSL Certificate Automation Tool can be used to generate certificate signing requests for the vCenter services, that can be send to the CA to be signed. The Automation Tool can then be used to implement the certificates for the services. Use the planner option to be presented with the correct order of steps. Possibly in a second screen.

Procedure

The procedure is as follow:

– Download the VMware SSL Certificate Automation Tool.
– Generate CSR’s for the vCenter Service with the automation tool.
– Sign the CSR’s in the CA.
– Creating the PEM chain files.
– Update the vCenter services in the correct order. (Replacing one service and not re-trusting the interacting services will have some errors as result)

Example

The installation starts with downloading the Automation Tool at https://my.vmware.com/group/vmware/get-download?downloadGroup=SSL-TOOL-101. Download and unzip to a preferred location. For this example I have unzipped to D:Scripts, a the folder structure starts from SSLAutomationTool1.0.1.

Predefining the default values in the tool helps prevent typing errors and save time, you can edit ssl-environment.bat. This lets the tool automatically include specific information that you have defined as the default instead of prompting you for it. Details for the serveral services we will fill in after generating the CSR (plus key’s) and PEM chain files.

For now we add the parameters to: Common parameters (SSO and VC users) and parameters that will be used to generate a CSR (fqdn, ips, country, state, organization and so on).

Now we start the tool: ssl-updater.bat.

image

We first generate the CSR by choosing action 2. At that menu we choose every service we have to update. With the common parameters we only have to take the default entries (be sure to check them). The CSR’s are saved to the requests directory. There you will also find the keys. Repeat until you have selected all options.

With all the necessary services done, we have a collection of CSR’s to be signed by the CA. The process of signing differs from what product is used as CA. From every directory the rui.csr is the signing request we have to upload or open and copy the contents.

For example Microsoft CA works with a web site you can use to request the certificate:

image

Choose the request a certificate and an advanced request. Request an BASE64-encoded web server certificate, with the option Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.

You will have to repeat the request for every service you wish to update.

We continue when we have received the signed requests. Save the .crt file in a appropriate location and open the file to verify the proper names.

After repeating this we have to create PEM files with the complete certificate chain in one file. Depending on the hierarchy used in the certificate infrastructure you will need the root and all the intermediate (if any) Base64-encoded certificates. Get them from your trusted certificate store or from the CA signing service. Also save these files to the directory. Open a text editor and create a PEM file (name including the service). Open the signed certificate and copy and paste to the PEM, open any intermediate certifcate and past the contents to the PEM (after the —-END Certificate — of the certificate), repeat for any other intermediate. Finish by copying the root certificate in the PEM. This should look like something like this (truncated):

——-BEGIN CERTIFICATE——- <——-Certificate
MIIFxTCCBK2gAwIBAgIKYaLJSgAAAAAAITANBgkqhkiG9w0BAQUFADBGMRMwEQYK
CZImiZPyLGQBGRYDbmV0MRYwFAYKCZImiZPyLGQBGRYGbW5uZXh0MRcwFQYDVQQD
Ew5tbm5leHQtQUQtMS1DQTAeFw0xMzAyMDExNjAxMDNaFw0xNTAyMDExNjExMDNa
SMhYhbv3wr7XraAnsIaBYCeg+J7fKTFgjA8bTwC+dVTaOSXQuhnZfrOVxlfJ/Ydm
NS7WBBBFd9V4FPyRDPER/QMVl+xyoaMGw0QKnslmq/JvID4FPd0/QD62RAsTntXI
ATa+CS6MjloKFgRaGnKAAFPsrEeGjb2JgMOpIfbdx4KT3WkspsK3KPwFPoYza4ih
4eT2HwhcUs4wo7X/XQd+CZjttoLsSyCk5tCmOGU6xLaE1s08R6sz9mM=
——-END CERTIFICATE——-
——-BEGIN CERTIFICATE——- <——-Intermediate Certificate
MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG
K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq
GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr
/Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC
TLqwbQm6tNyFB8c=
——-END CERTIFICATE——-
——-BEGIN CERTIFICATE——- <——-Root Certificate
MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG
K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq
GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr
/Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC
TLqwbQm6tNyFB8c=
——-END CERTIFICATE——-

Close and repeat. And we now have the PEM chain’s for all (or you choice) of vCenter services.

Now we need to implement them at the appropriate services. First edit the ssl-environment.bat for the services and include location for keys and pem files. When you are finished the menu options Update will handle the updates for you.

Here is a screenshot of the the updating of the SSO service. It handles the update and restarting of the appropriate services.

image

Type 3 to return to the main menu and repeat for the other actions the planner .

In case something goes amiss rollback and you can find the service-update-ssl.log in the logs directory of the automation tool.

One more thing, when updating the vCenter certificate one of the wanted information is “Enter vCenter server original database password”. This is a somewhat confusing line. But it is the vCenter database user password (ODBC System DSN Password for connection to VCDB) that is requested. That is either a Windows user or a SQL user.

—- Enjoy updating!

vCenter Collectors – Syslog and Dump collector (with some PowerCLI)

With stateless VMware Auto deploy hosts, or hosts with SD or USB Stick installation, these hosts typically do not have local storage attached. There will be no space assigned to the scratch partition for dumps and logs. Defaulted they will be stored in RAM (/tmp/scratch). After a reboot the /tmp will be cleared.

We need to redirect the scratch partition to a shared VMFS and a central log / dump location. Scratch redirection is performed with the following steps:

  • Decide on a shared data store where preferably all hosts in a physical location connect, beware of physical location Bias. The volume is like /vmfs/volumes/UUID
  • Create a locker directory on that data store, for example .locker-Hostname
  • Connect to the ESXi host with Connect-ViServer -Server <Hostname>
  • Set the scratch partition with Set-VMHostAdvancedConfiguration -Name “ScratchConfig.ConfiguredScratchLocation” -Value “/vmfs/volumes/UUID/.locker-Hostname”
  • Maintenance mode (Set-VMHost -VMHost <host> -State ‘Maintenance’) and reboot (Restart-VMHost <host>).

With central management of an environment, a central log/dump location is recommended (also for stateful deployments this should be a standard). For a central log we need some services added to the vCenter:

– dump collector; collecting ESXi dumps of the infamous PSOD.

– syslog collector; collecting the host logs or syslogs.

On Windows we use the vCenter installer to install both services. With the vCenter Server Appliance (VCSA) these service are bundled with the appliance.

image

Dump Collector installation


Installation of the dump collector is straightforward. Determine in advance where your dumps will be saved, how much space (2GB default is sufficient for most environments) and to do a standalone or integrate with vCenter (last option is preferred for single management).

image

(if you got a data disk, change the default C: parameters)

image

And fill in the vCenter and account details on the next screen. The account will be used to register the extension.

Syslog Collector installation

Just like the dump collector the installation is straightforward. Determine the syslog rotation and log location. And start the installation to the VMware vCenter Server.

image

(if you got a data disk change the default C:).

image

Change the ports when needed for your environment, standard default is sufficient for most environments.

And finish the installation.

If you still use the vSphere client, then install both service add-ins.

Configuring the hosts

The ESXi hosts need to be configured to use the remote syslog and network dump location. You can use ESXi CLI and the advanced system settings or you can use PowerCLI. If we use PowerCLI for syslog we can use the Set-VMHostAdvancedConfiguration cmdlet and for the dump collector we need the Get-EsxCli cmdlet to use esxcli.

When we do this for a list of hosts we can use the following script. Just replace the vcenter servername at $vCenterFQDN and vCenter IP at the $esxcli.system.coredump.network.set line.

#set vCenter IP/FQDN

$vCenterFQDN = “<vcenter server>”

#connect to vCenter

$viServer = Connect-VIServer -Server $vCenterFQDN -Credential (Get-Credential)

#get all hosts in vCenter so we can cycle thru them

$Hosts = Get-VMHost

#cycle through each host

ForEach ($VMHost in $Hosts)

{

   #Add Syslog to Syslog Collector

   Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.global.logHost -Value $vCenterFQDN

    # Add DUmp to collector

    $esxcli = Get-EsxCli -vmhost $VMHost

    $esxcli.system.coredump.network.set($null, “vmk0”, “<vcenter ip>”, “6500”)

    $esxcli.system.coredump.network.set($true)

    #write out host name that was changed:

    write “Updated host: $VMHost”

}

#disconnect from ESX server

Disconnect-VIServer -Server $viServer -Confirm:$false

Checking the configuration

A host can be checked by going to the dump directories and checking the contents, just like the following for a host’s syslog directory.

image

I don’t have a crashed ESXi host at my disposal, so it is easier to check on the hosts via esxcli:

esxcli system coredump network get.

The IP and port should be in the output at:

Network Server IP:

Network Server Port: 6500