Amongst the various improvements and fixes in Gru orchestration framework one of the highlights of version 0.5.0 of the project is the added support for managing VMware vSphere environments.

If you haven’t heard about Gru yet - Gru is a concurrent orchestration and configuration management framework written in Go and Lua as the DSL language, which allows you to manage your environment in an idempotent manner.

You can read more about the project at Github, where you can also find various code examples and use cases.

In this post we will see how we can use Gru for automating the management of VMware vSphere environment by using the recently introduced features in the project.

The resources we will talk about in this post are Datacenter, Cluster, ClusterHost, Host, DatastoreNfs and VirtualMachine.

In order to keep things simple and not repeat ourselves we will first create a Lua table, which will contain the credentials used to authenticate against the remote VMware vSphere API endpoint.

--
-- The credentials to the remote VMware vSphere API endpoint
--
credentials = {
   username = "root",
   password = "r00tp4ssw0rd",
   endpoint = "https://vc01.example.org/sdk",
   insecure = true, --> Needed if the vCenter is using a self-signed certificate
}

The very first thing that we are going to automate is the creation of the vSphere Datacenter. In order to do that we will use the Datacenter resource as shown in the code below.

--
-- Manage the VMware vSphere Datacenter
--
dc = vsphere.datacenter.new("MyDatacenter")
dc.username = credentials.username
dc.password = credentials.password
dc.endpoint = credentials.endpoint
dc.insecure = credentials.insecure
dc.state = "present"

catalog:add(dc)

The next thing we need to take care of in our VMware vSphere environment is the cluster. The code below takes care of creating a vSphere Cluster and also enables the DRS service for it.

--
-- Manage the VMware vSphere Cluster
--
cluster = vsphere.cluster.new("MyCluster")
cluster.endpoint = credentials.endpoint
cluster.username = credentials.username
cluster.password = credentials.password
cluster.insecure = credentials.insecure

cluster.state = "present"
cluster.path = "/MyDatacenter/host"
cluster.config = {
   enable_drs = true,
   drs_behavior = "fullyAutomated",
}
cluster.require = { dc:ID() } --> The cluster depends on the datacenter

catalog:add(cluster)

One thing to note in the code above is that the Cluster resource depends on the Datacenter resource we’ve created previously, which will ensure proper evaluation and processing of resources in the resulting resources DAG graph.

Once we have the cluster ready we can now add some compute resources to it by adding an ESXi host to the cluster.

The code below shows how to add an ESXi host a vSphere cluster.

--
-- Add an ESXi host to the VMware vSphere Cluster
--
host = vsphere.cluster_host.new("esxi01.example.org")
host.endpoint = credentials.endpoint
host.username = credentials.username
host.password = credentials.password
host.insecure = credentials.insecure

host.state = "present"
host.path = "/MyDatacenter/host/MyCluster"
host.esxi_username = "root"
host.esxi_password = "esxi_p4ssw0rd"
host.ssl_thumbprint = "ssl-thumbprint-of-host"
host.require = { cluster:ID() } --> The ESXi host depends on the cluster

catalog:add(host)

Make sure to provide the correct SSL thumbprint of your ESXi host. This post in virtualGhetto provides details on how to extract the SSL thumbprint of an ESXi host.

Gru also provides a resource for managing various configuration settings of ESXi hosts such as DNS and lockdown mode settings. For more information about the various settings that you can manage on ESXi hosts make sure to check the Host resource.

Another resource that we will use is the DatastoreNfs one, which allows you to manage NFS datastores on ESXi hosts.

The following example shows how to mount an NFS datastore on our ESXi host.

--
-- Mount an NFS datastore on our ESXi host
--
datastore = vsphere.datastore_nfs.new("vm-storage01")
datastore.endpoint = credentials.endpoint
datastore.username = credentials.username
datastore.password = credentials.password
datastore.insecure = credentials.insecure

datastore.state = "present"
datastore.path = "/MyDatacenter/datastore"
datastore.hosts = {
   "/MyDatacenter/host/MyCluster/esxi01.example.org",
}
datastore.nfs_server =  "nfs01.example.org"
datastore.nfs_path = "/storage/vm-storage01"
datastore.mode = "readWrite"
datastore.require = { host:ID() } --> The datastore depends on the ESXi host

catalog:add(datastore)

The last resource that we will see in this post is the VirtualMachine resource.

Using the Lua-based DSL of the project we will create a few Virtual Machines.

--
-- Manage VMware vSphere Virtual Machines
--
names = { "kevin", "bob", "stuart" } --> You know these guys, right?

for _, name in ipairs(names) do
   vm = vsphere.vm.new(name)
   vm.endpoint = credentials.endpoint
   vm.username = credentials.username
   vm.password = credentials.password
   vm.insecure = credentials.insecure

   vm.state = "present"
   vm.path = "/MyDatacenter/vm"
   vm.pool = "/MyDatacenter/host/MyCluster"
   vm.datastore = "/MyDatacenter/datastore/vm-storage01"
   vm.wait_for_ip = true
   vm.power_state = "poweredOn"
   vm.template_config = {
      use = "/MyDatacenter/vm/Templates/centos-7-x86-64-template",
   }
   vm.require = { host:ID(), datastore:ID() } --> The VM depends on the ESXi host and datastore

   catalog:add(vm)
end

Before we run the code we have written so far, let us first have a look at the resulting resource dependency graph.

In order to do that we will use the gructl graph command, e.g.

$ gructl graph site/code/vsphere.lua | dot -Tpng -O

The resulting DAG graph looks like this.

What is worth mentioning about Gru is that it is good in executing operations concurrently. Looking at the above dependency graph we can see that if all dependencies are satisfied by the time we reach the VirtualMachine resources then we could actually create our Virtual Machines concurrently.

Being able to execute operations like this in concurrent way can greatly reduce the time required for standing up a new environment.

Time to actually run the code we’ve written so far!

$ gructl apply site/code/vsphere.lua

The result of the above command looks like this.

$ gructl apply site/code/vsphere.lua
2016/12/12 13:59:55 Loaded 7 resources
2016/12/12 13:59:55 Starting 4 goroutines for concurrent processing
2016/12/12 13:59:55 datacenter[MyDatacenter@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 13:59:55 datacenter[MyDatacenter@https://vc01.example.org/sdk] creating datacenter in /
2016/12/12 13:59:55 cluster[MyCluster@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 13:59:55 cluster[MyCluster@https://vc01.example.org/sdk] creating cluster
2016/12/12 13:59:55 cluster[MyCluster@https://vc01.example.org/sdk] property 'cluster-config' is out of date
2016/12/12 13:59:55 cluster[MyCluster@https://vc01.example.org/sdk] setting cluster config
2016/12/12 13:59:55 cluster_host[esxi01.example.org@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 13:59:55 cluster_host[esxi01.example.org@https://vc01.example.org/sdk] adding host to /MyDatacenter/host/MyCluster
2016/12/12 14:00:16 datastore_nfs[vm-storage01@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 14:00:16 datastore_nfs[vm-storage01@https://vc01.example.org/sdk] mounting datastore on esxi01.example.org
2016/12/12 14:00:18 vm[stuart@https://vc01.example.org/sdk] is concurrent
2016/12/12 14:00:18 vm[bob@https://vc01.example.org/sdk] is concurrent
2016/12/12 14:00:18 vm[kevin@https://vc01.example.org/sdk] is concurrent
2016/12/12 14:00:18 vm[stuart@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 14:00:18 vm[bob@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 14:00:18 vm[kevin@https://vc01.example.org/sdk] is absent, should be present
2016/12/12 14:00:18 vm[stuart@https://vc01.example.org/sdk] cloning virtual machine from /MyDatacenter/vm/centos-7-x86-64-template
2016/12/12 14:00:18 vm[kevin@https://vc01.example.org/sdk] cloning virtual machine from /MyDatacenter/vm/centos-7-x86-64-template
2016/12/12 14:00:18 vm[bob@https://vc01.example.org/sdk] cloning virtual machine from /MyDatacenter/vm/centos-7-x86-64-template
2016/12/12 14:02:03 vm[bob@https://vc01.example.org/sdk] property 'power-state' is out of date
2016/12/12 14:02:03 vm[bob@https://vc01.example.org/sdk] setting power state to poweredOn
2016/12/12 14:02:03 vm[stuart@https://vc01.example.org/sdk] property 'power-state' is out of date
2016/12/12 14:02:03 vm[stuart@https://vc01.example.org/sdk] setting power state to poweredOn
2016/12/12 14:02:04 vm[kevin@https://vc01.example.org/sdk] property 'power-state' is out of date
2016/12/12 14:02:04 vm[kevin@https://vc01.example.org/sdk] setting power state to poweredOn
2016/12/12 14:02:06 vm[stuart@https://vc01.example.org/sdk] waiting for IP address
2016/12/12 14:02:06 vm[kevin@https://vc01.example.org/sdk] waiting for IP address
2016/12/12 14:02:06 vm[bob@https://vc01.example.org/sdk] waiting for IP address
2016/12/12 14:03:04 vm[bob@https://vc01.example.org/sdk] virtual machine IP address is 10.xxx.yyy.zzz
2016/12/12 14:03:04 vm[stuart@https://vc01.example.org/sdk] virtual machine IP address is 10.xxx.yyy.zzz
2016/12/12 14:03:04 vm[kevin@https://vc01.example.org/sdk] virtual machine IP address is 10.xxx.yyy.zzz
2016/12/12 14:03:04 0 up-to-date, 7 changed, 0 failed

What is interesting to note in the output above is that Gru is actually creating the Virtual Machines concurrently.

Since all dependencies have been satisfied by the time we have reached the Virtual Machine resources and the VirtualMachine resource type itself is concurrent this allows Gru to schedule such operations for concurrent execution.

This example shows one of the key features of the project - it’s ability to execute operations concurrently.

Hopefully this introduction gave you a nice overview on how to start managing your virtual infrastructure using Gru.

Make sure to also check the project at Github for more information and other code examples.