Introduction to the vSphere Clustering Service (vCLS)
The vSphere Clustering Service (vCLS) is a new capability that is introduced in the vSphere 7 Update 1 release. It’s first release provides the foundation to work towards creating a decoupled and distributed control plane for clustering services in vSphere.
The challenge being that cluster services, like the vSphere Distributed Resource Scheduler (DRS), depend on vCenter Server availability for its configuration and operation. And while there’s ways to increase availability for vCenter Server, think about vSphere High Availability (HA) and vCenter Server High Availability (VCHA), its dependency is not ideal. Also, when thinking about vCenter Server scalability in large on-prem and public clouds, we need a better solution to support clustering services. That’s why vCLS is introduced. In the first release, a subset of DRS capabilities is already using the new vCLS feature.
Basic Architecture
The basic architecture for the vCLS control plane consists of maximum 3 virtual machines (VM), also referred to as system or agent VMs which are placed on separate hosts in a cluster. These are lightweight agent VMs that form a cluster quorum. On smaller clusters with less than 3 hosts, the number of agent VMs is equal to the numbers of ESXi hosts. The agent VMs are managed by vSphere Cluster Services. Users are not expected to maintain the lifecycle or state for the agent VMs, they should not be treated like the typical workload VMs.
Cluster Service Health
The agent VMs that form the cluster quorum state, are self correcting. This means that when the agent VMs are not available, vCLS will try to instantiate or power-on the VMs automatically.
There are 3 health states for the cluster services:
- Healthy – The vCLS health is green when at least 1 agent VM is running in the cluster. To maintain agent VM availability, there’s a cluster quorum of 3 agent VMs deployed.
- Degraded – This is a transient state when at least 1 of the agent VMs is not available but DRS has not skipped it’s logic due to the unavailability of agent VMs. The cluster could be in this state when either vCLS VMs are being re-deployed or getting powered-on after some impact to the running VMs.
- Unhealthy – A vCLS unhealthy state happens when a next run of the DRS logic (workload placement or balancing operation) skips due to the vCLS control-plane not being available (at least 1 agent VM).
Agent VM Resources
The vCLS agent VMs are lightweight, meaning that resource consumption is kept to a minimum. vCLS automatically creates a max of 3 agent VMs per cluster in an existing deployment when vCenter Server is upgraded to vSphere 7 update 1. In a greenfield scenario, they are created when ESXi hosts are added to a new cluster. If no shared storage is available, the agent VMs are placed on local storage. If a cluster is formed before shared storage is configured on the ESXi hosts, as would be the case when using vSAN, it is strongly recommended to move the vCLS agent VMs to shared storage after.
The agent VMs run a customized Photon OS. The resource specification per agent VM is listed in the following table:
The 2 GB virtual disk is thin provisioned. Also, there’s no networking involved, so there’s no network adapter configured. The agent VMs are not shown in the Hosts and Clusters overview in the vSphere Client. The VMs and Templates view now contains a new folder, vCLS, that contains all vCLS agent VMs. With multiple clusters, all vCLS agent VMs will show, numbered consecutively.
UI Overview
The vSphere Client contains messages and notes to show information about the vCLS agent VMs, also stating that the power state and resources of these VMs is handled by the vCLS.
Operations
As stated before, the agent VMs are maintained by vCLS. There’s no need for VI admins to power-off the VMs. In fact, the vSphere Client shows a warning when a agent VM is powered-off.
When a host is placed into maintenance mode, the vCLS agent VMs are migrated to other hosts within the cluster like regular VMs. Customers should refrain from removing, or renaming the agent VMs or its folder in order to keep the cluster services healthy.
The lifecycle for vCLS agent VMs is maintained by the vSphere ESX Agent Manager (EAM). The Agent Manager creates the VMs automatically, or re-creates/powers-on the VMs when users try to power-off or delete the VMs. Is the example below, you’ll see a power-off and a delete operation. Both from which the EAM recovers the agent VM automatically.
Automation and vCLS
For customer using scripts to automate tasks, it’s important to build in awareness to ignore the agent VMs in, for example clean-up scripts to delete stale VMs. Identifying the vCLS agent VMs is quickly done in the vSphere Client where the agent VMs are listed in the vCLS folder. Also, examining the VMs tab under Administration > vCenter Server Extensions > vSphere ESX Agent Manager lists the agent VMs from all clusters managed by that vCenter Server instance.
Every agent VM has additional properties so they can be ignored with specific automated tasks. These properties can also be found using the Managed Object Browser (MOB). The specific properties include:
vSphere Clustering Service (vCLS)
vSphere Clustering Service (vCLS) is a new capability that is introduced in the vSphere 7 Update 1 release. It’s first release provides the foundation to work towards creating a decoupled and distributed control plane for clustering services in vSphere.
The basic architecture for the vCLS control plane consists of maximum 3 virtual machines (VM), also referred to as system or agent VMs which are placed on separate hosts in a cluster. These are lightweight agent VMs that form a cluster quorum. On smaller clusters with less than 3 hosts, the number of agent VMs is equal to the numbers of ESXi hosts. The agent VMs are managed by vSphere Cluster Services. Users are not expected to maintain the life-cycle or state for the agent VMs, they should not be treated like the typical workload VMs.
Cluster Service Health
The agent VMs that form the cluster quorum state, are self correcting. This means that when the agent VMs are not available, vCLS will try to instantiate or power-on the VMs automatically.
There are 3 health states for the cluster services:
- Healthy – The vCLS health is green when at least 1 agent VM is running in the cluster. To maintain agent VM availability, there’s a cluster quorum of 3 agent VMs deployed.
- Degraded – This is a transient state when at least 1 of the agent VMs is not available but DRS has not skipped it’s logic due to the unavailability of agent VMs. The cluster could be in this state when either vCLS VMs are being re-deployed or getting powered-on after some impact to the running VMs.
- Unhealthy – A vCLS unhealthy state happens when a next run of the DRS logic (workload placement or balancing operation) skips due to the vCLS control-plane not being available (at least 1 agent VM).
vSphere client and click the view where you can see all the VMs, you’ll find there is a new folder created called vCLS that contains the vCLS VMs. You should not rename the vCLS folder or rename the vCLS VM(s).
Automation and vCLS
For customer using scripts to automate tasks, it’s important to build in awareness to ignore the agent VMs in, for example clean-up scripts to delete stale VMs. Identifying the vCLS agent VMs is quickly done in the vSphere Client where the agent VMs are listed in the vCLS folder. Also, examining the VMs tab under Administration > vCenter Server Extensions > vSphere ESX Agent Manager lists the agent VMs from all clusters managed by that vCenter Server instance.
Every agent VM has additional properties so they can be ignored with specific automated tasks. These properties can also be found using the Managed Object Browser (MOB). The specific properties include:
ManagedByInfo
extensionKey == “com.vmware.vim.eam”
type == “cluster-agent”
ExtraConfig keys
“eam.agent.ovfPackageUrl”
“eam.agent.agencyMoId”
“eam.agent.agentMoId”
vCLS Agent VMs have an additional data property key “HDCS.agent” set to “true”. This property is automatically pushed down to the ESXi host along with the other VM ExtraConfig properties explicitly.
VMware vSphere Cluster Service, which is responsible for maintaining DRS operations in the event of vCenter Server unavailability. There will be more services added to future releases. I imagine that vSphere would be capable of managing not only vSphere services, but probably also some networking services, storage, or application services.
VMware vSphere Cluster Services (vCLS) considerations, questions and answers.
In the vSphere 7.0 Update 1 release VMware introduced a new service called the VMware vSphere Cluster Services (vCLS). vCLS provides a mechanism that allows VMware to decouple both vSphere DRS and vSphere HA from vCenter Server. Niels Hagoort wrote a lengthy article on this topic here. You may wonder why VMware introduces this, well as Niels states. by decoupling the clustering services (DRS and HA) from vCenter Server via vCLS we ensure the availability of critical services even when vCenter Server is impacted by a failure.
vCLS is a collection of multiple VMs which, over time, will be the backbone for all clustering services. In the 7.0 U1 release a subset of DRS functionality is enabled through vCLS. Over the past week(s) I have seen many questions coming in and I wanted to create a blog with answers to these questions. When new questions or considerations come up, I will add these to the list below.
- Do I need to maintain/update/manage the vCLS VMs?
No, the VMs are managed by the ESX Agent Manager, you as an administrator should not need to manage these! - How many vCLS VMs will there be running?
At most, there will be 3 VMs running. If you have 1 host you will have 1 VM, with 2 hosts 2 VMs and with 3 or more hosts you will have 3 VMs. - Why do I see vCLS VMs with a number higher than 3? (for example “vCLS (5)” or “vCLS (28)”)
During maintenance ESX Agent Manager (EAM) can delete and re-provision the VMs when deemed needed. When a new VM is provisioned the counter will go up. - What is the resource overhead of the vCLS VMs?
Each VM has 1 vCPU, 128MB memory, 2GB thin disk, no NIC - If the vCLS has no NIC how does it communicate?
vCLS leverages a VMCI/vSOCKET interface to communicate with the hypervisor. - On which datastore will the vCLS VMs be provisioned?
It will be provisioned on shared storage, if available during provisioning, otherwise, it will go to local VMFS. - Can I specify which datastores should be considered for vCLS VMs?
Yes, starting with vSphere 7.0 U3 you can pick the datastore the vCLS VMs need to be provisioned to. More details in this blog. - If one, or more, vCLS VMs are provisioned on the wrong datastore, can I SvMotion it?
Yes, you are allowed to SvMotion the vCLS VMs to a datastore of choice, this should preferably be a datastore which is presented to all hosts in the cluster! - Can I use Storage DRS (SDRS) to place datastore into maintenance mode which holds vCLS VMs?
SDRS does not consider vCLS VMs for migration, vCLS VMs right now need to be manually migrated, even when using SDRS. - Why do I have multiple vCLS VMs running on the same host, and can I vMotion them?
After maintenance mode you can end up in a situation where multiple (or all) vCLS VMs are running on the same host, you will need to manually vMotion the vCLS VMs to different hosts in your cluster. The development team is aware of this issue, and is aiming to fix this in an upcoming release. - Why aren’t the vCLS VMs deleted when I disable DRS?
The vCLS VMs are linked to the “cluster object” and not directly to the DRS functionality. Disabling DRS doesn’t impact the vCLS VMs. - Why don’t I see the vCLS VMs in the vSphere Client?
Only members of the Administrators Group will see the vCLS VMs in the UI! - Can I disable the provisioning of the vCLS VMs?
Yes you can, but keep in mind that vCLS is required for DRS to function. Meaning that if you disable the provisioning of the VMs DRS will not work any longer, which also means that HA can’t leverage DRS for failover placement and will need to resort to the simple placement mechanism. - I can’t find the advanced setting mentioned in the documentation, what is the setting for disabling vCLS?
Go to your vCenter Server object, go to the configure tab, then go to “Advanced Settings”, add the key “config.vcls.clusters.domain-c<identifier>.enabled” and set it to false. The domain “c-number” for your cluster can be found in the URL when you click on the cluster in the HTML-5 interface. It should look something like the following, where the bold part is the important bit: https://vcsa-06.rainpole.com/ui/app/cluster;nav=h/urn:vmomi:ClusterComputeResource:domain-c22:4df0badc-1655-40de-9181-3422d6c36a3e/summary. - When I enable “retreat mode” and the vCLS VMs are deleted, will that also delete my resource pools?
No, resource pools are not deleted when vCLS is disabled. Only DRS load balancing is impacted! - Can I use the API/PowerCLI to enable Retreat Mode?
Yes, an example of this can be found in this VMware Community Thread. - Should I create anti-affinity rules for the vCLS VMs?
No, starting with vSphere 7.0 Update 2, the vCLS VMs have anti-affinity defined within the system! - Can I create anti-affinity rules for my vCLS VMs when I want to always separate them from certain workloads? (SAP HANA for instance)
Starting with 7.0 U3 this is possible when using the new compute policies. How you configure this you can find in this blog post. - Can I create a custom naming scheme for the vCLS VMs?
Today it is not possible to create a custom naming scheme vCLS VM, this has been filed as a feature request and is considered for a future release. We also discourage renaming the VMs at this point in time. Starting with 7.0 U3 though, the VMs do have a unique identifier in the name, which prevents some issues customers had with reporting and automation. - Can I rename the folder in which the vCLS VMs are stored in the VMs & Templated view?
No, we discourage changing the name of the folder as this could lead to issues when EAM needs to delete VMs. - Can I log in to the vCLS VMs?
Yes, but this is only intended for troubleshooting. This should not be needed during normal operations as these VMs are managed by EAM. You need to SSH into vCenter Server and then run the following command to retrieve the password. This will then allow you to login with “root” into the vCLS VMs, although I personally have not found a reason to do so.
/usr/lib/vmware-wcp/decrypt_clustervm_pw.py - Does vCLS require Kubernetes for vSphere to be configured?
No, it does not. - Do I need to back up or replicate the vCLS VMs?
No, there’s no need to create a backup or replicate the VMs. If the VMs are impacted by an outage than HA will restart them or EAM will recreate them automatically. - Does vCLS work with ESXi on ARM?
No, in the current release vCLS VMs cannot be provisioned to a cluster using ESXi on ARM. You can disable the creation of the VMs following this article. - If I need to power off my cluster, what do I do with these VMs?
These VMs are migrated by DRS to the next host until the last host needs to go into maintenance mode and then they are automatically powered off by EAM. - Which data evacuation option should I use when going into maintenance mode cluster-wide?
I use the “no data migration” option after I have powered off all normal VMs. - Is vCLS compatible with Site Recovery Manager (SRM)?
SRM is supported with vCLS starting vCenter Server 7.0 U1a. - Can my vCenter Server instance be 7.0 U1 while my hosts are vSphere 6.5 or 6.7?
Yes, this is fully supported. Just check the vCenter/ESXi compatibility matrix. Everything which is listed is also supported for DRS/vCLS - Do I need to do anything for a stretched cluster?
No that is not required, we would recommend however to ensure there’s at least 1 VM in each location from a compute perspective. - Can I identify these VMs as “special VMs” when automating certain tasks or creating reports through my automation tools?
Yes, you can easily identify them, Niels has listed the properties in this blog article at the bottom. - If one of the VMs fails, is there a warning?
Yes, you will see a warning triggered on the cluster level. This is the result of a new Skyline Health Check that was also introduced as part of 7.0 U1. I created a demo that shows that you can watch the demo here. - I have upgraded my vCenter Server and the vCLS VMs are not getting provisioned, what can I do?
We have seen some situations where an error is logged (eam.log) which states “Can’t provision VM for ClusterAgent” and “due to lack of suitable datastore” and “Couldn’t acquire token due to: Signature validation failed”. In this case, the customer had two linked vCenter Server instances.
If you see this error, reset the STS certificate and restart the vCenter STS service and see if the VMs are now being provisioned (Thanks to Mina Botros(GSS) for providing this tip!). More details on how to stop/start a service can be found here. Also look at this VMTN thread which details the steps required to fix this proble, - I am getting an error stating “insufficient resources” on the vCLS VM power on event, how can I fix this problem?
We are not sure what is causing this problem right now, but it can be fixed by changing the per-VM EVC to disabled. This blog describes how to do this. - I am getting an error stating the following “Feature ‘MWAIT’ was absent but must be present” when the vCLS VM is powered on.
Please enable the MONITOR/MWAIT flag in the bios of the hosts in the cluster, as documented here. - I am getting the following error, how do I solve it? ““Feature ‘bad_requirement.hv.capable’ was 0, but must be at least 1′. Failed to start the virtual machine. Module ‘FeatureCompatLate’ power on failed.”
You most likely have the setting vhv.enable = “TRUE” configured on each host in “/etc/vmware/config”. If you set it to false the vCLS VMs should start.
Related
Reader Interactions
Comments
“Can I customize the name of the vCLS VMs?
Today it is not possible to customize the name of the vCLS VM, this has been filed as a feature request and is considered for a future release”
In a lab environment, I was able to rename the vCLS VMs and DRS remained functional. Functionality also persisted after SvMotioning all vCLS VMs to another Datastore and after a complete shutdown/startup of the cluster.
Original vCLS VM names were vCLS (4), vCLS (5), vCLS (6).
New vCLs VM names are now vCLS (1), vCLS (2), vCLS (3).
Does this question, pertain to customizing the names of newly generated vCLS VMs, existing vCLS VMs, or both? Functionality “seems” to be working even though I’ve renamed the generated vCLS VMs in my lab.
Yes you can rename the vCenter entity, but if this VM is for whatever reason deleted by EAM a new VM with a new name with the “vCLS” naming scheme will be created again. Also, you are not supposed to do any ops on the VMs, so renaming is also discouraged. In other words, that it works doesn’t mean VMware will support it ��
but it is a valid point, I just updated my post and clarified that section
Also, any impact to rename the vCLS folder?
Yes there could be an impact, I just asked the engineering team and they (right now) advise against renaming the folder!
Another question. Does vCLS currently manage vSAN functions when VC is unavailable?
Hope it is in the roadmap
This sounds like steps towards a cluster vCenter, with real cluster FT like availability. Would not be surprised if Corfu was down there ��
Nice to have DRS, HA was already distributed, but given the number of solutions that use vCenter as an endpoint, I hope to see full vCenter cluster soon.
how to put all hosts in maintenance mode in a vSAN cluster running vCLS VMs (without having a secondary storage)? The question is about my lab env (but can be interesting also for a prod env).
I’ve 4 hosts with vSAN enabled and sometime i shutdown all the environment.
Before the upgrade to 7U1, to shutdown my env, I turn off all the VMs, put all hosts in maintenance (without data migration) and finally power off the ESXi. Now i’ve some problems to perform a clean shutdown because the vCLS are running on vSAN (seems a chicken egg problem or i missing something)…
In my lab (3 clusters) I simply place all hosts in maintenance mode (one by one) and power them all off, I use the “no data migration” option after I power off all my VMs. It takes a bit of time for the last host to go into maintenance mode, but so far it just works. You are saying that this doesn’t work for you?
Hi Duncan, I just force shutdown the vCLS VM’s and then jump into Maintenance Mode. Quick and dirty. When I bring my lab back up the vCLS VM’s start automatically. What is the impact of doing this, even though the direction from VMware is to never interact with the vCLS VM’s.
I have just tested it 3 times, I do “maintenance mode” from the vCenter UI using “No Data Migration” and it just powers off the VMs running on the vSAN Datastore when I get to the last host.
Hi Duncan, well my vCenter Appliance is on that vSAN Cluster, so I have to shut it down. Trying to enter Maintenance Mode on the ESXi Host interface doesn’t allow you to stop the vCLS VM’s, as it might through the vCenter GUI. Kind of caught be-twixt & between perhaps.
In that situation you should use the “retreat” mode I discussed in this post. It will disable/delete the vCLS VMs, which will allow you to place it in maintenance mode.
yes this doesn’t work for me. Today i changed some configuration on vSAN to have a clean shutdown….
My vSAN is configured with a default storage policy as Erasure Coding with FTT=1, all the VMs and vCLS VMs was configured to using it and with this configuration I’m not able to put the third host in maintenance (i wait about 45 minutes to put the host in maintenance without success). Due to unavailability of some vCLS VMs vSAN storage objects when the second host is put in maintenance, I’m able to power off the vCLS VMs without they can be restarted automatically and then i’m able to put in maintenance the 3rd and 4th Hosts.
Now i changed the vCLS VMs vSAN storage policy with a “simple” FTT=1 (2 mirrors & a witness component), whit this change i can have a clean ESXi maintenance/shutdown. The 3rd host is very slow to reach the maintenance (about 19 Minutes), but the 4th work as expected.
I don’t know if the problem can be related to my env but if not, can be useful have a “maintenance mode” for the vCLS VMs to be able to shutdown it.
One more question, the 3 vCLS VMs are deployed for every vSphere Cluster?
Are you doing the maintenance mode from the vCenter Server or from the command line? Which “data evac” option did you select?
Via vCenter (deployed on another ESXi outside the cluster) and “data evac” = “No Data Migration”
Is this a shift towards moving to microservices for vCenter or all vmware apps?
Mainly for clustering services right now, but I would expect decoupling to continue!
Doug McIntyre says
Why do I have four vCLS machines in my test lab instead of the max of 3 you state?
(vCLS (1), vCLS (2), vCLS (3), vCLS (4))
How many clusters do you have?
Doug McIntyre says
There’s two clusters, one management of one host, and one of compute with four hosts.
So you’re implying I’m picking up the forth for the single host management cluster? If I left the single management host out of a cluster, I’d drop to three?
Correct, drop the cluster and that 4th vcls VM will disappear.
Gilles Le Ridou says
We have several two nodes clusters (FC SAN, not VSAN). Some of them have two vCLS VMs, some others three vCLS. Some reason? (vCenter 7.0 u1, ESX still 6.7)
Single host cluster gets 1 VCLS VM. Two node gets two, three node or more gets 3.
Gilles Le Ridou says
I can post an image of a two nodes cluster with three vCLS VMs on Twitter if you think this is interesting.
I’ve seen it in my lab as well, it is usually an issue with a clean up. Just enable “retreat mode” (False) and disable it again (True). This should clean up the environment.
Greg Merideth says
Bit of a nitpick but why does skyline health require CEIP? Are the health services of clusters sent back to vmare?
This is used for the “Skyline” part of the Healthcheck, which helps the support team during troubleshooting when you file a support request.
Thank you for this article. Had missing vCLS VMs after upgrade to vCenter 7.0.1. Tried Vmware support, and they basically told me to either restore vCenter from backup / snapshot or wait for a patch with no current ETA… Those solutions turned out to be unacceptable, as DRS was out of action and losing current vCenter was not an option. Resetting the STS certificate + restart all services absolutely did the trick and now the VMs were deployed – DRS working. This saved me a lot of trouble as a VI admin and I really appreciate it. �� ��
Thanks for the feedback, great to hear it solved your problem!
I have a 2 host cluster and as expected 2 vCLS VMs. I started rebooting my hosts to check my HA setup (which worked fine) however more vCLS VMs are appearing and the old ones aren’t being deleted.
Should the oldest VMs be automatically deleted?
Is this a lab environment or production? If you enable “retreat mode” and disable it again, all VMs should be cleaned up and new VMs should be created. However, if this is production, it may be useful to create a support request and upload the logs so it can be debugged.
Christoph Hochsticher says
Come on. This is a Beta release where you couldn’t specify the Datastores. It costs me hours to change all vCLS VMs away from some ISO, Logging and Repo Datastores. That is shiddy!
I’m shocked that this in the final U1.
Martin Schenker says
We tried to upgrade a 6.7 system to 7.0U1, first stage VCenter upgrade, then three hosts w. VSan.
Two servers went OK, the vCLS VMs were migrated to the last 6.7 host. Enterning maintenance mode failed for that host, the vCLS VMs didn’t migrate off it or shut down.
After a manual shutdown of the three vCLS VMs, the third system finally entered maintenance mode and could be upgraded to 7.01. The VCenter and all hosts are now up-to-date w. current patches and software.
Now the vCLS VMs won’t start; we’re just getting the repetetive error (every 30s):
“Feature ‘bad_requirement.hv.capable’ was 0, but must be at least 1′. Failed to start the virtual machine. Module ‘FeatureCompatLate’ power on failed.
Only one vCLS is spawned and tries to come online, not three as expected.
The system was put into retreat mode (config.vcls.clusters.domain-cXY.enabled -> False) but leaving retreat mode does not fix the problem.
Any pointers?
If it has been enabled, makes sure to disable the following:
File on each host: /etc/vmware/config
vhv.enable = “TRUE”
Should be FALSE.
Does that mean that 7.0 U1 is not compatoble with virtual hardware assisted virtualizadion ?
Martin Schenker says
Wow! Instant success… thanks a lot!!
Just changed the config file on the three hosts, all three vCLS VMs powered on and the cluster is healthy again!
I have automated the deployment of load VM’s post ESXi installation. Some time load VM deployment/reconfiguration is failing with an error ‘Another task is already in progress’, then I looked into the tasks in VCSA, could see vCLS VMs deployment task at the same time. Looks like due to vCLS VM deployment task is active another re-configuration of load VM is failed.
Can you please suggest what is the best way to handle this situation?
I have no idea to be honest.
For our Citrix Clusters we want to use DRS setting of vCPU:Core 1:1 overcommit. Are these vCLS excluded from the calculations of DRS?
Never tested it, but it is a VM with vCPUs, and I would assume they are counted for as such.
Duncan, what happens to the vCLS VMs if the hosts were being decommissioned, and the hosts are placed in MM and moved out of the cluster, but still have a vCLS VM on them, what will happen when the host is removed from vCenter, will the vCLS VM still exist, or will they be removed entirely?
Thomas Bende says
I am trying to storage vmotion the vCLS VMs to new LUNs but am getting the following DRS warning and the svMotion fails:
Vew SDRS Faults > A specified parameter was not correct: StoragePlacementSpec.vm
If turned Full to manual – no joy
What Parameter is missing?
I have not seen this issue, I would recommend contacting support.
Renee E. Riley says
I have the same issue, if anyone found a solution.
I had the same issue. Disabling SDRS on the datastore cluster fixed it. I re-enabled SDRS on the datastore cluster after the vCLS VMs were moved.
Rob Steenbakkers says
Indeed if you Disable SDRS then you can storage migrate the vCLS to one off the datastores in that SDRS cluster and when finished you can enable SDRS again.
Can’t you disable SDRS for those VMs specifically? Or doesn’t that work?
Tom Borcher says
We also got the same “SDRS Fault” trying to Storage vMotion the vCLS VM’s. So we tried disabling SRDS, but it failed with this error: “A specified parameter was not correct: PodConfigSpec.SpaceLoadBalanceConfig.FreeSpaceThreshold” Through trial and error I determined that ANY edit to the SDRS settings resulted in the same error. Before I open a case with Support, does anyone have any ideas? Thanks.
try the following sequence:
– enable retreat mode (this deletes the vCLS VMs)
– disable SDRS
– disable retreat mode (creates the VMs again)
– SvMotion them to where you want them to be
– enable SDRS
PS: Engineering is working on fixing this for an upcoming release.
Tom Borcher says
We also got the same “SDRS Fault” trying to Storage vMotion the vCLS VM’s. So we tried disabling SRDS, but it failed with this error: “A specified parameter was not correct: PodConfigSpec.SpaceLoadBalanceConfig.FreeSpaceThreshold” Through trial and error I determined that ANY edit to the SDRS settings resulted in the same error. Before I open a case with Support, does anyone have any ideas? Thanks.
I have now the 2nd SR Ticket open beacuse of that vCLS “improvements”.
SR21244921408
Introduction to vSphere Clustering Service (vCLS)
The vSphere Clustering Service (vCLS) is a new capability that is introduced in the vSphere 7 Update 1 release. It’s first release provides the foundation to work towards creating a decoupled and distributed control plane for clustering services in vSphere.
vSphere Cluster Services (vCLS)
Starting with vSphere 7.0 Update 1, vSphere Cluster Services (vCLS) is enabled by default and runs in all vSphere clusters. vCLS ensures that if vCenter Server becomes unavailable, cluster services remain available to maintain the resources and health of the workloads that run in the clusters. vCenter Server is still required in 7.0 update 1 to run DRS and HA.
vCLS uses agent virtual machines to maintain cluster services health. The vCLS agent virtual machines (vCLS VMs) are created when you add hosts to clusters. Up to three vCLS VMs are required to run in each vSphere cluster, distributed within a cluster. vCLS is also enabled on clusters which contain only one or two hosts. In these clusters the number of vCLS VMs is one and two.
How to? Retrieving Password for vCLS VMs
You can retrieve the password to login to the vCLS VMs. To ensure cluster services health, avoid accessing the vCLS VMs. This document is intended for explicit diagnostics on vCLS VMs.
Procedure
- Use SSH to login to the vCenter Server Appliance.
- Run the following python script:
/usr/lib/vmware-wcp/decrypt_clustervm_pw.py - Read the output for the password.
pwd-script-output
Read key from file
Connected to PSQL
PWD: (password displayed here)
Operations that might disrupt the healthy functioning of vCLS VMs:
- Changing the power state of the vCLS VMs
- Resource reconfiguration of the vCLS VMs such as changing CPU, Memory, Disk size, Disk placement
- VM encryption
- Triggering vMotion of the vCLS VMs
- Changing the BIOS
- Removing the vCLS VMs from the inventory
- Deleting the vCLS VMs from disk
- Enabling FT of vCLS VMs
- Cloning vCLS VMs
- Configuring PMem
- Moving vCLS VM to a different folder
- Renaming the vCLS VMs
- Renaming the vCLS folders
- Enabling DRS rules and overrides on vCLS VMs
- Enabling HA admission control policy on vCLS VMs
- Enabling HA overrides on vCLS VMs
- Moving vCLS VMs to a resource pool
- Recovering vCLS VMs from a snapshot
Author: Daniel Micanek
Senior Service Architect, SAP Platform Services Team at TietoEVRY | SUSE SCA | vExpert ⭐⭐⭐⭐ | vExpert NSX | VCIX-DCV | VCAP-NV Design | VCAP-DCV Design+Deploy | VCP-DCV/NV/CMA | NCIE-DP | OCP | Azure Solutions Architect | Certified Kubernetes Administrator (CKA) View all posts by Daniel Micanek