Kubernetes vs Virtual Machines
Cloud native is widely regarded as the framework for building applications that are future-ready, and Kubernetes is one of the most critical components of this framework. As an open-source system, Kubernetes enables developers to automate the deployment, scaling, and management of containerized applications.
Application containers are a single or group of processes that are isolated but share the same kernel, they are designed to run a single service.
Virtual Machines (VMs), unlike containers, run on software called the hypervisor. VMs start a whole new operating system with a wide set of responsibilities, on which you can run multiple services and tasks.
What is KubeVirt?
In 2017, two years after the Kubernetes 1.0 release, Red Hat published the first alpha version of KubeVirt. KubeVirt is an addon that enables Virtual Machine workload on Kubernetes.
KubeVirt addresses the needs of development teams that want to or have adopted Kubernetes but have virtual machine workloads that cannot easily be containerized. One of the biggest advantages of KubeVirt is the ability to run virtual machines and containers side by side.
At Kubermatic we knew this is a great opportunity to change the way the community provision multiple kubernetes clusters on-premise.
Current Situation
Nowadays, more and more organizations with a decent number of bare-metal machines also want to switch to Kubernetes. By leveraging Kubernetes, companies can take advantage of its robust capabilities in automating deployment, scaling, and management of containerized applications.
Most of the time these organizations run kubernetes on virtual machines created on bare-metal servers. This requires installing and managing additional platforms like Proxmox, OpenStack, OpenNebula, etc. to handle the lifecycle of the virtual machine.
KubeVirt combined with Kubermatic eliminates the need for running dedicated platforms to manage virtual machines on-premise and allows the automatic deployment of Kubernetes clusters on them.
We created a dedicated cloud provider called KubeVirt cloud provider which reached GA status with KKP 2.22 and makes all of the above possible!
All you need to do is install Kubernetes directly on bare-metal and let Kubermatic create virtual machines where other kubernetes clusters can be created in a fully automated way.
The result is that you can manage virtual machines with just a few clicks and leverage full automation on a single platform in on-premise environments.
This simple picture should give you a good idea of our KubeVirt cloud provider architecture, where virtual machines are used as other k8s nodes.
The Cloud Provider Essentials
At Kubermatic, we put a lot of effort into making KubeVirt a feature-complete cloud provider.
First, we had to refresh and re-implement some parts of KubeVirt Cloud Controller Manager that positioned us to be one of the maintainers.
The project provides a set of functionalities that are crucial for creating a functional Kubernetes cluster on top of KubeVirt virtual machines (called a tenant or user cluster). These are:
- Initializing virtual machines nodes objects
- Exposing load balancer services outside of user clusters
- Passing cluster metadata
Another challenge was bringing storage to user clusters. Initially, our idea was to mount storage directly to nodes over the network. But, this would bring isolation issues and could have a performance impact on the cluster.
We decided to integrate with the KubeVirt CSI driver project, which allows the use of hot-pluggable volumes. We can safely deliver storage from the infrastructure cluster (a cluster where KubeVirt is installed) to user clusters. Also, we developed an operator that will deploy the driver and storage classes that can be safely used from tenant clusters.
Putting the pieces together
Today, customers can easily create a user cluster with just a few clicks. Compared to our previous technology preview versions, we reworked many important functionalities that had to graduate to GA. Let’s go through some of them point by point.
Templating
Kubermatic offers a few standard virtual machine templates that are tested by us. In addition, we are constantly working to deliver new ones that will cover more specific use cases in order to meet all of our customers’ needs. However, customers have the freedom to also bring their own templates at will, at any moment of the cluster lifecycle.
So how does all of this work under the hood?
Initially, we relied on KubeVirt Virtual Machine Presets, however, it had some limitations, and the decision to deprecate it convinced us to move to a new Instance Type and Preference API, even though it is still in alpha.
Over InstanceType, you can set:
Once all templates have been initialized in the cluster, users can easily select a specific virtual machine type to work with via our dashboard.
It is also very easy to check details about the type by clicking the View button.
Scheduling
Virtual machines are distributed across all nodes by default; thanks to topology spread constraints and Node Affinity that have been integrated with KubeVirt. Each machine we create over Machine Controller has a rule:
maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
This way, we make sure that user cluster nodes are spread across bare-metal nodes.
However, sometimes it is necessary to have strict control over node placement in an infrastructure cluster. For instance, users’ requirements might be to disable the scheduling of user cluster nodes on database bare-metal nodes, etc. It can be easily achieved with just a few clicks in the KKP dashboard. Users can easily add their own scheduling restrictions by setting new topology spread constraints combined with node affinity presets.
Networking and Storage
Currently, our way of providing a network to user clusters is by running nested CNIs.
This is a standard k8s in k8s scenario. KubeVirt’s infrastructure cluster runs its own CNI and a new user cluster on top does exactly the same. The below illustration shows the concept:
Our tests showed decent performance and stability of this architecture.
Although, in the meantime, we are working on a solution that will allow user cluster connections to go directly over networks attached to bare-metal nodes. It will be up to customers to decide which solution fits their needs.
As described in the previous section, we explored a few options for user cluster storage.
We decided to use hot-pluggable volumes over the KubeVirt CSI driver as our standard approach. During a virtual machine lifecycle, we can safely add or remove VM disks at our will without exposing storage directly to users.
We can summarize the steps involved in providing storage as follows:
- An administrator brings any storage solution that works with k8s to the KubeVirt infrastructure cluster.
- The administrator has control over which infra storage classes can be used within a user cluster.
- Our k8s operator installs the KubeVirt CSI Driver on the user cluster and creates appropriate storage classes based on the previous step.
- Users can easily mount storage to their pods by using the storage classes.
Bare-metal and user cluster node maintenance
In our previous versions of KubeVirt cloud provider that were in the technology preview stage, we had a problem that could cause user cluster outage. Infrastructure cluster administrators could not safely drain bare-metal nodes because evicting virtual machines (user cluster nodes) led to their instant termination.
The picture illustrates the problem very well:
To call KubeVirt integration a fully functional cloud provider we had to deal with this problem.
Our Machine Controller and KKP controllers have been adjusted to handle this kind of situation properly, they watch for virtual machines and read their status to safely trigger eviction mechanisms for user cluster nodes. It was important for us that customers don’t have to take any extra steps for workload eviction and do not have to go beyond k8s’ instruction.
Conclusion
After all the hard work that has been done, today we can say that it’s just a few clicks to create and scale Kubernetes clusters created on top of KubeVirt. It is as easy as ever before. The biggest advantage is that customers can have only one solution from one place to handle hundreds of clusters and virtual machines. We encourage you to try it with our free community edition!
