This week I had to work on a PoC to deploy OpenShift in Virtual Machines instead of bare-metal, like we did recently for the CentOS CI infra

Why in Virtual Machines (KVM guests) and not on bare-metal ? Well, there are cases where you have powerful/beefy machines, but not enough to meet the minimum number of nodes (at least 3 etcd nodes, and not even counting the real workers, at least 2 so 5 in total for bare minimum), while these nodes would perfectly (both at cpu/memory and storage) support the whole infra (assuming that you don't deploy all etcd/control planes nodes on the same physical node of course, and same for workers)

If you have a look at the official openshift documentation, you'll see that while all major cloud providers (AWS, Azure, GCP) are listed, there are also ways to deploy on bare-metal (what we did for CI infra), but also on RHEV, vSphere and Openstack too .. but nothing for plain KVM hypervisors (managed by libvirt in our cases).

But a VM is more or less like a bare-metal install, so what about we treat the VMs as bare-metal ? problem solved, right ? For our bare-metal deployment, we just used Ansible and with a simple ad-hoc playbook, so nothing fancy : just creating pxe boot entries, using ipmi to remotely power on the nodes and ensure they'd boot on network, RHCOS is installed and has all the kernel parameters for network settings and where to find RHCOS image to install, and where to find ignition files

So reusing that was my first idea, as we can easily create a VM with a fixed mac-address, and boot from the network. But then I thought about what we already use for our traditional KVM deploy : a simple ad-hoc playbook just templating a virt-install command that is kicked on the hypervisor.

If you have used virt-install yourself, you know that there is the --location parameter (that we used already). Extracted from man virt-install :

      -l, --location OPTIONS
           Distribution tree installation source. virt-install can recognize certain distribution trees and fetches a
           bootable kernel/initrd pair to launch the install.

How does that work ? Well, virt-install grabs kernel and initrd from that location, but to know where to find it (name/path), it uses a .treeinfo file. Example of http://mirror.centos.org/centos/7/os/x86_64/.treeinfo : 

[general]
name = CentOS-7
family = CentOS
timestamp = 1587405659.3
variant = 
version = 7
packagedir = 
arch = x86_64

[stage2]
mainimage = LiveOS/squashfs.img

[images-x86_64]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img
boot.iso = images/boot.iso

[images-xen]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img

So let's combine this option with the Red Hat CoreOS tree that we'll generate on our httpd deployment server : such .treeinfo doesn't exist, but let's just template it . From that point, it's easy, let's just use a variant for ad-hoc playbook that will :

  • Download kernel/initrd.img and deployer image for openshift to our local httpd server
  • Ensure we'll have correct .treeinfo file in place
  • Create a virt-install wrapper that will just point to correct path with --location, and deploy VMs with RHCOS and automatically calling ignition

While I admit that I'm surely not the most experienced openshift admin (just started to play with it), I still like the fact that RHCOS is still more or less linux that we use to know, and so combinining tools we already use allow us to deploy it , but surely not the way it's officially documented :)