Using xhyve for development on macOS
2020-06-30
My work computer is a MacBook Pro but the product we make at work is Linux software. When this distinction matters I reach for a virtual machine (VM). I got in the habit of working with Linux VM's on macOS years ago and it stuck with me. Never learned to like Docker for this (and Docker containers are not the same as VM's anyway).
For a long time I used VMWare Fusion which works really well. At one point my work laptop was a bit underpowered though so I tried to use my home server, which runs FreeBSD, for Linux VM's. That led me to learn about bhyve, the FreeBSD equivalent of KVM/QEMU. It is a very low-level tool so I ended up writing a script that automates the tasks I learned how to do with it. People writing their own automation around bhyve appears to be a common thing.
Fast forward to now, where I have a faster work laptop, and it was time to get VM's on my laptop again. I was reminded of the existence of xhyve, which is a macOS port of bhyve. Lucky me, I could apply what I learned with bhyve.
What working with xhyve looks like
At the bare minimum, to run an xhyve VM, you need three things:
- virtual hardware configuration
- a Linux kernel
- a Linux ramdisk
The virtual hardware configuration is a set of command line flags
for the xhyve command that specify the number of CPU's, amount
of memory, attached virtual PCI devices etc. This sounds imposing
but you can just copy examples from the repo. The Linux kernel is
a file usually called vmlinuz. When Linux boots it wants to have
a filesystem tree and start running a userspace process. These
things are contained in the "ramdisk", which is a read-only in-memory
filesystem with essential system utilities on it.
Normally, software on the ramdisk will then look for your actual hard disk, mount that, and continue the startup process from there. But this is not strictly necessary. There is a cute example in the xhyve repository that lets you boot into Linux using just a kernel and a ramdisk that have been committed into the repository.
One of the amenities you typically provide in the virtual hardware
configuration is a virtual serial port. This is enough to interact
with Linux; with the right boot options (injected via the xhyve
command line) Linux will let you interact with it via that virtual
serial port. From the user perspective, you are just typing into
the terminal where you launched xhyve.
Xhyve makes it easy to attach a virtual PCI network card to your VM. This allows you to connect to your VM via the local network, and it allows your Linux VM to connect to the internet. It all feels an awful lot like a cloud VM and that is no coincidence because this is more or less the technology that underpins the cloud.
This is all very nice, but how do we get that kernel and ramdisk to begin with? This is one of the main awkward points of xhyve I have learned about so far. Normally, your PC has a BIOS. The BIOS knows how to read a hard drive and can start up a boot loader on the hard drive. The boot loader then knows where your Linux kernel and ramdisk are, and loads those into memory.
No such luck with xhyve. There is no BIOS. You have to solve a chicken-and-egg problem to get the kernel+ramdisk off of the disk you are trying to boot from onto your Mac. Both when you are installing Linux on the VM, and everytime you update software on the VM (because software updates tend to refresh the ramdisk).
Unless you know how to solve such problems and have the patience for them, you probably don't want to deal with xhyve.
So far I am able to solve these problems and I have not run out of patience yet. We'll see how long that lasts. In the mean time it's rewarding to have VM's that I can manage using the command line and open source software.