abstract: MirageOS solo5 multiboot native on bhyve
---
There were some busy times, several pull requests are still waiting to get merged (e.g. some cosmetics in [mirage](https://github.com/mirage/mirage/pull/544) as preconditions for treemaps and dependency diagrams), I [proposed](https://github.com/mirage/mirage/pull/547) to use `sleep_ns : int64 -> unit io` instead of the `sleep : float -> unit io` (nobody wants floating point numbers); also an RFC for [random](https://github.com/mirage/mirage/pull/551), Matt Gray [proposed](https://github.com/mirage/mirage/pull/548) to get rid of `CLOCK` (and have a `PCLOCK` and a `MCLOCK` instead). Soon there will be a major MirageOS release which breaks all the previous unikernels! :)
## What?
As described [earlier](https://hannes.nqsb.io/Posts/OperatingSystem), MirageOS is a library operating system developed in [OCaml](https://hannes.nqsb.io/Posts/OCaml). The code size is already pretty small, deployments are so far either as a UNIX binary or as a Xen virtual machine.
Xen is a hypervisor, providing concrete device drivers for the actual hardware of a physical machine, memory management, scheduling, etc. The initial release of Xen was done in 2003, since then the code size and code complexity of Xen is growing. It also has various different mechanisms for virtualisation, hardware assisted ones or purely software based ones, some where the guest operating system needs to cooperate others where it does not need to cooperate.
Since 2005, Intel CPUs (as well as AMD CPUs) provide hardware assistance for virtualisation (the VT-x extension), since 2008 extended page tables (EPT) are around which allow a guest to safely access the MMU. Those features gave rise to much smaller hypervisors, such as KVM (mainly Linux), [bhyve](http://bhyve.org) (FreeBSD), [xhyve](http://xhyve.org) (MacOSX), [vmm](http://undeadly.org/cgi?action=article&sid=20150831183826) (OpenBSD), which do not need to emulate the MMU and other things in software. The boot sequence in those hypervisors uses kexec or multiboot, instead of doing all the 16 bit, 32 bit, 64 bit mode changes manually.
MirageOS initially targeted only Xen, in 2015 there was a port to use rumpkernel (a modularised NetBSD), and 2016 [solo5](https://github.com/djwillia/solo5) emerged where you can run MirageOS on. Solo5 comes in two shapes, either as `ukvm` on top of KVM, or as a multiboot image using `virtio` interfaces (block and network, plus a serial console). Solo5 is only ~1000 lines of code (plus dlmalloc), and ISC-licensed.
A recent [paper](https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_williams.pdf) describes the advantages of a tiny virtual machine monitor in detail, namely no more [venom](http://venom.crowdstrike.com/) like security issues since there is no legacy hardware emulated. Also, each virtual machine monitor can be customised to the unikernel running on top of it: if the unikernel does not need a block device, the monitor shouldn't contain code for it. The idea is to have one customised monitor for each unikernel.
While lots of people seem to like KVM and Linux, I still prefer FreeBSD, their jails, and nowadays bhyve. I finally found some time, thanks to various cleanups to the solo5 code base, to finally look into porting solo5 to FreeBSD/bhyve. It runs and can output to console.
## How?
These instructions are still slightly bumpy. If you've a FreeBSD with bhyve (I use FreeBSD-CURRENT), and OCaml and opam (>=1.2.2) installed, it is pretty straightforward to get solo5 running. First, I'd suggest to use a fresh opam switch in case you work on other OCaml projects: `opam switch -A solo5 4.03.0` (followed by ``eval `opam config env` `` to setup some environment variables).
You need some software from the ports (this should be cleaned up at some point), at the moment `devel/gmake`, `lang/gcc48`, `devel/binutils`, and `sysutils/grub2-bhyve`.
Some header files from the system need to be included in the search path (please tell if there is a better solution, the build uses `nostdinc` and then includes the `gcc48` include path for `stddef.h` and friends):
A bunch of opam pins are needed (using pins instead of the [solo5 repository](https://github.com/djwillia/opam-solo5/) since pins are local to compiler switches, whereas repositories are global, and I needed to modify some bits anyways):
An `opam install mirage mirage-logs solo5-kernel-virtio mirage-bootvar-solo5 mirage-console mirage-solo5` should provide you with a basic set of libraries.
Now you can get the [mirage-skeleton](https://github.com/mirage/mirage-skeleton) repository, and inside of `console`, run `mirage configure --no-opam --virtio` followed by `gmake`. There should be a resulting `mir-console.virtio`.
I have not yet played with network, TLS, and other things...
## Open issues
- There are several things to upstream, such as the clang changes and FreeBSD support for ocaml-freestanding and other libraries
- The `Makefile` generated by mirage should be in the intersection of BSD and GNU make
- I'm not happy to require `ld` from the ports (but the one in base does not produce sensible binaries with `-z max-page-size=0x1000` [related](https://github.com/djwillia/solo5/pull/56))
- I'm also not happy to copy around include files and depend on gcc just for `stddef.h`, is there any sensible way to have freestanding includes? Should `-nostdinc` be used? (It looks necessary on Linux).
- Via [twitter](https://twitter.com/bhyve_dev/status/748930600581492736), bhyve devs suggested to port ukvm to ubhyve. This is a great idea, to no longer depend on virtio, and get more speed. Any takers?
- Debugging via gdb should be doable somehow, bhyve has [some support for gdb](https://wiki.freebsd.org/bhyve/DebuggingWithGdb), but it is unclear to me what I need to do to enter the debugger (busy looping in the VM and a gdb remote to the port opened by bhyve does not work).
I managed to get solo5 to work with bhyve. I even use clang instead of gcc and don't need to link `libgcc.a`. :) It is great to see further development in hypervisors and virtual machine monitors. Especially thanks to [Martin Lucina](https://lucina.net) for getting things sorted.
