91 lines
6.5 KiB
Text
91 lines
6.5 KiB
Text
---
|
|
title: Minimising the virtual machine monitor
|
|
author: hannes
|
|
tags: future, mirageos, security
|
|
abstract: MirageOS solo5 multiboot native on bhyve
|
|
---
|
|
|
|
Update (2016-10-19): all has been merged upstream now!
|
|
Update (2016-10-30): `static_website_tls` works (TLS,HTTP,network via tap device)!
|
|
|
|
## What?
|
|
|
|
As described [earlier](https://hannes.nqsb.io/Posts/OperatingSystem), MirageOS is a library operating system developed in [OCaml](https://hannes.nqsb.io/Posts/OCaml). The code size is already pretty small, deployments are so far either as a UNIX binary or as a Xen virtual machine.
|
|
|
|
Xen is a hypervisor, providing concrete device drivers for the actual hardware of a physical machine, memory management, scheduling, etc. The initial release of Xen was done in 2003, since then the code size and code complexity of Xen is growing. It also has various different mechanisms for virtualisation, hardware assisted ones or purely software based ones, some where the guest operating system needs to cooperate others where it does not need to cooperate.
|
|
|
|
Since 2005, Intel CPUs (as well as AMD CPUs) provide hardware assistance for virtualisation (the VT-x extension), since 2008 extended page tables (EPT) are around which allow a guest to safely access the MMU. Those features gave rise to much smaller hypervisors, such as KVM (mainly Linux), [bhyve](http://bhyve.org) (FreeBSD), [xhyve](http://xhyve.org) (MacOSX), [vmm](http://undeadly.org/cgi?action=article&sid=20150831183826) (OpenBSD), which do not need to emulate the MMU and other things in software. The boot sequence in those hypervisors uses kexec or multiboot, instead of doing all the 16 bit, 32 bit, 64 bit mode changes manually.
|
|
|
|
MirageOS initially targeted only Xen, in 2015 there was a port to use rumpkernel (a modularised NetBSD), and 2016 [solo5](https://github.com/Solo5/solo5) emerged where you can run MirageOS on. Solo5 comes in two shapes, either as `ukvm` on top of KVM, or as a multiboot image using `virtio` interfaces (block and network, plus a serial console). Solo5 is only ~1000 lines of code (plus dlmalloc), and ISC-licensed.
|
|
|
|
A recent [paper](https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_williams.pdf) describes the advantages of a tiny virtual machine monitor in detail, namely no more [venom](http://venom.crowdstrike.com/) like security issues since there is no legacy hardware emulated. Also, each virtual machine monitor can be customised to the unikernel running on top of it: if the unikernel does not need a block device, the monitor shouldn't contain code for it. The idea is to have one customised monitor for each unikernel.
|
|
|
|
While lots of people seem to like KVM and Linux, I still prefer FreeBSD, their jails, and nowadays bhyve. I finally found some time, thanks to various cleanups to the solo5 code base, to finally look into porting solo5 to FreeBSD/bhyve. It runs and can output to console.
|
|
|
|
## How?
|
|
|
|
These instructions are still slightly bumpy. If you've a FreeBSD with bhyve (I use FreeBSD-CURRENT), and OCaml and opam (>=1.2.2) installed, it is pretty straightforward to get solo5 running. First, I'd suggest to use a fresh opam switch in case you work on other OCaml projects: `opam switch -A 4.03.0 solo5` (followed by ``eval `opam config env` `` to setup some environment variables).
|
|
|
|
You need some software from the ports: `devel/pkgconf`, `devel/gmake`, `devel/binutils`, and `sysutils/grub2-bhyve`.
|
|
|
|
Lots of libraries are not yet released, use the [`mirage-dev`](https://github.com/mirage/mirage-dev) repository:
|
|
```bash
|
|
opam repo add mirage-dev https://github.com/mirage/mirage-dev.git
|
|
```
|
|
|
|
An `opam install mirage mirage-logs solo5-kernel-virtio mirage-bootvar-solo5 mirage-console mirage-solo5` should provide you with a basic set of libraries.
|
|
|
|
Now you can get the [mirage-skeleton](https://github.com/mirage/mirage-skeleton) repository, and inside of `console`, run `mirage configure --no-opam --virtio` followed by `make`. There should be a resulting `mir-console.virtio`.
|
|
|
|
Once that is in place, start your VM:
|
|
```bash
|
|
sudo grub-bhyve -M 128M console
|
|
> multiboot (host)/home/hannes/mirage-skeleton/console/mir-console.virtio
|
|
> boot
|
|
|
|
sudo bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -l com1,stdio -m 128M console
|
|
```
|
|
|
|
The following output will appear on your controlling terminal:
|
|
|
|
```bash
|
|
| ___|
|
|
__| _ \ | _ \ __ \
|
|
\__ \ ( | | ( | ) |
|
|
____/\___/ _|\___/____/
|
|
multiboot: Using memory: 0x100000 - 0x8000000
|
|
TSC frequency estimate is 2593803000 Hz
|
|
Solo5: new bindings
|
|
STUB: getenv() called
|
|
hello
|
|
world
|
|
hello
|
|
world
|
|
hello
|
|
world
|
|
hello
|
|
world
|
|
solo5_app_main() returned with 0
|
|
Kernel done.
|
|
Goodbye!
|
|
```
|
|
|
|
Network and TLS stack works as well (tested 30th October).
|
|
|
|
## Open issues
|
|
|
|
- I'm not happy to require `ld` from the ports (but the one in base does not produce sensible binaries with `-z max-page-size=0x1000` [related](https://github.com/Solo5/solo5/pull/56))
|
|
- Via [twitter](https://twitter.com/bhyve_dev/status/748930600581492736), bhyve devs suggested to port ukvm to ubhyve. This is a great idea, to no longer depend on virtio, and get more speed. Any takers?
|
|
- Debugging via gdb should be doable somehow, bhyve has [some support for gdb](https://wiki.freebsd.org/bhyve/DebuggingWithGdb), but it is unclear to me what I need to do to enter the debugger (busy looping in the VM and a gdb remote to the port opened by bhyve does not work).
|
|
|
|
## Conclusion
|
|
|
|
I managed to get solo5 to work with bhyve. I even use clang instead of gcc and don't need to link `libgcc.a`. :) It is great to see further development in hypervisors and virtual machine monitors. Especially thanks to [Martin Lucina](https://lucina.net) for getting things sorted.
|
|
|
|
I'm interested in feedback, either via
|
|
[twitter](https://twitter.com/h4nnes) or as an issue on the [data repository on
|
|
GitHub](https://github.com/hannesm/hannes.nqsb.io/issues).
|
|
|
|
# Other updates in the MirageOS ecosystem
|
|
|
|
There were some busy times, several pull requests are still waiting to get merged (e.g. some cosmetics in [mirage](https://github.com/mirage/mirage/pull/544) as preconditions for treemaps and dependency diagrams), I [proposed](https://github.com/mirage/mirage/pull/547) to use `sleep_ns : int64 -> unit io` instead of the `sleep : float -> unit io` (nobody wants floating point numbers); also an RFC for [random](https://github.com/mirage/mirage/pull/551), Matt Gray [proposed](https://github.com/mirage/mirage/pull/548) to get rid of `CLOCK` (and have a `PCLOCK` and a `MCLOCK` instead). Soon there will be a major MirageOS release which breaks all the previous unikernels! :)
|