hannes/hannes.robur.coop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

retreat 2024

Browse source
This commit is contained in:
Hannes Mehnert 2024-05-15 16:38:51 +02:00
parent 1b759a1698
commit fd990ef4ae
1 changed files with 42 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

42
Posts/Retreat2024 Normal file
View file

@ -0,0 +1,42 @@
---
title: The MirageOS retreat 2024
author: hannes
tags: mirageos, community
abstract: My involvement and experience of the MirageOS retreat 2024
---
End of April I spent a week in Marrakech to work on MirageOS at our annual [retreat](https://retreat.mirage.io). This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.
Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our [website](https://mirage.io) only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.
## Operator handbook
I sat down with some people to start off a [MirageOS operator handbook](https://mirage.github.io/operator-handbook/index.html) - we talked about the scope and the early chapters (and what is not in scope). This is still very much work in progress, [issues](https://github.com/mirage/operator-handbook) and pull requests are welcome. The target audience is someone who wants to execute a MirageOS unikernel with hvt on a laptop or server. It will not be necessary to install OCaml, but the focus is really on the operational aspects with binaries that we provide from our [reproducible build infrastructure](https://builds.robur.coop).
In contrast to earlier attempts, this time we started with the focus and the pain points that we encountered at the retreat. We didn't plan all the chapters upfront or tried to start by developing and compiling a MirageOS "hello world". I have confidence that our approach is a great start for a handbook that has been missing in the last decade.
## Mollymawk
Also related to the deployment discussion, I re-started work from last year's retreat on a Web UI for [Albatross](/Posts/Albatross). Welcome [mollymawk](https://github.com/robur-coop/mollymawk). The idea is to have both a UI but also a HTTP API for deploying MirageOS unikernels via e.g. GitHub actions by a simple HTTP POST request. Observation and management of unikernels is also part of mollymawk. [Pixie](https://github.com/PizieDust) worked a lot on the JavaScript and web side of it, while I put some effort into the MirageOS and Albatross interaction. Within days we got an initial prototype up and running and could show it to others. This already is able to gather information of running unikernels, the console output, and create and destroy unikernels. We will continue on mollymawk in the future.
From the design point, mollymawk communicates via TLS to albatross. It contains the TLS certificate and private key to access albatross (and generates temporary certificates for all the actions required). This also means that mollymawk needs to protect its secrets, otherwise anyone can mess around with albatross. Mollymawk itself is a unikernel that runs on top of albatross. What could possibly go wrong? We plan to add authentication soon to mollymawk, and can then use it for our own purposes. This will greatly improve the state of our running unikernels, since updating our unikernels from our reproducible build infrastructure will then be a click on a button, with a revert button if there are issues.
## Defunctorising MirageOS
Other discussions were about porting existing OCaml code to MirageOS, and what are the hurdles. The excessive use of functors are usually cumbersome for existing projects that need to revise their internal code structure. I have some issues with the use of functors as well, especially at places where they don't seem to be necessary. A rule of thumb I learned is that a functor is great if you want to have something in the same application instantiated with different modules -- i.e. a set of string and a set of integers. Now, in MirageOS we use even for time (providing the function `sleep`), clocks (monotonic and posix time), random (random data), network interface functors. The reason is that in earlier days (before mirage 4.0.0) our compilation strategy used installed opam packages. Since mirage 4, a monorepo is used to compile the unikernel.
Talking about time - I don't think anyone wants to have two different sleep functions in their application, similar for monotonic clock and posix time. Also the existing open source unikernels only use a single time and clock. The random interface has as well some history connected, since our cryptographic library used to bundle everything in a single opam package, and required arbitrary precision integers (via zarith using gmp), and we didn't want to impose a dependency on gmp for every unikernel. Nowadays, the CSPRNG code is separated from gmp, and we can just use it. The network interface: indeed if you're using solo5 (hvt or virtio or whatnot), you will never need a unix (tap-based) network implementation, neither xen netfront/netback. Thus, the functorisation is not necessary.
At the retreat we discussed and thought about replacing the functors in MirageOS with other mechanisms - conditional compilation (depending on the target) or the linking trick (dune variants) or a ppx are candidates, apart from other things (maybe first-class modules fit the requirement as well).
I got into coding a bit, and first looked into conditional compilation with the [`select`](https://dune.readthedocs.io/en/stable/reference/library-dependencies.html#alternative-dependencies) clauses from dune. They turned out to be not very flexible (you've to decide in the package what are all the options, and which one to take). I switched to [dune variants](https://dune.readthedocs.io/en/stable/variants.html#dune-variants), where the idea is that you provide a virtual interface, and only when compiling the executable you specify the implementation. This is great, since the mirage tool (at mirage configure time) can emit the dependencies. I went ahead and defunctorised the whole network stack - it was nice to see it is possible, but this is not easily upstreamable. There's as well at least [one issue](https://github.com/ocaml/dune/issues/10460) in the dune variant code, I'm sure it will get debugged and fixed once we decide that dune variants are the solution we want to try out.
## Conclusion
I found the retreat very inspiring, there were lots of interest in various topics, and a lot of projects have been worked on. I'm also very happy that I managed to contribute and start some projects that will hopefully ease MirageOS adaption -- both on the deployment side, but as well on the developer experience side.
I'm looking forward to other writeups and future retreats, or other events where I can say hi to attendees.