End of April I spent a week in Marrakech to work on MirageOS at our annual retreat. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
End of April I spent a week in Marrakech to work on MirageOS at our annual retreat. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.
Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our website only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.
@@ -13,6 +13,7 @@
Other discussions were about porting existing OCaml code to MirageOS, and what are the hurdles. The excessive use of functors are usually cumbersome for existing projects that need to revise their internal code structure. I have some issues with the use of functors as well, especially at places where they don't seem to be necessary. A rule of thumb I learned is that a functor is great if you want to have something in the same application instantiated with different modules -- i.e. a set of string and a set of integers. Now, in MirageOS we use even for time (providing the function sleep), clocks (monotonic and posix time), random (random data), network interface functors. The reason is that in earlier days (before mirage 4.0.0) our compilation strategy used installed opam packages. Since mirage 4, a monorepo is used to compile the unikernel.
Talking about time - I don't think anyone wants to have two different sleep functions in their application, similar for monotonic clock and posix time. Also the existing open source unikernels only use a single time and clock. The random interface has as well some history connected, since our cryptographic library used to bundle everything in a single opam package, and required arbitrary precision integers (via zarith using gmp), and we didn't want to impose a dependency on gmp for every unikernel. Nowadays, the CSPRNG code is separated from gmp, and we can just use it. The network interface: indeed if you're using solo5 (hvt or virtio or whatnot), you will never need a unix (tap-based) network implementation, neither xen netfront/netback. Thus, the functorisation is not necessary.
At the retreat we discussed and thought about replacing the functors in MirageOS with other mechanisms - conditional compilation (depending on the target) or the linking trick (dune variants) or a ppx are candidates, apart from other things (maybe first-class modules fit the requirement as well).
+
Another benefit of conditional compilation and dune variants is editor support -- jumping to the definition of time (clock, etc.) now simply works because the compiler knows which code is used at runtime (in contrast to when a functor is used).
I got into coding a bit, and first looked into conditional compilation with the select clauses from dune. They turned out to be not very flexible (you've to decide in the package what are all the options, and which one to take). I switched to dune variants, where the idea is that you provide a virtual interface, and only when compiling the executable you specify the implementation. This is great, since the mirage tool (at mirage configure time) can emit the dependencies. I went ahead and defunctorised the whole network stack - it was nice to see it is possible, but this is not easily upstreamable. There's as well at least one issue in the dune variant code, I'm sure it will get debugged and fixed once we decide that dune variants are the solution we want to try out.
diff --git a/atom b/atom
index bb419f6..9a9610c 100644
--- a/atom
+++ b/atom
@@ -1,4 +1,4 @@
-urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156full stack engineer2024-05-15T14:44:14-00:00<p>My involvement and experience of the MirageOS retreat 2024</p>
+urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156full stack engineer2024-05-16T09:46:39-00:00<p>My involvement and experience of the MirageOS retreat 2024</p>
2024-05-15T14:38:51-00:00<p>End of April I spent a week in Marrakech to work on MirageOS at our annual <a href="https://retreat.mirage.io">retreat</a>. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.</p>
<p>Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.</p>
<p>Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our <a href="https://mirage.io">website</a> only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).</p>
@@ -13,6 +13,7 @@
<p>Other discussions were about porting existing OCaml code to MirageOS, and what are the hurdles. The excessive use of functors are usually cumbersome for existing projects that need to revise their internal code structure. I have some issues with the use of functors as well, especially at places where they don't seem to be necessary. A rule of thumb I learned is that a functor is great if you want to have something in the same application instantiated with different modules -- i.e. a set of string and a set of integers. Now, in MirageOS we use even for time (providing the function <code>sleep</code>), clocks (monotonic and posix time), random (random data), network interface functors. The reason is that in earlier days (before mirage 4.0.0) our compilation strategy used installed opam packages. Since mirage 4, a monorepo is used to compile the unikernel.</p>
<p>Talking about time - I don't think anyone wants to have two different sleep functions in their application, similar for monotonic clock and posix time. Also the existing open source unikernels only use a single time and clock. The random interface has as well some history connected, since our cryptographic library used to bundle everything in a single opam package, and required arbitrary precision integers (via zarith using gmp), and we didn't want to impose a dependency on gmp for every unikernel. Nowadays, the CSPRNG code is separated from gmp, and we can just use it. The network interface: indeed if you're using solo5 (hvt or virtio or whatnot), you will never need a unix (tap-based) network implementation, neither xen netfront/netback. Thus, the functorisation is not necessary.</p>
<p>At the retreat we discussed and thought about replacing the functors in MirageOS with other mechanisms - conditional compilation (depending on the target) or the linking trick (dune variants) or a ppx are candidates, apart from other things (maybe first-class modules fit the requirement as well).</p>
+<p>Another benefit of conditional compilation and dune variants is editor support -- jumping to the definition of time (clock, etc.) now simply works because the compiler knows which code is used at runtime (in contrast to when a functor is used).</p>
<p>I got into coding a bit, and first looked into conditional compilation with the <a href="https://dune.readthedocs.io/en/stable/reference/library-dependencies.html#alternative-dependencies"><code>select</code></a> clauses from dune. They turned out to be not very flexible (you've to decide in the package what are all the options, and which one to take). I switched to <a href="https://dune.readthedocs.io/en/stable/variants.html#dune-variants">dune variants</a>, where the idea is that you provide a virtual interface, and only when compiling the executable you specify the implementation. This is great, since the mirage tool (at mirage configure time) can emit the dependencies. I went ahead and defunctorised the whole network stack - it was nice to see it is possible, but this is not easily upstreamable. There's as well at least <a href="https://github.com/ocaml/dune/issues/10460">one issue</a> in the dune variant code, I'm sure it will get debugged and fixed once we decide that dune variants are the solution we want to try out.</p>
<p>The current result is a <a href="https://lists.xenproject.org/archives/html/mirageos-devel/2024-05/msg00002.html">mail to the development list</a>, and a <a href="https://github.com/mirage/mirage/pull/1529">defunctorising time PR</a>, as well as a <a href="https://github.com/mirage/mirage/pull/1533">bootvar PR</a> (relying on other PRs).</p>
<h2 id="conclusion">Conclusion</h2>
@@ -20,7 +21,7 @@
<p>I'm looking forward to other writeups and future retreats, or other events where I can say hi to attendees.</p>
<p>We at <a href="https://robur.coop">Robur</a> are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by <a href="https://aenderwerk.de/donate/">donating</a> (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).</p>
<p>If you have any questions, reach us best via eMail to team AT robur DOT coop.</p>
-urn:uuid:2c9e2634-78c5-5957-b6f2-d0706f80d7ebThe MirageOS retreat 20242024-05-15T14:44:14-00:00hannes<p>Core Internet protocols require operational experiments, even if formally specified</p>
+urn:uuid:2c9e2634-78c5-5957-b6f2-d0706f80d7ebThe MirageOS retreat 20242024-05-16T09:46:39-00:00hannes<p>Core Internet protocols require operational experiments, even if formally specified</p>
2023-11-28T21:17:01-00:00<p>The <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">Transmission Control Protocol (TCP)</a> is one of the main Internet protocols. Usually spoken on top of the Internet Protocol (legacy version 4 or version 6), it provides a reliable, ordered, and error-checked stream of octets. When an application uses TCP, they get these properties for free (in contrast to UDP).</p>
<p>As common for Internet protocols, TCP is specified in a series of so-called requests for comments (RFC). The latest revised version from August 2022 is <a href="https://datatracker.ietf.org/doc/html/rfc9293">RFC 9293</a>; the initial one was <a href="https://datatracker.ietf.org/doc/html/rfc793">RFC 793</a> from September 1981.</p>
<h1 id="my-brief-personal-tcp-story">My brief personal TCP story</h1>