updated from main (commit 8db22fe152)

2023-11-20 16:55:40 +00:00 · 2023-11-20 16:55:40 +00:00 · 5cf66b4a7b
commit 5cf66b4a7b
parent 148deadf59
28 changed files with 289 additions and 289 deletions
--- a/6
+++ b/6
@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>About</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="About" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>About</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/myself" class="tag">myself</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-01 (last updated: 2021-11-19)</span><article><h2>What is a &quot;full stack engineer&quot;?</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>About</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="About" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>About</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/myself" class="tag">myself</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-01 (last updated: 2021-11-19)</span><article><h2 id="what-is-a-full-stack-engineer">What is a &quot;full stack engineer&quot;?</h2>
 <p>Analysing the word literally, we should start with silicon and some electrons,
 maybe a soldering iron, and build everything all the way up to our favourite
 communication system.</p>
@ -12,7 +12,7 @@ case you're interested in trustworthiness of hardware.</p>
 simple to setup, secure, decentralised infrastructure.  We're not there yet,
 which also means I've plenty of projects :).</p>
 <p>I will write about my projects, which cover topics on various software layers.</p>
-<h3>Myself</h3>
+<h3 id="myself">Myself</h3>
 <p>I'm Hannes Mehnert, a <a href="http://www.catb.org/jargon/html/H/hacker.html">hacker</a>
 (in the original sense of the word), 3X years old.  In my spare time, I'm not
 only a hacker, but also a barista.  I like to travel and repair my recumbent
@ -81,7 +81,7 @@ another post.</p>
 fast enough (&quot;Reassuring, because our blanket performance statement 'OCaml
 delivers at least 50% of the performance of a decent C compiler' is
 not invalidated :-)&quot; <a href="https://lwn.net/Articles/19378/">Xavier Leroy</a>), and the <a href="https://opam.ocaml.org/packages/">community</a> is sufficiently large.</p>
-<h3>Me on the intertubes</h3>
+<h3 id="me-on-the-intertubes">Me on the intertubes</h3>
 <p>You can find me on <a href="https://twitter.com/h4nnes">twitter</a> and on
 <a href="https://github.com/hannesm">GitHub</a>.</p>
 <p>The data of this blog is <a href="https://git.robur.io/hannes/hannes.robur.coop">stored in a git repository</a>.</p>
--- a/Posts/ARP
+++ b/Posts/ARP
@ -1,17 +1,17 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Re-engineering ARP</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Re-engineering ARP" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Re-engineering ARP</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2016-07-12 (last updated: 2021-11-19)</span><article><h2>What is ARP?</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Re-engineering ARP</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Re-engineering ARP" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Re-engineering ARP</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2016-07-12 (last updated: 2021-11-19)</span><article><h2 id="what-is-arp">What is ARP?</h2>
 <p>ARP is the <em>A</em>ddress <em>R</em>esolution <em>P</em>rotocol, widely used in legacy IP networks (which support only IPv4).  It is responsible to translate an IPv4 address to an Ethernet address.  It is strictly more general, abstracting over protocol and hardware addresses.  It is basically DNS (the domain name system) on a different layer.</p>
 <p>ARP is link-local: ARP frames are not routed into other networks, all stay in the same broadcast domain.  Thus there is no need for a hop limit (time-to-live).  A reverse lookup mechanism (hardware address to protocol) is also available, named reverse ARP ;).</p>
 <p>I will focus on ARP in this article, as used widely to translate IPv4 addresses into Ethernet addresses.  There are two operations in ARP: request and response.  A request is usually broadcasted to all hosts (by setting the destination to the broadcast Ethernet address, <code>ff:ff:ff:ff:ff:ff</code>), while a reply is send via unicast (to the host which requested that information).</p>
 <p>The frame format is pretty straightforward: 2 bytes hardware address type, 2 bytes protocol type, 1 byte length for both types, 2 bytes operation, followed by source addresses (hardware and protocol), and target addresses.  In total 28 bytes, considering 48 bit Ethernet addresses and 32 bit IPv4 addresses.</p>
 <p>It was initially specified in <a href="https://tools.ietf.org/html/rfc826">RFC 826</a>, but reading through <a href="https://tools.ietf.org/html/rfc1122">RFC 1122</a> (requirements for Internet Hosts - Communication layer), and maybe the newer <a href="https://tools.ietf.org/html/rfc5227">RFC 5227</a> (IPv4 address conflict detection) does not hurt.</p>
 <p>On UNIX systems, you can investigate your arp table, also called arp cache, using the <code>arp</code> command line utility.</p>
-<h3>Protocol logic</h3>
+<h3 id="protocol-logic">Protocol logic</h3>
 <p>Let us look what our ARP handler actually needs to do?  Translating IPv4 addresses to Ethernet addresses, but where does it learn new information?</p>
 <p>First of all, our ARP handler needs to know its own IPv4 address and its Ethernet address.  It will even broadcast them on startup, so-called gratuitous ARP.  The purpose of this is to inform all other hosts on the same network that we are here now.  And if another host, let's name it barf, has the same IPv4 address, some sort of conflict resolution needs to happen (otherwise all hosts on the network are confused to whether to send us or barf packets).</p>
 <p>Once initialisation is over, our ARP handler needs to wait for ARP requests from other hosts on the network, and if addresses to our IPv4 address, issue a reply.  The other event which might happen is that a user wants to send an IPv4 packet to another host on the network.  In this case, we either already have the Ethernet address in our cache, or we need to send an ARP request to the network and wait for a reply.  Since packets might get lost, we actually need to retry sending ARP requests until a limit is reached.  To keep the cache in a reasonable size, old entries should be dropped if unused.  Also, the Ethernet address of hosts may change, due to hardware replacement or failover.</p>
 <p>That's it.  Pretty straightforward.</p>
-<h2>Design</h2>
+<h2 id="design">Design</h2>
 <p>Back in 2008, together with Andreas Bogk, we just used a hash table and installed expiration and retransmission timers when needed.  Certainly timers sometimes needed to be cancelled, and testing the code was cumbersome.  It were only <a href="https://github.com/dylan-hackers/network-night-vision/blob/master/network/ip-stack/layers/network/arp/arp.dylan">250 lines of Dylan code</a> plus some <a href="https://github.com/dylan-hackers/network-night-vision/blob/master/protocols/ipv4.dylan">wire format definition</a>.</p>
 <p>Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated.  The code should not contain surprises, but straightforward to understand.  The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should.   Once this is achieved, testing is straightforward.  If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused.  It can also be repurposed to run as a test oracle.  You can read more marketing of this style in our <a href="https://usenix15.nqsb.io">Usenix security paper</a>.</p>
 <p>My proposed style and hash tables are not good friends, since hash tables in OCaml are imperative structures.  Instead, a <em>Map</em> (<a href="http://caml.inria.fr/pub/docs/manual-ocaml/libref/Map.html">documentation</a>) is a functional data structure for associating keys with values.  Its underlying data structure is a balanced binary tree.</p>
@ -25,15 +25,15 @@
 <li><em>Query</em> a query for an IPv4 address using some state leads to a successor state, and either an immediate answer with the Ethernet address, or an ARP request to be sent and waiting for an answer, or just waiting for an answer in the case another task has already requested that IPv4 address.  Since we don't want to convolute the protocol core with tasks, we'll let the effectful layer decide how to achieve that by abstracting over some alpha to store, and requiring a <code>merge : alpha option -&gt; alpha</code> function.
 </li>
 </ul>
-<h3>Excursion: security</h3>
+<h3 id="excursion-security">Excursion: security</h3>
 <p>ARP is a link-local protocol, thus attackers have to have access to the same link-layer: either a cable in the same switch or hub, or in the same wireless network (if you're into modern technology).</p>
 <p>A very common attack vector for protocols is the so called person in the middle attack, where the attacker sits between you and the remote host.  An attacker can achieve this using ARP spoofing: if they can convince your computer that the attacker is the gateway, your computer will send all packets to the attacker, who either forwards them to the remote host, or modifies them, or drops them.</p>
 <p>ARP does not employ any security mechanism, it is more a question of receiving the first answer (depending on the implementation).  A common countermeasure is to manually fill the cache with the gateway statically.  This only needs updates if the gateway is replaced, or gets a new network card.</p>
 <p>Denial of service attacks are also possible using ARP: if the implementation preserves all replies, the cache might expand immensely.  This happens sometimes in switch hardware, which have a limited cache, and once it is full, they go into hub mode.  This means all frames are broadcasted on all ports.  This enables an attacker to passively sniff all traffic in the local network.</p>
 <p>One denial of service attack vector is due to choosing a hash table as underlying store.  Its hash function should be collision-resistant, one way, and its output should be fixed length.  A good choice would be a cryptographic hash function (like SHA-256), but these are too expensive and thus rarely used for hash tables. <a href="https://www.usenix.org/conference/12th-usenix-security-symposium/denial-service-algorithmic-complexity-attacks">Denial of Service via Algorithmic Complexity Attacks</a> and <a href="https://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf">Efficient Denial of Service Attacks on Web Application Platforms</a> are worth studying.  If you expose your hash function to user input (and don't use a private seed), you might accidentally open your attack surface.</p>
-<h3>Back to our design</h3>
+<h3 id="back-to-our-design">Back to our design</h3>
 <p>To mitigate person in the middle attacks, we provide an API to add static entries, which are never overwritten by network input.  While our own IPv4 addresses are advertised if a matching ARP request was received, other static entries are not advertised (neither are dynamic entries).  We do only insert entries to our cache if we have an outstanding request or already an entry.  To provide low latency, just before a dynamic entry would timeout, we send another request for this IPv4 address to the network.</p>
-<h3>Implementation</h3>
+<h3 id="implementation">Implementation</h3>
 <p>I have the <a href="https://github.com/hannesm/arp">source</a>, its <a href="https://hannesm.github.io/arp">documentation</a>, a test suite and a <a href="https://hannesm.github.io/arp/coverage">coverage report</a> online.</p>
 <p>The implementation of the core logic still fits in less than 250 lines of code.  Below 100 more lines are needed for decoding and encoding byte buffers.  And another 140 lines to implement the Mirage ARP interface.  Tests are available which cover the protocol logic and decoding/encoding to 100%.</p>
 <p>The effectful layer is underspecified (especially regarding conflicts: what happens if there is an outstanding request for an IPv4 address and I add a static entry for this?).  There is an implementation based on hash tables, which I used to benchmark a bit.</p>
--- a/Posts/Albatross
+++ b/Posts/Albatross
@ -1,8 +1,8 @@
 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying reproducible unikernels with albatross</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying reproducible unikernels with albatross" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying reproducible unikernels with albatross</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2022-11-17 (last updated: 2023-05-16)</span><article><p>EDIT (2023-05-16): Updated with albatross release version 2.0.0.</p>
-<h2>Deploying MirageOS unikernels</h2>
+<h2 id="deploying-mirageos-unikernels">Deploying MirageOS unikernels</h2>
 <p>More than five years ago, I posted <a href="/Posts/VMM">how to deploy MirageOS unikernels</a>. My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described <a href="/Posts/Deploy">how to deploy reproducible unikernels</a>.</p>
-<h2>Albatross</h2>
+<h2 id="albatross">Albatross</h2>
 <p>In recent months we worked hard on the underlying infrastructure: <a href="https://github.com/roburio/albatross">albatross</a>. Albatross is the orchestration system for MirageOS unikernels that use solo5 with <a href="https://github.com/Solo5/solo5/blob/master/docs/architecture.md">hvt or spt tender</a>. It deals with three tasks:</p>
 <ul>
 <li>unikernel creation (destroyal, restart)
@ -13,50 +13,50 @@
 </li>
 </ul>
 <p>An addition to the above is dealing with multiple tenants on the same machine: remote management of your unikernel fleet via TLS, and resource policies.</p>
-<h2>History</h2>
+<h2 id="history">History</h2>
 <p>The initial commit of albatross was in May 2017. Back then it replaced the shell scripts and manual <code>scp</code> of unikernel images to the server. Over time it evolved and adapted to new environments. Initially a solo5 unikernel would only know of a single network interface, these days there can be multiple distinguished by name. Initially there was no support for block devices. Only FreeBSD was supported in the early days. Nowadays we built daily packages for Debian, Ubuntu, FreeBSD, and have support for NixOS, and the client side is supported on macOS as well.</p>
-<h3>ASN.1</h3>
+<h3 id="asn.1">ASN.1</h3>
 <p>The communication format between the albatross daemons and clients was changed multiple times. I'm glad that albatross uses ASN.1 as communication format, which makes extension with optional fields easy, and also allows &quot;choice&quot; (the sum type) to be not tagged (the binary is the same as no choice type), thus adding choice to an existing grammar, and preserving the old in the default (untagged) case is a decent solution.</p>
 <p>So, if you care about backward and forward compatibility, as we do, since we may be in control of which albatross servers are deployed on our machine, but not what albatross versions the clients are using -- it may be wise to look into ASN.1. Recent efforts (json with schema, ...) may solve similar issues, but ASN.1 is as well very tiny in size.</p>
-<h2>What resources does a unikernel need?</h2>
+<h2 id="what-resources-does-a-unikernel-need">What resources does a unikernel need?</h2>
 <p>A unikernel is just an operating system for a single service, there can't be much it can need.</p>
-<h3>Name</h3>
+<h3 id="name">Name</h3>
 <p>So, first of all a unikernel has a name, or a handle. This is useful for reporting statistics, but also to specify which console output you're interested in. The name is a string with printable ASCII characters (and dash '-' and dot '.'), with a length up to 64 characters - so yes, you can use an UUID if you like.</p>
-<h3>Memory</h3>
+<h3 id="memory">Memory</h3>
 <p>Another resource is the amount of memory assigned to the unikernel. This is specified in megabyte (as solo5 does), with the range being 10 (below not even a hello world wants to start) to 1024.</p>
-<h3>Arguments</h3>
+<h3 id="arguments">Arguments</h3>
 <p>Of course, you can pass via albatross boot parameters to the unikernel. Albatross doesn't impose any restrictions here, but the lower levels may.</p>
-<h3>CPU</h3>
+<h3 id="cpu">CPU</h3>
 <p>Due to multiple tenants, and side channel attacks, it looked right at the beginning like a good idea to restrict each unikernel to a specific CPU. This way, one tenant may use CPU 5, and another CPU 9 - and they'll not starve each other (best to make sure that these CPUs are in different packages). So, albatross takes a number as the CPU, and executes the solo5 tender within <code>taskset</code>/<code>cpuset</code>.</p>
-<h3>Fail behaviour</h3>
+<h3 id="fail-behaviour">Fail behaviour</h3>
 <p>In normal operations, exceptional behaviour may occur. I have to admit that I've seen MirageOS unikernels that suffer from not freeing all the memory they have allocated. To avoid having to get up at 4 AM just to start the unikernel that went out of memory, there's the possibility to restart the unikernel when it exited. You can even specify on which exit codes it should be restarted (the exit code is the only piece of information we have from the outside what caused the exit). This feature was implemented in October 2019, and has been very precious since then. :)</p>
-<h3>Network</h3>
+<h3 id="network">Network</h3>
 <p>This becomes a bit more complex: a MirageOS unikernel can have network interfaces, and solo5 specifies a so-called manifest with a list of these (name and type, and type is so far always basic). Then, on the actual server there are bridges (virtual switches) configured. Now, these may have the same name, or may need to be mapped. And of course, the unikernel expects a tap interface that is connected to such a bridge, not the bridge itself. Thus, albatross creates tap devices, attaches these to the respective bridges, and takes care about cleaning them up on teardown. The albatross client verifies that for each network interface in the manifest, there is a command-line argument specified (<code>--net service:my_bridge</code> or just <code>--net service</code> if the bridge is named service). The tap interface name is not really of interest to the user, and will not be exposed.</p>
-<h3>Block devices</h3>
+<h3 id="block-devices">Block devices</h3>
 <p>On the host system, it's just a file, and passed to the unikernel. There's the need to be able to create one, dump it, and ensure that each file is only used by one unikernel. That's all that is there.</p>
-<h2>Metrics</h2>
+<h2 id="metrics">Metrics</h2>
 <p>Everyone likes graphs, over time, showing how much traffic or CPU or memory or whatever has been used by your service. Some of these statistics are only available in the host system, and it is also crucial for development purposes to compare whether the bytes sent in the unikernel sum up to the same on the host system's tap interface.</p>
 <p>The albatross-stats daemon collects metrics from three sources: network interfaces, getrusage (of a child process), VMM debug counters (to count VM exits etc.). Since the recent 1.5.3, albatross-stats now connects at startup to the albatross-daemon and then retrieves the information which unikernels are up and running, and starts periodically collecting data in memory.</p>
 <p>Other clients, being it a dump on your console window, a write into an rrd file (good old MRTG times), or a push to influx, can use the stats data to correlate and better analyse what is happening on the grand scale of things. This helped a lot by running several unikernels with different opam package sets to figure out which opam packages leave their hands on memory over time.</p>
 <p>As a side note, if you make the unikernel name also available in the unikernel, it can tag its own metrics with the same identifier, and you can correlate high-level events (such as amount of HTTP requests) with low-level things &quot;allocated more memory&quot; or &quot;consumed a lot of CPU&quot;.</p>
-<h2>Console</h2>
+<h2 id="console">Console</h2>
 <p>There's not much to say about the console, just that the albatross-console daemon is running with low privileges, and reading from a FIFO that the unikernel writes to. It never writes anything to disk, but keeps the last 1000 lines in memory, available from a client asking for it.</p>
-<h2>The daemons</h2>
+<h2 id="the-daemons">The daemons</h2>
 <p>So, the main albatross-daemon runs with superuser privileges to create virtual machines, and opens a unix domain socket where the clients and other daemons are connecting to. The other daemons are executed with normal user privileges, and never write anything to disk.</p>
 <p>The albatross-daemon keeps state about the running unikernels, and if it is restarted, the unikernels are started again. Maybe worth to mention that this lead sometimes to headaches (due to data being dumped to disk, and the old format should always be supported), but was also a huge relief to not have to care about creating all the unikernels just because albatross-daemon was killed.</p>
-<h2>Remote management</h2>
+<h2 id="remote-management">Remote management</h2>
 <p>There's one more daemon program: albatross-tls-endpoint. It accepts clients via a remote TCP connection, and establish a mutual-authenticated TLS handshake. When done, the command is forwarded to the respective Unix domain socket, and the reply is sent back to the client.</p>
 <p>The daemon itself has a X.509 certificate to authenticate, but the client is requested to show its certificate chain as well. This by now requires TLS 1.3, so the client certificates are sent over the encrypted channel.</p>
 <p>A step back, X.509 certificate contains a public key and a signature from one level up. When the server knows about the root (or certificate authority (CA)) certificate, and following the chain can verify that the leaf certificate is valid. Additionally, a X.509 certificate is a ASN.1 structure with some fixed fields, but also contains extensions, a key-value store where the keys are object identifiers, and the values are key-dependent data. Also note that this key-value store is cryptographically signed.</p>
 <p>Albatross uses the object identifier, assigned to Camelus Dromedarius (MirageOS - 1.3.6.1.4.1.49836.42) to encode the command to be executed. This means that once the TLS handshake is established, the command to be executed is already transferred.</p>
 <p>In the leaf certificate, there may be the &quot;create unikernel&quot; command with the unikernel image, it's boot parameters, and other resources. Or a &quot;read the console of my unikernel&quot;. In the intermediate certificates (from root to leaf), resource policies are encoded (this path may only have X unikernels running with a total of Y MB memory, and Z MB of block storage, using CPUs A and B, accessing bridges C and D). From the root downwards these policies may only decrease. When a unikernel should be created (or other commands are executed), the policies are verified to hold. If they do not, an error is reported.</p>
-<h2>Fleet management</h2>
+<h2 id="fleet-management">Fleet management</h2>
 <p>Of course it is very fine to create your locally compiled unikernel to your albatross server, go for it. But in terms of &quot;what is actually running here?&quot; and &quot;does this unikernel need to be updated because some opam package had a security issues?&quot;, this is not optimal.</p>
 <p>Since we provide <a href="https://builds.robur.coop">daily reproducible builds</a> with the current HEAD of the main opam-repository, and these unikernels have no configuration embedded (but take everything as boot parameters), we just deploy them. They come with the information what opam packages contributed to the binary, which environment variables were set, and which system packages were installed with which versions.</p>
 <p>The whole result of reproducible builds for us means: we have a hash of a unikernel image that we can lookup in our build infrastructure, and take a look whether there is a newer image for the same job. And if there is, we provide a diff between the packages contributed to the currently running unikernel and the new image. That is what the albatross-client update command is all about.</p>
 <p>Of course, your mileage may vary and you want automated deployments where each git commit triggers recompilation and redeployment. The downside would be that sometimes only dependencies are updated and you've to cope with that.</p>
 <p>There is a client <code>albatross-client</code>, depending on arguments either connects to a local Unix domain socket, or to a remote albatross instance via TCP and TLS, or outputs a certificate signing request for later usage. Data, such as the unikernel ELF image, is compressed in certificates.</p>
-<h2>Installation</h2>
+<h2 id="installation">Installation</h2>
 <p>For Debian and Ubuntu systems, we provide package repositories. Browse the dists folder for one matching your distribution, and add it to <code>/etc/apt/sources.list</code>:</p>
 <pre><code>$ wget -q -O /etc/apt/trusted.gpg.d/apt.robur.coop.gpg https://apt.robur.coop/gpg.pub
 $ echo &quot;deb https://apt.robur.coop ubuntu-20.04 main&quot; &gt;&gt; /etc/apt/sources.list # replace ubuntu-20.04 with e.g. debian-11 on a debian buster machine
@ -77,7 +77,7 @@ $ pkg install solo5 albatross
 </code></pre>
 <p>Please ensure to have at least version 2.0.0 of albatross installed.</p>
 <p>For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (<code>opam install solo5 albatross</code>). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>After five years of development and operating albatross, feel free to get it and try it out. Or read the code, discuss issues and shortcomings with us - either at the issue tracker or via eMail.</p>
 <p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
 </article></div></div></main></body></html>
--- a/Posts/BadRecordMac
+++ b/Posts/BadRecordMac
@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Catch the bug, walking through the stack</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Catch the bug, walking through the stack" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a></div><span class="date">Published: 2016-05-03 (last updated: 2021-11-19)</span><article><h2>BAD RECORD MAC</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Catch the bug, walking through the stack</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Catch the bug, walking through the stack" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a></div><span class="date">Published: 2016-05-03 (last updated: 2021-11-19)</span><article><h2 id="bad-record-mac">BAD RECORD MAC</h2>
 <p>Roughly 2 weeks ago, <a href="https://github.com/Engil">Engil</a> informed me that a TLS alert pops up in his browser sometimes when he reads this website.  His browser reported that the <a href="https://en.wikipedia.org/wiki/Message_authentication_code">message authentication code</a> was wrong.  From <a href="https://tools.ietf.org/html/rfc5246">RFC 5246</a>: This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).</p>
 <p>I tried hard, but could not reproduce, but was very worried and was eager to find the root cause (some little fear remained that it was in our TLS stack).  I setup this website with some TLS-level tracing (extending the code from our <a href="https://tls.openmirage.org">TLS handshake server</a>).  We tried to reproduce the issue with traces and packet captures (both on client and server side) in place from our computer labs office with no success.  Later, Engil tried from his home and after 45MB of wire data, ran into this issue.  Finally, evidence!  Isolating the TCP flow with the alert resulted in just about 200KB of packet capture data (TLS ASCII trace around 650KB).</p>
 <p><img src="/static/img/encrypted-alert.png" alt="encrypted alert" /></p>
@ -43,7 +43,7 @@
 <p>Certainly, interfacing the outside world is complex.  The <a href="https://github.com/mirage/mirage-block-xen">mirage-block-xen</a> library uses a similar protocol to access block devices.  From a brief look, that library seems to be safe (using 64bit identifiers).</p>
 <p>I'm interested in feedback, either via
 <a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
-<h2>Other updates in the MirageOS ecosystem</h2>
+<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
 <ul>
 <li>Canopy uses a <a href="https://github.com/Engil/Canopy/issues/30#issuecomment-215010365">map instead of a hashtable</a>, <a href="https://hannes.nqsb.io/tags">tags</a> now contains a list of tags (<a href="https://github.com/Engil/Canopy/pull/39">PR here</a>), both thanks to voila!  I also use the <a href="https://github.com/Engil/Canopy/pull/38">new CSS</a> from Engil
 </li>
--- a/Posts/BottomUp
+++ b/Posts/BottomUp
@ -1,6 +1,6 @@
 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Counting Bytes</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Counting Bytes" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Counting Bytes</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-06-11 (last updated: 2021-11-19)</span><article><p>I was busy writing code, text, talks, and also spend a week without Internet, where I ground and brewed 15kg espresso.</p>
-<h2>Size of a MirageOS unikernel</h2>
+<h2 id="size-of-a-mirageos-unikernel">Size of a MirageOS unikernel</h2>
 <p>There have been lots of claims and myths around the concrete size of MirageOS unikernels.  In this article I'll apply some measurements which overapproximate the binary sizes.  The tools used for the visualisations are available online, and soon hopefully upstreamed into the mirage tool.  This article uses mirage-2.9.0 (which might be outdated at the time of reading).</p>
 <p>Let us start with a very minimal unikernel, consisting of a <code>unikernel.ml</code>:</p>
 <pre><code class="language-OCaml">module Main (C: V1_LWT.CONSOLE) = struct
@ -36,24 +36,24 @@ PKGS   = functoria lwt mirage-clock-unix mirage-console mirage-logs mirage-types
 21964 mirage/mirage-runtime.a
 </code></pre>
 <p>This still does not sum up to 2.8MB since we're missing the transitive dependencies.</p>
-<h3>Visualising recursive dependencies</h3>
+<h3 id="visualising-recursive-dependencies">Visualising recursive dependencies</h3>
 <p>Let's use a different approach: first recursively find all dependencies.  We do this by using <code>ocamlfind</code> to read <code>META</code> files which contain a list of dependent libraries in their <code>requires</code> line.  As input we use <code>LIBS</code> from the Makefile snippet above.  The code (OCaml script) is <a href="https://gist.github.com/hannesm/bcbe54c5759ed5854f05c8f8eaee4c79">available here</a>.  The colour scheme is red for pieces of the OCaml distribution, yellow for input packages, and orange for the dependencies.</p>
 <p><a href="/static/img/mirage-console.svg"><img src="/static/img/mirage-console.svg" title="UNIX dependencies of hello world" width="700" /></a></p>
 <p>This is the UNIX version only, the Xen version looks similar (but worth mentioning).</p>
 <p><a href="/static/img/mirage-console-xen.svg"><img src="/static/img/mirage-console-xen.svg" title="Xen dependencies of hello world" width="700" /></a></p>
 <p>You can spot at the right that <code>mirage-bootvar</code> uses <code>re</code>, which provoked me to <a href="https://github.com/mirage/mirage-bootvar-xen/pull/19">open a PR</a>, but Jon Ludlam <a href="https://github.com/mirage/mirage-bootvar-xen/pull/18">already had a nicer PR</a> which is now merged (and a <a href="https://github.com/mirage/mirage-bootvar-xen/pull/20">new release is in preparation</a>).</p>
-<h3>Counting bytes</h3>
+<h3 id="counting-bytes">Counting bytes</h3>
 <p>While a dependency graphs gives a big picture of what the composed libraries of a MirageOS unikernel, we also want to know how many bytes they contribute to the unikernel.  The dependency graph only contains the OCaml-level dependencies, but MirageOS has in addition to that a <code>pkg-config</code> universe of the libraries written in C (such as mini-os, openlibm, ...).</p>
 <p>We overapproximate the sizes here by assuming that a linker simply concatenates all required object files.  This is not true, since the sum of all objects is empirically factor two of the actual size of the unikernel.</p>
 <p>I developed a pie chart visualisation, but a friend of mine reminded me that such a chart is pretty useless for comparing slices for the human brain.  I spent some more time to develop a treemap visualisation to satisfy the brain.  The implemented algorithm is based on <a href="http://www.win.tue.nl/~vanwijk/stm.pdf">squarified treemaps</a>, but does not use implicit mutable state.  In addition, the <a href="https://gist.github.com/hannesm/c8a9b2e75bb4f98b5100a838ea125f3b">provided script</a> parses common linker flags (<code>-o -L -l</code>) and collects arguments to be linked in.  It can be passed to <code>ocamlopt</code> as the C linker, more instructions at the end of <code>treemap.ml</code> (which should be cleaned up and integrated into the mirage tool, as mentioned earlier).</p>
 <p><a href="/static/img/mirage-console-bytes.svg"><img src="/static/img/mirage-console-bytes.svg" title="byte sizes of hello-world (UNIX)" width="700" /></a></p>
 <p><a href="/static/img/mirage-console-xen-bytes-full.svg"><img src="/static/img/mirage-console-xen-bytes-full.svg" title="byte sizes of hello-world (Xen)" width="700" /></a></p>
 <p>As mentioned above, this is an overapproximation.  The <code>libgcc.a</code> is only needed on Xen (see <a href="https://github.com/mirage/mirage/commit/c17f2f60a6309322ba45cecb00a808f62f05cf82#commitcomment-17573123">this comment</a>), I have not yet tracked down why there is a <code>libasmrun.a</code> and a <code>libxenasmrun.a</code>.</p>
-<h3>More complex examples</h3>
+<h3 id="more-complex-examples">More complex examples</h3>
 <p>Besides the hello world, I used the same tools on our <a href="http://ownme.ipredator.se">BTC Piñata</a>.</p>
 <p><a href="/static/img/pinata-deps.svg"><img src="/static/img/pinata-deps.svg" title="Piñata dependencies" width="700" /></a></p>
 <p><a href="/static/img/pinata-bytes.svg"><img src="/static/img/pinata-bytes.svg" title="Piñata byte sizes" width="700" /></a></p>
-<h3>Conclusion</h3>
+<h3 id="conclusion">Conclusion</h3>
 <p>OCaml does not yet do dead code elimination, but there <a href="https://github.com/ocaml/ocaml/pull/608">is a PR</a> based on the flambda middle-end which does so.  I haven't yet investigated numbers using that branch.</p>
 <p>Those counting statistics could go into more detail (e.g. using <code>nm</code> to count the sizes of concrete symbols - which opens the possibility to see which symbols are present in the objects, but not in the final binary).  Also, collecting the numbers for each module in a library would be great to have.  In the end, it would be great to easily spot the source fragments which are responsible for a huge binary size (and getting rid of them).</p>
 <p>I'm interested in feedback, either via
--- a/Posts/Conex
+++ b/Posts/Conex
@ -57,7 +57,7 @@ authenticity.  There are different kinds of resources:</p>
 <p>Modifications to identities and authorisations need to be approved by a quorum
 of janitors, package index and release files can be modified either by an authorised
 id or by a quorum of janitors.</p>
-<h2>Documentation</h2>
+<h2 id="documentation">Documentation</h2>
 <p><a href="https://hannesm.github.io/conex/doc/">API documentation</a> is
 available online, also a <a href="https://hannesm.github.io/conex/coverage/">coverage
 report</a>.</p>
@ -74,7 +74,7 @@ adapted to the opam repository.</p>
 <p>The <a href="https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt">TUF
 spec</a>
 has a good overview of attacks and threat model, both of which are shared by conex.</p>
-<h2>What's missing</h2>
+<h2 id="whats-missing">What's missing</h2>
 <ul>
 <li>See <a href="https://github.com/hannesm/conex/issues/7">issue 7</a> for a laundry list
 </li>
@ -89,7 +89,7 @@ has a good overview of attacks and threat model, both of which are shared by con
 <li>Integration into release management systems
 </li>
 </ul>
-<h2>Getting started</h2>
+<h2 id="getting-started">Getting started</h2>
 <p>At the moment, our <a href="https://github.com/ocaml/opam-repository">opam repository</a>
 does not include any metadata needed for signing.  We're in a bootstrap phase:
 we need you to generate a keypair, claim your packages, and approve your releases.</p>
@ -107,7 +107,7 @@ membership and authorised packages were inferred correctly.</p>
 their public key and starts approving releases (and old ones after careful
 checking that the build script, patches, and tarball checksum are valid).
 Each resource can be approved in multiple versions at the same time.</p>
-<h3>Installation</h3>
+<h3 id="installation">Installation</h3>
 <p>TODO: remove clone once <a href="https://github.com/ocaml/opam-repository/pull/8494">PR 8494</a> is merged.</p>
 <pre><code class="language-bash">$ git clone -b auth https://github.com/hannesm/opam-repository.git repo
 $ opam install conex
@ -119,7 +119,7 @@ opam file format.  This means can always manually modify them (but be careful,
 modifications need to increment counters, add checksums, and be signed).  Conex
 does not deal with git, you have to manually <code>git add</code> files and open pull
 requests.</p>
-<h3>Author enrollment</h3>
+<h3 id="author-enrollment">Author enrollment</h3>
 <p>For the opam repository, we will use GitHub ids as conex ids.  Thus, your conex
 id and your GitHub id should match up.</p>
 <pre><code class="language-bash">repo$ conex_author init --repo ~/repo --id hannesm
@ -176,7 +176,7 @@ repo$ git diff //abbreviated output
 +  ]
 +]
 </code></pre>
-<h3>Status</h3>
+<h3 id="status">Status</h3>
 <p>If you have a single identity and contribute to a single signed opam repository,
 you don't need to specify <code>--id</code> or <code>--repo</code> from now on.</p>
 <p>The <code>status</code> subcommand presents an author-specific view on the repository.  It
@ -199,7 +199,7 @@ authorised to modify (inferred as described
 authorised for.  The <code>--id</code> argument presents you with a view of another author,
 or from a team perspective.  The positional argument is a prefix matching on
 package names (leave empty for all).</p>
-<h3>Resource approval</h3>
+<h3 id="resource-approval">Resource approval</h3>
 <p>Each resource needs to be approved individually.  Each author has a local queue
 for to-be-signed resources, which is extended with <code>authorisation</code>, <code>init</code>,
 <code>key</code>, <code>release</code>, and <code>team</code> (all have a <code>--dry-run</code> flag).  The queue can be
@ -256,7 +256,7 @@ repo$ git add packages/arp/arp.0.1.1/release packages/arp/package
 repo$ git commit -m &quot;hannesm key enrollment and some fixes&quot; id packages
 </code></pre>
 <p>Now push this to your fork, and open a PR on opam-repository!</p>
-<h3>Editing a package</h3>
+<h3 id="editing-a-package">Editing a package</h3>
 <p>If you need to modify a released package, you modify the opam file (as before,
 e.g. introducing a conflict with a dependency), and then approve the
 modifications.  After your local modifications, <code>conex_author status</code> will
@ -285,7 +285,7 @@ added it to your queue.  The <code>sign</code> command signed the approved resou

 repo$ git commit -m &quot;fixed broken arp package&quot; id packages
 </code></pre>
-<h3>Janitor tools</h3>
+<h3 id="janitor-tools">Janitor tools</h3>
 <p>Janitors need to approve teams, keys, accounts, and authorisations.</p>
 <p>To approve resources which are already in the repository on disk,
 the <code>key</code> subcommand queues approval of keys and accounts of the provided author:</p>
@ -301,7 +301,7 @@ authorisations and teams.</p>
 approved by you.  Similar for the <code>key</code> and <code>team</code> subcommands, which also
 accept <code>all</code>.</p>
 <p>Don't forget to <code>conex_author sign</code> afterwards (or <code>yes | conex_author sign</code>).</p>
-<h3>Verification</h3>
+<h3 id="verification">Verification</h3>
 <p>The two command line utlities, <code>conex_verify_openssl</code> and
 <code>conex_verify_nocrypto</code> contain the same logic and same command line arguments.</p>
 <p>For bootstrapping purposes (<code>nocrypto</code> is an opam package with dependencies),
--- a/Posts/DNS
+++ b/Posts/DNS
@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>My 2018 contains robur and starts with re-engineering DNS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="My 2018 contains robur and starts with re-engineering DNS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2018-01-11 (last updated: 2021-11-19)</span><article><h2>2018</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>My 2018 contains robur and starts with re-engineering DNS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="My 2018 contains robur and starts with re-engineering DNS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2018-01-11 (last updated: 2021-11-19)</span><article><h2 id="section">2018</h2>
 <p>At the end of 2017, I resigned from my PostDoc position at University of
 Cambridge (in the <a href="https://www.cl.cam.ac.uk/~pes20/rems/">rems</a> project).  Early
 December 2017 I organised the <a href="https://mirage.io/blog/2017-winter-hackathon-roundup">4th MirageOS hack
@ -46,7 +46,7 @@ coverage is not high enough, and the lack of documentation).  I appreciate early
 adopters, please let me know if you find any issues or find a use case which is
 not straightforward to solve.  This won't be the last article about DNS this
 year - persistent storage, resolver, let's encrypt support are still missing.</p>
-<h2>What is DNS?</h2>
+<h2 id="what-is-dns">What is DNS?</h2>
 <p>The <a href="https://en.wikipedia.org/wiki/DNS">domain name system</a> is a core Internet
 protocol, which translates domain names to IP addresses.  A domain name is
 easier to memorise for human beings than an IP address.  DNS is hierarchical and
@ -79,7 +79,7 @@ transported via TCP, and even via TLS over UDP or TCP.  If a DNS packet
 transferred via UDP is larger than 512 bytes, it is cut at the 512 byte mark,
 and a bit in its header is set.  The receiver can decide whether to use the 512
 bytes of information, or to throw it away and attempt a TCP connection.</p>
-<h3>DNS packet</h3>
+<h3 id="dns-packet">DNS packet</h3>
 <p>The packet encoding starts with a 16bit identifier followed by a 16bit header
 (containing operation, flags, status code), and four counters, each 16bit,
 specifying the amount of resource records in the body: questions, answers,
@ -123,7 +123,7 @@ payload is a 16bit priority and a domain name.  A CNAME record is an alias to
 another domain name.  These days, there are even records to specify the
 certificate authority authorisation (CAA) records containing a flag (critical),
 a tag (&quot;issue&quot;) and a value (&quot;letsencrypt.org&quot;).</p>
-<h2>Server</h2>
+<h2 id="server">Server</h2>
 <p>The operation of a DNS server is to listen for a request and serve a reply.
 Data to be served can be canonically encoded (the RFC describes the format) in a
 zone file.  Apart from insecurity in DNS server implementations, another attack
@ -146,7 +146,7 @@ take time logarithmic in the size of the map&quot;).</p>
 type. The found resource records are sent as answer, which also includes the
 question and authority information (NS records of the zone) and additional glue
 records (IP addresses of names mentioned earlier in the same zone).</p>
-<h3>Dns_map</h3>
+<h3 id="dns_map">Dns_map</h3>
 <p>The data structure which contains resource record types as key, and a collection
 of matching resource records as values.  In OCaml the value type must be
 homogenous - using a normal sum type leads to an unneccessary unpacking step
@ -228,7 +228,7 @@ let get : type a. a Key.t -&gt; t -&gt; a = fun k t -&gt;
 <p>This helps me to programmaticaly retrieve tightly typed values from the cache,
 important when code depends on concrete values (i.e. when there are domain
 names, look these up as well and add as additional records).  Look into <a href="https://github.com/roburio/udns/blob/master/server/dns_server.ml">server/dns_server.ml</a></p>
-<h3>Dynamic updates, notifications, and authentication</h3>
+<h3 id="dynamic-updates-notifications-and-authentication">Dynamic updates, notifications, and authentication</h3>
 <p><a href="https://tools.ietf.org/html/rfc2136">Dynamic updates</a> specify in-protocol
 record updates (supported for example by <code>nsupdate</code> from ISC bind-tools),
 <a href="https://tools.ietf.org/html/rfc1996">notifications</a> are used by primary servers
@ -241,7 +241,7 @@ all protocol extensions, there is no need to use out-of-protocol solutions.</p>
 and includes a dependency upon an authenticator (implemented using the
 <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a> library, and
 <a href="http://erratique.ch/software/ptime">ptime</a>).</p>
-<h3>Deployment and Let's Encrypt</h3>
+<h3 id="deployment-and-lets-encrypt">Deployment and Let's Encrypt</h3>
 <p>To deploy servers without much persistent data, an authentication schema is
 hardcoded in the dns-server: shared secrets are also stored as DNS entries
 (DNSKEY), and <code>_transfer.zone</code>, <code>_update.zone</code>, and <code>_key-management.zone</code> names
--- a/Posts/Deploy
+++ b/Posts/Deploy
@ -1,16 +1,16 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying binary MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying binary MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2021-06-30 (last updated: 2021-11-15)</span><article><h2>Introduction</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying binary MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying binary MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2021-06-30 (last updated: 2021-11-15)</span><article><h2 id="introduction">Introduction</h2>
 <p>MirageOS development focus has been a lot on tooling and the developer experience, but to accomplish <a href="https://robur.coop">our</a> goal to &quot;get MirageOS into production&quot;, we need to lower the barrier. This means for us to release binary unikernels. As described <a href="/Posts/NGI">earlier</a>, we received a grant for &quot;Deploying MirageOS&quot; from <a href="https://pointer.ngi.eu">NGI Pointer</a> to work on the required infrastructure. This is joint work with <a href="https://reynir.dk/">Reynir</a>.</p>
 <p>We provide at <a href="https://builds.robur.coop">builds.robur.coop</a> binary unikernel images (and supplementary software). Doing binary releases of MirageOS unikernels is challenging in two aspects: firstly to be useful for everyone, a binary unikernel should not contain any configuration (such as private keys, certificates, etc.). Secondly, the binaries should be <a href="https://reproducible-builds.org">reproducible</a>. This is crucial for security; everyone can reproduce the exact same binary and verify that our build service did only use the sources. No malware or backdoors included.</p>
 <p>This post describes how you can deploy MirageOS unikernels without compiling it from source, then dives into the two issues outlined above - configuration and reproducibility - and finally describes how to setup your own reproducible build infrastructure for MirageOS, and how to bootstrap it.</p>
-<h2>Deploying MirageOS unikernels from binary</h2>
+<h2 id="deploying-mirageos-unikernels-from-binary">Deploying MirageOS unikernels from binary</h2>
 <p>To execute a MirageOS unikernel, apart from a hypervisor (Xen/KVM/Muen), a tender (responsible for allocating host system resources and passing these to the unikernel) is needed. Using virtio, this is conventionally done with qemu on Linux, but its code size (and attack surface) is huge. For MirageOS, we develop <a href="https://github.com/solo5/solo5">Solo5</a>, a minimal tender. It supports <em>hvt</em> - hardware virtualization (Linux KVM, FreeBSD BHyve, OpenBSD VMM), <em>spt</em> - sandboxed process (a tight seccomp ruleset (only a handful of system calls allowed, no hardware virtualization needed), Linux only). Apart from that, <a href="https://muen.sk"><em>muen</em></a> (a hypervisor developed in Ada), <em>virtio</em> (for some cloud deployments), and <em>xen</em> (PVHv2 or Qubes 4.0) - <a href="https://github.com/Solo5/solo5/blob/master/docs/building.md">read more</a>. We deploy our unikernels as hvt with FreeBSD BHyve as hypervisor.</p>
 <p>On <a href="https://builds.robur.coop">builds.robur.coop</a>, next to the unikernel images, <a href="https://builds.robur.coop/job/solo5-hvt/"><em>solo5-hvt</em> packages</a> are provided - download the binary and install it. A <a href="https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/solo5">NixOS package</a> is already available - please note that <a href="https://github.com/Solo5/solo5/pull/494">soon</a> packaging will be much easier (and we will work on packages merged into distributions).</p>
 <p>When the tender is installed, download a unikernel image (e.g. the <a href="https://builds.robur.coop/job/traceroute/build/latest/">traceroute</a> described in <a href="/Posts/Traceroute">an earlier post</a>), and execute it:</p>
 <pre><code>$ solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1
 </code></pre>
 <p>If you plan to orchestrate MirageOS unikernels, you may be interested in <a href="https://github.com/roburio/albatross">albatross</a> - we provide <a href="https://builds.robur.coop/job/albatross/">binary packages as well for albatross</a>. An upcoming post will go into further details of how to setup albatross.</p>
-<h2>MirageOS configuration</h2>
+<h2 id="mirageos-configuration">MirageOS configuration</h2>
 <p>A MirageOS unikernel has a specific purpose - composed of OCaml libraries - selected at compile time, which allows to only embed the required pieces. This reduces the attack surface drastically. At the same time, to be widely useful to multiple organisations, no configuration data must be embedded into the unikernel.</p>
 <p>Early MirageOS unikernels such as <a href="https://github.com/mirage/mirage-www">mirage-www</a> embed content (blog posts, ..) and TLS certificates and private keys in the binary (using <a href="https://github.com/mirage/ocaml-crunch">crunch</a>). The <a href="https://github.com/mirage/qubes-mirage-firewall">Qubes firewall</a> (read the <a href="http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/">blog post by Thomas</a> for more information) used to include the firewall rules until <a href="https://github.com/mirage/qubes-mirage-firewall/releases/tag/v0.6">v0.6</a> in the binary, since <a href="https://github.com/mirage/qubes-mirage-firewall/tree/v0.7">v0.7</a> the rules are read dynamically from QubesDB. This is big usability improvement.</p>
 <p>We have several possibilities to provide configuration information in MirageOS, on the one hand via boot parameters (can be pre-filled at development time, and further refined at configuration time, but those passed at boot time take precedence). Boot parameters have a length limitation.</p>
@ -18,7 +18,7 @@
 <p>Several other unikernels, such as <a href="https://github.com/Engil/Canopy">this website</a> and <a href="https://github.com/roburio/caldav">our CalDAV server</a>, store the content in a remote git repository. The git URI and credentials (private key seed, host key fingerprint) are passed via boot parameter.</p>
 <p>Finally, another option that we take advantage of is to introduce a post-link step that rewrites the binary to embed configuration. The tool <a href="https://github.com/dinosaure/caravan">caravan</a> developed by Romain that does this rewrite is used by our <a href="https://github.com/roburio/openvpn/tree/robur/mirage-router">openvpn router</a> (<a href="https://builds.robur.coop/job/openvpn-router/build/latest/">binary</a>).</p>
 <p>In the future, some configuration information - such as monitoring system, syslog sink, IP addresses - may be done via DHCP on one of the private network interfaces - this would mean that the DHCP server has some global configuration option, and the unikernels no longer require that many boot parameters. Another option we want to investigate is where the tender shares a file as read-only memory-mapped region from the host system to the guest system - but this is tricky considering all targets above (especially virtio and muen).</p>
-<h2>Behind the scenes: reproducible builds</h2>
+<h2 id="behind-the-scenes-reproducible-builds">Behind the scenes: reproducible builds</h2>
 <p>To provide a high level of assurance and trust, if you distribute binaries in 2021, you should have a recipe how they can be reproduced in a bit-by-bit identical way. This way, different organisations can run builders and rebuilders, and a user can decide to only use a binary if it has been reproduced by multiple organisations in different jurisdictions using different physical machines - to avoid malware being embedded in the binary.</p>
 <p>For a reproduction to be successful, you need to collect the checksums of all sources that contributed to the built, together with other things (host system packages, environment variables, etc.). Of course, you can record the entire OS and sources as a tarball (or file system snapshot) and distribute that - but this may be suboptimal in terms of bandwidth requirements.</p>
 <p>With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the <code>opam switch export</code> includes <a href="https://github.com/ocaml/opam/pull/4040">extra-files (patches)</a> and <a href="https://github.com/ocaml/opam/pull/4055">records the VCS version</a>. Based on this functionality, <a href="https://github.com/roburio/orb">orb</a>, an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are <code>PATH</code> (sanitized with an allow list, <code>/bin</code>, <code>/sbin</code>, with <code>/usr</code>, <code>/usr/local</code>, and <code>/opt</code> prefixes), and <code>HOME</code>. To enable Debian's <code>apt</code> to install packages, <code>DEBIAN_FRONTEND</code> is set to <code>noninteractive</code>. The <code>SWITCH_PATH</code> is recorded to allow orb to use the same path during a rebuild. The <code>SOURCE_DATE_EPOCH</code> is set to enable tools that record a timestamp to use a static one. The <code>OS*</code> variables are only used for recording the host OS and version.</p>
@ -51,7 +51,7 @@
 </li>
 </ul>
 <p>These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>Thanks to NGI funding we now have reproducible MirageOS binary builds available at <a href="https://builds.robur.coop">builds.robur.coop</a>. The underlying infrastructure is reproducible, available for multiple platforms (Ubuntu using docker, FreeBSD using jails), and can be easily bootstrapped from source (once you have OCaml and opam working, getting builder and orb should be easy). All components are open source software, mostly with permissive licenses.</p>
 <p>We also have an index over sha-256 checksum of binaries - in the case you find a running unikernel image where you forgot which exact packages were used, you can do a reverse lookup.</p>
 <p>We are aware that the web interface can be improved (PRs welcome). We will also work on the rebuilder setup and run some rebuilds.</p>
--- a/Posts/DnsServer
+++ b/Posts/DnsServer
@ -1,13 +1,13 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying authoritative OCaml-DNS servers as MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying authoritative OCaml-DNS servers as MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2019-12-23 (last updated: 2023-03-02)</span><article><h2>Goal</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying authoritative OCaml-DNS servers as MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying authoritative OCaml-DNS servers as MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2019-12-23 (last updated: 2023-03-02)</span><article><h2 id="goal">Goal</h2>
 <p>Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as <code>nqsb.io</code> and <code>robur.coop</code>. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as <a href="/Posts/DNS">µDNS</a>.</p>
-<h2>Prerequisites</h2>
+<h2 id="prerequisites">Prerequisites</h2>
 <p>You need to own a domain, and be able to delegate the name service to your own servers.
 You also need two spare public IPv4 addresses (in different /24 networks) for your name servers.
 A git server or remote repository reachable via git over ssh.
 Servers which support <a href="https://github.com/solo5/solo5">solo5</a> guests, and have the corresponding tender installed.
 A computer with <a href="https://opam.ocaml.org">opam</a> (&gt;= 2.0.0) installed.</p>
-<h2>Data preparation</h2>
+<h2 id="data-preparation">Data preparation</h2>
 <p>Figure out a way to get the DNS entries of your domain in a <a href="https://tools.ietf.org/html/rfc1034">&quot;master file format&quot;</a>, i.e. what bind uses.</p>
 <p>This is a master file for the <code>mirage</code> domain, defining <code>$ORIGIN</code> to avoid typing the domain name after each hostname (use <code>@</code> if you need the domain name only; if you need to refer to a hostname in a different domain end it with a dot (<code>.</code>), i.e. <code>ns2.foo.com.</code>). The default time to live <code>$TTL</code> is an hour (3600 seconds).
 The zone contains a <a href="https://tools.ietf.org/html/rfc1035#section-3.3.13">start of authority (<code>SOA</code>) record</a> containing the nameserver, hostmaster, serial, refresh, retry, expiry, and minimum.
@ -21,7 +21,7 @@ ns1     A       127.0.0.1
 www	A	1.1.1.1
 git-repo&gt; git add mirage &amp;&amp; git commit -m initial &amp;&amp; git push
 </code></pre>
-<h2>Installation</h2>
+<h2 id="installation">Installation</h2>
 <p>On your development machine, you need to install various OCaml packages. You don't need privileged access if common tools (C compiler, make, libgmp) are already installed. You have <code>opam</code> installed.</p>
 <p>Let's create a fresh <code>switch</code> for the DNS journey:</p>
 <pre><code class="language-shell">$ opam init
@ -31,7 +31,7 @@ $ opam switch create udns 4.14.1
 $ eval `opam env` #sets some environment variables
 </code></pre>
 <p>The last command set environment variables in your current shell session, please use the same shell for the commands following (or run <code>eval $(opam env)</code> in another shell and proceed in there - the output of <code>opam switch</code> sohuld point to <code>udns</code>).</p>
-<h3>Validation of our zonefile</h3>
+<h3 id="validation-of-our-zonefile">Validation of our zonefile</h3>
 <p>First let's check that OCaml-DNS can parse our zonefile:</p>
 <pre><code class="language-shell">$ opam install dns-cli #installs ~/.opam/udns/bin/ozone and other binaries
 $ ozone &lt;git-repo&gt;/mirage # see ozone --help
@ -39,7 +39,7 @@ successfully checked zone
 </code></pre>
 <p>Great. Error reporting is not great, but line numbers are indicated (<code>ozone: zone parse problem at line 3: syntax error</code>), <a href="https://github.com/mirage/ocaml-dns/tree/v4.2.0/zone">lexer and parser are lex/yacc style</a> (PRs welcome).</p>
 <p>FWIW, <code>ozone</code> accepts <code>--old &lt;filename&gt;</code> to check whether an update from the old zone to the new is fine. This can be used as <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">pre-commit hook</a> in your git repository to avoid bad parse states in your name servers.</p>
-<h3>Getting the primary up</h3>
+<h3 id="getting-the-primary-up">Getting the primary up</h3>
 <p>The next step is to compile the primary server and run it to serve the domain data. Since the git-via-ssh client is not yet released, we need to add a custom opam repository to this switch.</p>
 <pre><code class="language-shell"># get the `mirage` application via opam
 $ opam install lwt mirage
@ -70,7 +70,7 @@ $ dig any mirage @127.0.0.1
 # a DNS packet printout with all records available for mirage
 </code></pre>
 <p>That's exciting, the DNS server serving answers from a remote git repository.</p>
-<h3>Securing the git access with ssh</h3>
+<h3 id="securing-the-git-access-with-ssh">Securing the git access with ssh</h3>
 <p>Let's authenticate the access by using ssh, so we feel ready to push data there as well. The primary-git unikernel already includes an experimental <a href="https://github.com/haesbaert/awa-ssh">ssh client</a>, all we need to do is setting up credentials - in the following a RSA keypair and the server fingerprint.</p>
 <pre><code class="language-shell"># collect the RSA host key fingerprint
 $ ssh-keyscan &lt;git-server&gt; &gt; /tmp/git-server-public-keys
@ -91,7 +91,7 @@ $ ./primary-git --authenticator=SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBa
 # started up, you can try the host and dig commands from above if you like
 </code></pre>
 <p>To wrap up, we now have a primary authoritative name server for our zone running as Unix process, which clones a remote git repository via ssh on startup and then serves it.</p>
-<h3>Authenticated data updates</h3>
+<h3 id="authenticated-data-updates">Authenticated data updates</h3>
 <p>Our remote git repository is the source of truth, if you need to add a DNS entry to the zone, you git pull, edit the zone file, remember to increase the serial in the SOA line, run <code>ozone</code>, git commit and push to the repository.</p>
 <p>So, the <code>primary-git</code> needs to be informed of git pushes. This requires a communication channel from the git server (or somewhere else, e.g. your laptop) to the DNS server. I prefer in-protocol solutions over adding yet another protocol stack, no way my DNS server will talk HTTP REST.</p>
 <p>The DNS protocol has an extension for <a href="https://tools.ietf.org/html/rfc1996">notifications of zone changes</a> (as a DNS packet), usually used between the primary and secondary servers. The <code>primary-git</code> accepts these notify requests (i.e. bends the standard slightly), and upon receival pulls the remote git repository, and serves the fresh zone files. Since a git pull may be rather excessive in terms of CPU cycles and network bandwidth, only authenticated notifications are accepted.</p>
@ -117,7 +117,7 @@ $ onotify 127.0.0.1 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31
 # further changes to the hmac secrets don't require a restart anymore, a notify packet is sufficient :D
 </code></pre>
 <p>Ok, this onotify command line could be setup as a git post-commit hook, or run manually after each manual git push.</p>
-<h3>Secondary</h3>
+<h3 id="secondary">Secondary</h3>
 <p>It's time to figure out how to integrate the secondary name server. An already existing bind or something else that accepts notifications and issues zone transfers with hmac-sha256 secrets should work out of the box. If you encounter interoperability issues, please get in touch with me.</p>
 <p>The <code>secondary</code> unikernel is available from another git repository:</p>
 <pre><code class="language-shell"># get the secondary sources
@ -130,7 +130,7 @@ $ cd dns-secondary
 $ make
 $ ./dist/secondary
 </code></pre>
-<h3>IP addresses and routing</h3>
+<h3 id="ip-addresses-and-routing">IP addresses and routing</h3>
 <p>Both primary and secondary serve the data on the DNS port (53) on UDP and TCP. To run both on the same machine and bind them to different IP addresses, we'll use a layer 2 network (ethernet frames) with a host system software switch (bridge interface <code>service</code>), the unikernels as virtual machines (or seccomp-sandboxed) via the <a href="https://github.com/solo5/solo5">solo5</a> backend. Using xen is possible as well. As IP address range we'll use 10.0.42.0/24, and the host system uses the 10.0.42.1.</p>
 <p>The primary git needs connectivity to the remote git repository, thus on a laptop in a private network we need network address translation (NAT) from the bridge where the unikernels speak to the Internet where the git repository resides.</p>
 <pre><code class="language-shell"># on FreeBSD:
@ -156,7 +156,7 @@ tap1
 # add them to the bridge
 $ ifconfig service addm tap0 addm tap1
 </code></pre>
-<h3>Primary and secondary setup</h3>
+<h3 id="primary-and-secondary-setup">Primary and secondary setup</h3>
 <p>Let's update our zone slightly to reflect the IP changes.</p>
 <pre><code class="language-shell">git-repo&gt; cat mirage
 $ORIGIN mirage.
@ -208,7 +208,7 @@ $ dig foo.mirage @10.0.42.2 # primary
 $ dig foo.mirage @10.0.42.3 # secondary got notified and transferred the zone
 </code></pre>
 <p>You can also check the behaviour when restarting either of the VMs, whenever the primary is available the zone is synchronised. If the primary is down, the secondary still serves the zone. When the secondary is started while the primary is down, it won't serve any data until the primary is online (the secondary polls periodically, the primary sends notifies on startup).</p>
-<h3>Dynamic data updates via DNS, pushed to git</h3>
+<h3 id="dynamic-data-updates-via-dns-pushed-to-git">Dynamic data updates via DNS, pushed to git</h3>
 <p>DNS is a rich protocol, and it also has builtin <a href="https://tools.ietf.org/html/rfc2136">updates</a> that are supported by OCaml DNS, again authenticated with hmac-sha256 and shared secrets. Bind provides the command-line utility <code>nsupdate</code> to send these update packets, a simple <code>oupdate</code> unix utility is available as well (i.e. for integration of dynamic DNS clients). You know the drill, add a shared secret to the primary, git push, notify the primary, and voila we can dynamically in-protocol update. An update received by the primary via this way will trigger a git push to the remote git repository, and notifications to the secondary servers as described above.</p>
 <pre><code class="language-shell"># being lazy, I reuse the key above
 $ oupdate 10.0.42.2 personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg= my-other.mirage 1.2.3.4
@ -223,7 +223,7 @@ $ dig my-other.mirage @10.0.42.2
 $ dig my-other.mirage @10.0.42.3
 </code></pre>
 <p>So we can deploy further <code>oupdate</code> (or <code>nsupdate</code>) clients, distribute hmac secrets, and have the DNS zone updated. The source of truth is still the git repository, where the primary-git pushes to. Merge conflicts and timing of pushes is not yet dealt with. They are unlikely to happen since the primary is notified on pushes and should have up-to-date data in storage. Sorry, I'm unsure about the error semantics, try it yourself.</p>
-<h3>Let's encrypt!</h3>
+<h3 id="lets-encrypt">Let's encrypt!</h3>
 <p><a href="https://letsencrypt.org/">Let's encrypt</a> is a certificate authority (CA), which certificate is shipped as trust anchor in web browsers. They specified a protocol for <a href="https://tools.ietf.org/html/draft-ietf-acme-acme-05">automated certificate management environment (ACME)</a>, used to get X509 certificates for your services. In the protocol, a certificate signing request (publickey and hostname) is sent to let's encrypt servers, which sends a challenge to proof the ownership of the hostnames. One widely-used way to solve this challenge is running a web server, another is to serve it as text record from the authoritative DNS server.</p>
 <p>Since I avoid persistent storage when possible, and also don't want to integrate a HTTP client stack in the primary server, I developed a third unikernel that acts as (hidden) secondary server, performs the tedious HTTP communication with let's encrypt servers, and stores all data in the public DNS zone.</p>
 <p>For encoding of certificates, the DANE working group specified <a href="https://tools.ietf.org/html/rfc6698.html#section-7.1">TLSA</a> records in DNS. They are quadruples of usage, selector, matching type, and ASN.1 DER-encoded material. We set usage to 3 (domain-issued certificate), matching type to 0 (no hash), and selector to 0 (full certificate) or 255 (private usage) for certificate signing requests. The interaction is as follows:</p>
@ -268,7 +268,7 @@ $ ocertify 10.0.42.2 foo.mirage
 </code></pre>
 <p>For actual testing with let's encrypt servers you need to have the primary and secondary deployed on your remote hosts, and your domain needs to be delegated to these servers. Good luck. And ensure you have backup your git repository.</p>
 <p>As fine print, while this tutorial was about the <code>mirage</code> zone, you can stick any number of zones into the git repository. If you use a <code>_keys</code> file (without any domain prefix), you can configure hmac secrets for all zones, i.e. something to use in your let's encrypt unikernel and secondary unikernel. Dynamic addition of zones is supported, just create a new zonefile and notify the primary, the secondary will be notified and pick it up. The primary responds to a signed SOA for the root zone (i.e. requested by the secondary) with the SOA response (not authoritative), and additionally notifications for all domains of the primary.</p>
-<h3>Conclusion and thanks</h3>
+<h3 id="conclusion-and-thanks">Conclusion and thanks</h3>
 <p>This tutorial presented how to use the OCaml DNS based unikernels to run authoritative name servers for your domain, using a git repository as the source of truth, dynamic authenticated updates, and let's encrypt certificate issuing.</p>
 <p>There are further steps to take, such as monitoring (<code>mirage configure --monitoring</code>), which use a second network interface for reporting syslog and metrics to telegraf / influx / grafana. Some DNS features are still missing, most prominently DNSSec.</p>
 <p>I'd like to thank all people involved in this software stack, without other key components, including <a href="https://github.com/mirage/ocaml-git">git</a>, <a href="https://github.com/mirage/mirage-crypto">mirage-crypto</a>, <a href="https://github.com/mirage/awa-ssh">awa-ssh</a>, <a href="https://github.com/solo5/sol5">solo5</a>, <a href="https://github.com/mirage/mirage">mirage</a>, <a href="https://github.com/mmaker/ocaml-letsencrypt">ocaml-letsencrypt</a>, and more.</p>
--- a/Posts/EC
+++ b/Posts/EC
@ -1,7 +1,7 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Cryptography updates in OCaml and MirageOS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Cryptography updates in OCaml and MirageOS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/tls" class="tag">tls</a></div><span class="date">Published: 2021-04-23 (last updated: 2021-11-19)</span><article><h2>Introduction</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Cryptography updates in OCaml and MirageOS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Cryptography updates in OCaml and MirageOS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/tls" class="tag">tls</a></div><span class="date">Published: 2021-04-23 (last updated: 2021-11-19)</span><article><h2 id="introduction">Introduction</h2>
 <p>Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a> stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.</p>
-<h2>Elliptic curve cryptography</h2>
+<h2 id="elliptic-curve-cryptography">Elliptic curve cryptography</h2>
 <p><a href="https://mirage.io/blog/tls-1-3-mirageos">Since May 2020</a>, our <a href="https://usenix15.nqsb.io">OCaml-TLS</a> stack supports TLS 1.3 (since tls version 0.12.0 on opam).</p>
 <p>TLS 1.3 requires elliptic curve cryptography - which was not available in <a href="https://github.com/mirage/mirage-crypto">mirage-crypto</a> (the maintained fork of <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a>).</p>
 <p>There are two major uses of elliptic curves: <a href="https://en.wikipedia.org/wiki/Elliptic-curve_Diffie%E2%80%93Hellman">key exchange (ECDH)</a> for establishing a shared secret over an insecure channel, and <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm">digital signature (ECDSA)</a> for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)</p>
@ -9,37 +9,37 @@
 <p>In addition, to use the code in MirageOS, it should be boring C code: no heap allocations, only using a very small amount of C library functions -- the code needs to be compiled in an environment with <a href="https://github.com/mirage/ocaml-freestanding/tree/v0.6.4/nolibc">nolibc</a>.</p>
 <p>Two projects started in semantics, to solve the issue from the grounds up: <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a> and <a href="https://github.com/project-everest/hacl-star/">hacl-star</a>: their approach is to use a proof system (<a href="https://coq.inria.fr">Coq</a> or <a href="https://www.fstar-lang.org/">F*</a> to verify that the code executes in constant time, not depending on data input. Both projects provide as output of their proof systems C code.</p>
 <p>For our initial TLS 1.3 stack, <a href="https://github.com/pascutto/">Clément</a>, <a href="https://github.com/NathanReb/">Nathan</a> and <a href="https://github.com/emillon/">Etienne</a> developed <a href="https://github.com/mirage/fiat">fiat-p256</a> and <a href="https://github.com/mirage/hacl">hacl_x5519</a>. Both were one-shot interfaces for a narrow use case (ECDH for NIST P-256 and X25519), worked well for their purpose, and allowed to gather some experience from the development side.</p>
-<h3>Changed requirements</h3>
+<h3 id="changed-requirements">Changed requirements</h3>
 <p>Revisiting our cryptography stack with the elliptic curve perspective had several reasons, on the one side the customer project <a href="https://www.nitrokey.com/products/nethsm">NetHSM</a> asked for feasibility of ECDSA/EdDSA for various elliptic curves, on the other side <a href="https://github.com/mirage/ocaml-dns/pull/251">DNSSec</a> uses elliptic curve cryptography (ECDSA), and also <a href="https://www.wireguard.com/">wireguard</a> relies on elliptic curve cryptography. The number of X.509 certificates using elliptic curves is increasing, and we don't want to leave our TLS stack in a state where it can barely talk to a growing number of services on the Internet.</p>
 <p>Looking at <a href="https://github.com/project-everest/hacl-star/"><em>hacl-star</em></a>, their <a href="https://hacl-star.github.io/Supported.html">support</a> is limited to P-256 and Curve25519, any new curve requires writing F*. Another issue with hacl-star is C code quality: their C code does neither <a href="https://github.com/mirage/hacl/issues/46">compile with older C compilers (found on Oracle Linux 7 / CentOS 7)</a>, nor when enabling all warnings (&gt; 150 are generated). We consider the C compiler as useful resource to figure out undefined behaviour (and other problems), and when shipping C code we ensure that it compiles with <code>-Wall -Wextra -Wpedantic --std=c99 -Werror</code>. The hacl project <a href="https://github.com/mirage/hacl/tree/master/src/kremlin">ships</a> a bunch of header files and helper functions to work on all platforms, which is a clunky <code>ifdef</code> desert. The hacl approach is to generate a whole algorithm solution: from arithmetic primitives, group operations, up to cryptographic protocol - everything included.</p>
 <p>In contrast, <a href="https://github.com/mit-plv/fiat-crypto/"><em>fiat-crypto</em></a> is a Coq development, which as part of compilation (proof verification) generates executables (via OCaml code extraction from Coq). These executables are used to generate modular arithmetic (as C code) given a curve description. The <a href="https://github.com/mirage/mirage-crypto/tree/main/ec/native">generated C code</a> is highly portable, independent of platform (word size is taken as input) - it only requires a <code>&lt;stdint.h&gt;</code>, and compiles with all warnings enabled (once <a href="https://github.com/mit-plv/fiat-crypto/pull/906">a minor PR</a> got merged). Supporting a new curve is simple: generate the arithmetic code using fiat-crypto with the new curve description. The downside is that group operations and protocol needs to implemented elsewhere (and is not part of the proven code) - gladly this is pretty straightforward to do, especially in high-level languages.</p>
-<h3>Working with fiat-crypto</h3>
+<h3 id="working-with-fiat-crypto">Working with fiat-crypto</h3>
 <p>As mentioned, our initial <a href="https://github.com/mirage/fiat">fiat-p256</a> binding provided ECDH for the NIST P-256 curve. Also, BoringSSL uses fiat-crypto for ECDH, and developed the code for group operations and cryptographic protocol on top of it.</p>
 <p>The work needed was (a) ECDSA support and (b) supporting more curves (let's focus on NIST curves). For ECDSA, the algorithm requires modular arithmetics in the field of the group order (in addition to the prime). We generate these primitives with fiat-crypto (named <code>npYYY_AA</code>) - that required <a href="https://github.com/mit-plv/fiat-crypto/commit/e31a36d5f1b20134e67ccc5339d88f0ff3cb0f86">a small fix in decoding hex</a>. Fiat-crypto also provides inversion <a href="https://github.com/mit-plv/fiat-crypto/pull/670">since late October 2020</a>, <a href="https://eprint.iacr.org/2021/549">paper</a> - which allowed to reduce our code base taken from BoringSSL. The ECDSA protocol was easy to implement in OCaml using the generated arithmetics.</p>
 <p>Addressing the issue of more curves was also easy to achieve, the C code (group operations) are macros that are instantiated for each curve - the OCaml code are functors that are applied with each curve description.</p>
 <p>Thanks to the test vectors (as structured data) from <a href="https://github.com/google/wycheproof/">wycheproof</a> (and again thanks to Etienne, Nathan, and Clément for their OCaml code decodin them), I feel confident that our elliptic curve code works as desired.</p>
 <p>What was left is X25519 and Ed25519 - dropping the hacl dependency entirely felt appealing (less C code to maintain from fewer projects). This turned out to require more C code, which we took from BoringSSL. It may be desirable to reduce the imported C code, or to wait until a project on top of fiat-crypto which provides proven cryptographic protocols is in a usable state.</p>
 <p>To avoid performance degradation, I distilled some <a href="https://github.com/mirage/mirage-crypto/pull/107#issuecomment-799701703">X25519 benchmarks</a>, turns out the fiat-crypto and hacl performance is very similar.</p>
-<h3>Achievements</h3>
+<h3 id="achievements">Achievements</h3>
 <p>The new opam package <a href="https://mirage.github.io/mirage-crypto/doc/mirage-crypto-ec/Mirage_crypto_ec/index.html">mirage-crypto-ec</a> is released, which includes the C code generated by fiat-crypto (including <a href="https://github.com/mit-plv/fiat-crypto/pull/670">inversion</a>), <a href="https://github.com/mirage/mirage-crypto/blob/main/ec/native/point_operations.h">point operations</a> from BoringSSL, and some <a href="https://github.com/mirage/mirage-crypto/blob/main/ec/mirage_crypto_ec.ml">OCaml code</a> for invoking these functions and doing bounds checks, and whether points are on the curve. The OCaml code are some functors that take the curve description (consisting of parameters, C function names, byte length of value) and provide Diffie-Hellman (Dh) and digital signature algorithm (Dsa) modules. The nonce for ECDSA is computed deterministically, as suggested by <a href="https://tools.ietf.org/html/rfc6979">RFC 6979</a>, to avoid private key leakage.</p>
 <p>The code has been developed in <a href="https://github.com/mirage/mirage-crypto/pull/101">NIST curves</a>, <a href="https://github.com/mirage/mirage-crypto/pull/106">removing blinding</a> (since we use operations that are verified to be constant-time), <a href="https://github.com/mirage/mirage-crypto/pull/108">added missing length checks</a> (reported by <a href="https://github.com/greg42">Greg</a>), <a href="https://github.com/mirage/mirage-crypto/pull/107">curve25519</a>, <a href="https://github.com/mirage/mirage-crypto/pull/117">a fix for signatures that do not span the entire byte size (discovered while adapting X.509)</a>, <a href="https://github.com/mirage/mirage-crypto/pull/118">fix X25519 when the input has offset &lt;&gt; 0</a>. It works on x86 and arm, both 32 and 64 bit (checked by CI). The development was partially sponsored by Nitrokey.</p>
 <p>What is left to do, apart from further security reviews, is <a href="https://github.com/mirage/mirage-crypto/issues/109">performance improvements</a>, <a href="https://github.com/mirage/mirage-crypto/issues/112">Ed448/X448 support</a>, and <a href="https://github.com/mirage/mirage-crypto/issues/105">investigating deterministic k for P521</a>. Pull requests are welcome.</p>
 <p>When you use the code, and encounter any issues, please <a href="https://github.com/mirage/mirage-crypto/issues">report them</a>.</p>
-<h2>Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed</h2>
+<h2 id="layer-up---x.509-now-with-ecdsa-eddsa-and-pkcs-12-support-and-a-long-standing-issue-fixed">Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed</h2>
 <p>With the sign and verify primitives, the next step is to interoperate with other tools that generate and use these public and private keys. This consists of serialisation to and deserialisation from common data formats (ASN.1 DER and PEM encoding), and support for handling X.509 certificates with elliptic curve keys. Since X.509 0.12.0, it supports EC private and public keys, including certificate validation and issuance.</p>
 <p>Releasing X.509 also included to go through the issue tracker and attempt to solve the existing issues. This time, the <a href="https://github.com/mirleft/ocaml-x509/issues/69">&quot;country name is encoded as UTF8String, while RFC demands PrintableString&quot;</a> filed more than 5 years ago by <a href="https://github.com/reynir">Reynir</a>, re-reported by <a href="https://github.com/paurkedal">Petter</a> in early 2017, and again by <a href="https://github.com/NightBlues">Vadim</a> in late 2020, <a href="https://github.com/mirleft/ocaml-x509/pull/140">was fixed by Vadim</a>.</p>
 <p>Another long-standing pull request was support for <a href="https://tools.ietf.org/html/rfc7292">PKCS 12</a>, the archive format for certificate and private key bundles. This has <a href="https://github.com/mirleft/ocaml-x509/pull/114">been developed and merged</a>. PKCS 12 is a widely used and old format (e.g. when importing / exporting cryptographic material in your browser, used by OpenVPN, ...). Its specification uses RC2 and 3DES (see <a href="https://unmitigatedrisk.com/?p=654">this nice article</a>), which are the default algorithms used by <code>openssl pkcs12</code>.</p>
-<h2>One more layer up - TLS</h2>
+<h2 id="one-more-layer-up---tls">One more layer up - TLS</h2>
 <p>In TLS we are finally able to use ECDSA (and EdDSA) certificates and private keys, this resulted in slightly more complex configuration - the constraints between supported groups, signature algorithms, ciphersuite, and certificates are intricate:</p>
 <p>The ciphersuite (in TLS before 1.3) specifies which key exchange mechanism to use, but also which signature algorithm to use (RSA/ECDSA). The supported groups client hello extension specifies which elliptic curves are supported by the client. The signature algorithm hello extension (TLS 1.2 and above) specifies the signature algorithm. In the end, at load time the TLS configuration is validated and groups, ciphersuites, and signature algorithms are condensed depending on configured server certificates. At session initiation time, once the client reports what it supports, these parameters are further cut down to eventually find some suitable cryptographic parameters for this session.</p>
 <p>From the user perspective, earlier the certificate bundle and private key was a pair of <code>X509.Certificate.t list</code> and <code>Mirage_crypto_pk.Rsa.priv</code>, now the second part is a <code>X509.Private_key.t</code> - all provided constructors have been updates (notably <code>X509_lwt.private_of_pems</code> and <code>Tls_mirage.X509.certificate</code>).</p>
-<h2>Finally, conduit and mirage</h2>
+<h2 id="finally-conduit-and-mirage">Finally, conduit and mirage</h2>
 <p>Thanks to <a href="https://github.com/dinosaure">Romain</a>, conduit* 4.0.0 was released which supports the modified API of X.509 and TLS. Romain also developed patches and released mirage 3.10.3 which supports the above mentioned work.</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>Elliptic curve cryptography is now available in OCaml using verified cryptographic primitives from the fiat-crypto project - <code>opam install mirage-crypto-ec</code>. X.509 since 0.12.0 and TLS since 0.13.0 and MirageOS since 3.10.3 support this new development which gives rise to smaller EC keys. Our old bindings, fiat-p256 and hacl_x25519 have been archived and will no longer be maintained.</p>
 <p>Thanks to everyone involved on this journey: reporting issues, sponsoring parts of the work, helping with integration, developing initial prototypes, and keep motivating me to continue this until the release is done.</p>
 <p>In the future, it may be possible to remove zarith and gmp from the dependency chain, and provide EC-only TLS servers and clients for MirageOS. The benefit will be much less C code (libgmp-freestanding.a is 1.5MB in size) in our trusted code base.</p>
 <p>Another potential project that is very close now is a certificate authority developed in MirageOS - now that EC keys, PKCS 12, revocation lists, ... are implemented.</p>
-<h2>Footer</h2>
+<h2 id="footer">Footer</h2>
 <p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
 </article></div></div></main></body></html>
--- a/Posts/Functoria
+++ b/Posts/Functoria
@ -1,12 +1,12 @@
 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Configuration DSL step-by-step</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Configuration DSL step-by-step" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Configuration DSL step-by-step</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-05-10 (last updated: 2021-11-19)</span><article><p>Sorry for being late again with this article, I had other ones planned, but am not yet satisfied with content and code, will have to wait another week.</p>
-<h2>MirageOS configuration</h2>
+<h2 id="mirageos-configuration">MirageOS configuration</h2>
 <p>As described in an <a href="/Posts/OperatingSystem">earlier post</a>, MirageOS is a library operating system which generates single address space custom kernels (so called unikernels) for each application.  The application code is (mostly) independent on the used backend.  To achieve this, the language which expresses the configuration of a MirageOS unikernel is rather complex, and has to deal with package dependencies, setup of layers (network stack starting at the (virtual) ethernet device, or sockets), logging, tracing.</p>
 <p>The abstraction over concrete implementation of e.g. the network stack is done by providing a module signature in the <a href="https://github.com/mirage/mirage/tree/master/types">mirage-types</a> package.  The socket-based network stack, the tap device based network stack, and the Xen virtual network device based network stack implement this signature (depending on other module signatures).  The unikernel contains code which applies those dependent modules to instantiate a custom-tailored network stack for the specific configuration.  A developer should only describe what their requirements are, the user who wants to deploy it should provide the concrete configuration.  And the developer should not need to manually instantiate the network stack for all possible configurations, this is what the mirage tool should embed.</p>
 <p>Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code.  This turned out to be error prone.  In 2015 <a href="https://github.com/Drup">Drup</a> developed <a href="https://github.com/mirage/functoria">Functoria</a>, a domain-specific language (DSL) to organize functor applications, primarily for MirageOS.  It has been introduced in <a href="https://mirage.io/blog/introducing-functoria">a blog post</a>.  It is not limited to MirageOS (although this is the primary user right now).</p>
 <p>Functoria has been included in MirageOS since its <a href="https://github.com/mirage/mirage/releases/tag/v2.7.0">2.7.0 release</a> at the end of February 2016.  Functoria provides support for command line arguments which can then either be passed at configuration time or at boot time to the unikernel (such as IP address configuration) using the <a href="http://erratique.ch/software/cmdliner">cmdliner library</a> underneath (and includes dynamic man pages, help, sensible command line parsing, and even visualisation (<code>mirage describe</code>) of the configuration and data dependencies).</p>
 <p>I won't go into details about command line arguments in here, please have a look at the <a href="https://mirage.io/blog/introducing-functoria">functoria blog post</a> in case you're interested.  Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel (<a href="http://marrakech2016.mirage.io">running here</a>, <a href="https://github.com/mirage/marrakech2016">source</a>).  Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown.  Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).</p>
-<h3>Unikernel</h3>
+<h3 id="unikernel">Unikernel</h3>
 <p>Similar to <a href="/Posts/nqsbWebsite">my nqsb.io website post</a>, this unikernel only has a single resource and thus does not need to do any parsing (or even call <code>read</code>).  The main function is <code>start</code>:</p>
 <pre><code class="language-OCaml">let start stack _ =
  S.listen_tcpv4 stack ~port:80 (serve rendered) ;
@ -44,7 +44,7 @@ let header = http_header
 </code></pre>
 <p>This puts together the pieces we need for a simple HTML site.  This unikernel does not have any external dependencies, if we assume that the mirage toolchain, the types, and the network implementation are already provided (the latter two are implicitly added by the mirage tool depending on the configuration, the first you'll have to install manually <code>opam install mirage</code>).</p>
 <p>But wait, where do <code>Style</code> and <code>Content</code> come from?  There are no <code>ml</code> modules in the repository.  Instead, there is a <a href="https://github.com/mirage/marrakech2016/blob/master/data/content.md">content.md</a> and <a href="https://github.com/mirage/marrakech2016/blob/master/data/style.css">style.css</a> in the <code>data</code> subdirectory.</p>
-<h3>Configuration</h3>
+<h3 id="configuration">Configuration</h3>
 <p>We use the builtin configuration time magic of functoria to translate these into OCaml modules, in such a way that our unikernel does not need to embed code to render markdown to HTML and carry along a markdown data file.</p>
 <p>Inside of <code>config.ml</code>, let's look again at the bottom:</p>
 <pre><code class="language-OCaml">let () =
@ -97,11 +97,11 @@ end
 </code></pre>
 <p>Functoria uses classes internally, and we extend the <a href="https://mirage.github.io/functoria/Functoria.base_configurable-c.html">base_configurable</a> class, which extends <a href="https://mirage.github.io/functoria/Functoria.configurable-c.html">configurable</a> with some sensible defaults.</p>
 <p>The important bits are what actually happens during <code>configure</code> and <code>clean</code>: execution of some shell commands (<code>echo</code>, <code>omd</code>, and <code>rm</code>) using the <a href="https://mirage.github.io/functoria/Functoria_app.html">functoria application builder</a> interface.  Some information is as well exposed via the <a href="https://mirage.github.io/functoria/Functoria_info.html">Functoria_info</a> module.</p>
-<h3>Wrapup</h3>
+<h3 id="wrapup">Wrapup</h3>
 <p>We walked through the configuration magic of MirageOS, which is a domain-specific language designed for MirageOS demands.  We can run arbitrary commands at compile time, and do not need to escape into external files, such as Makefile or shell scripts, but can embed them in our <code>config.ml</code>.</p>
 <p>I'm interested in feedback, either via
 <a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
-<h2>Other updates in the MirageOS ecosystem</h2>
+<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
 <ul>
 <li>now using Html5.P.print instead of string concatenation, as suggested by Drup (both on <a href="https://github.com/mirleft/nqsb.io/commit/f16291b67d203bf6b2ebc0c5c8479b7cfd153683">nqsb.io</a> and in <a href="https://github.com/Engil/Canopy/pull/46">Canopy</a>)
 </li>
--- a/Posts/Jackline
+++ b/Posts/Jackline
@ -27,7 +27,7 @@ not happy with the code base, but neverthelss consider it to be a successful
 project: dozens of friends are using it (no exact numbers), I got <a href="https://github.com/hannesm/jackline/graphs/contributors">contributions from other people</a>
 (more than 25 commits from more than 8 individuals), I use it on a daily basis
 for lots of personal communication.</p>
-<h2>What is XMPP?</h2>
+<h2 id="what-is-xmpp">What is XMPP?</h2>
 <p>The eXtensible Messaging and Presence Protocol (previously known as Jabber)
 describes (these days as <a href="https://tools.ietf.org/html/rfc6120">RFC 6120</a>) a
 communication protocol based on XML fragments, which enables near real-time
@ -45,7 +45,7 @@ Facebook chat used to be based on XMPP.  Those big companies wanted something
 &quot;more usable&quot; (where they're more in control, reliable message delivery via
 caching in the server and mandatory delivery receipts, multiple devices all
 getting the same messages), and thus moved away from the open standard.</p>
-<h3>XMPP Security</h3>
+<h3 id="xmpp-security">XMPP Security</h3>
 <p>Authentication is done via a TLS channel (where your client should authenticate
 the server), and SASL that the server authenticates your client.  I
 <a href="https://berlin.ccc.de/~hannes/secure-instant-messaging.pdf">investigated in 2008</a> (in German)
@ -71,7 +71,7 @@ offline messages, nicknames you gave to your buddies, subscription information,
 and information every time you connect (research of privacy preserving presence
 protocols has been done, but is not widely used AFAIK,
 e.g. <a href="http://cacr.uwaterloo.ca/techreports/2014/cacr2014-10.pdf">DP5</a>).</p>
-<h3>XMPP client landscape</h3>
+<h3 id="xmpp-client-landscape">XMPP client landscape</h3>
 <p>See <a href="https://en.wikipedia.org/wiki/Comparison_of_XMPP_clients">wikipedia</a> for an
 extensive comparison (which does not mention jackline :P).</p>
 <p>A more opinionated analysis is that you were free to choose between C - where
@ -97,7 +97,7 @@ take our work away! :)) - also named garbage collection, often used together
 with automated bounds checking -- this doesn't mean that you're safe - there are
 still logical flaws, and integer overflows (and funny things which happen at
 resource starvation), etc.</p>
-<h3>Goals and non-goals</h3>
+<h3 id="goals-and-non-goals">Goals and non-goals</h3>
 <p>My upfront motivation was to write and use an XMPP client tailored to my needs.
 I personally don't use many graphical applications (coding in emacs, mail via
 thunderbird, firefox, mplayer, mupdf), but stick mostly to terminal
@ -153,7 +153,7 @@ worked roughly one day a week on jackline.</p>
 <a href="https://common-lisp.net/project/slime/">slime</a> project (<a href="https://www.youtube.com/watch?v=eZDWJfB9XY4">watch Luke's presentation from 2013</a>) - if
 there's someone complaining about an issue, fix it within 10 minutes and ask
 them to update.  This only works if each user compiles the git version anyways.</p>
-<h2>User interface</h2>
+<h2 id="user-interface">User interface</h2>
 <p><img src="/static/img/jackline.png" alt="other screenshot" /></p>
 <p>Stated goal is <em>minimalistic</em>.  No heavy use of colours.  Visibility on
 both black and white background (btw, as a Unix process there is no way to find
@ -181,11 +181,11 @@ messages are displayed.</p>
 7bit ASCII version (I output some unicode characters for layout).  Recently I
 added support to customise the colours.  I tried to ensure it looks fine on both
 black and white background.</p>
-<h2>Code</h2>
+<h2 id="code">Code</h2>
 <p>Initially I targeted GTK with OCaml, but that excursion only lasted <a href="https://github.com/hannesm/jackline/commit/17b674130f7b1fcf2542eb5e0911a40b81fc724e">two weeks</a>,
 when I switched to a <a href="https://github.com/diml/lambda-term">lambda-term</a> terminal
 interface.</p>
-<h3>UI</h3>
+<h3 id="ui">UI</h3>
 <p>The lambda-term interface survived for a good year (until <a href="https://github.com/hannesm/jackline/pull/117">7th Feb 2016</a>),
 when I started to use <a href="https://github.com/pqwy/notty">notty</a> - developed by
 David - using a decent <a href="http://erratique.ch/software/uutf">unicode library</a>.</p>
@ -195,7 +195,7 @@ the symbol 茶 gets two characters width (see screenshot at top of page), and th
 how many characters are already written on the terminal.</p>
 <p>I recommend to look into <a href="https://github.com/pqwy/notty">notty</a> if you want to
 do terminal graphics in OCaml!</p>
-<h3>Application logic and state</h3>
+<h3 id="application-logic-and-state">Application logic and state</h3>
 <p>Stepping back, an XMPP client reacts to two input sources: the user input
 (including terminal resize), and network input (or failure).  The output is a
 screen (80x25 characters) image.  Each input event can trigger output events on
@ -233,7 +233,7 @@ need to), configurable colours, tab completions for nicknames and commands,
 per-user input history, emacs keybindings.  It even works with the XMPP gateway
 provided by slack (some startup doing a centralised groupchat with picture embedding and
 animated cats).</p>
-<h3>Road ahead</h3>
+<h3 id="road-ahead">Road ahead</h3>
 <p>Common feature requests are: <a href="https://github.com/hannesm/jackline/issues/153">omemo support</a>,
 <a href="https://github.com/hannesm/jackline/issues/104">IRC support</a>,
 <a href="https://github.com/hannesm/jackline/issues/115">support for multiple accounts</a>
@ -317,7 +317,7 @@ to the user interface, others are specific to XMPP and/or OTR): a new transport
 similar minimal UI features and colours).</p>
 </li>
 </ol>
-<h3>Conclusion</h3>
+<h3 id="conclusion">Conclusion</h3>
 <p>Jackline started as a procrastination project, and still is one.  I only develop
 on jackline if I enjoy it.  I'm not scared to try new approaches in jackline,
 and either reverting them or rewriting some chunks of code again.  It is a
--- a/Posts/Maintainers
+++ b/Posts/Maintainers
@ -9,7 +9,7 @@ a list of strings (usually a mail address).</p>
 <p>The data sources we correlate are the <code>maintainers</code> entry in opam file, and who
 actually committed in the opam repository.  This is inspired by <a href="https://github.com/ocaml/opam/issues/2693">some GitHub
 discussion</a>.</p>
-<h3>GitHub id and email address</h3>
+<h3 id="github-id-and-email-address">GitHub id and email address</h3>
 <p>For simplicity, since conex uses any (unique) identifier for authors, and the opam
 repository is hosted on GitHub, we use a GitHub id as author identifier.
 Maintainer information is an email address, thus we need a mapping between them.</p>
@ -24,7 +24,7 @@ we ignore PRs from GitHub organisations <code>PR.ignore_github</code>, where com
 <code>PR.ignore_pr</code> are picked from a different author (manually), bad mail addresses,
 and <a href="https://github.com/yallop">Jeremy's</a> mail address (it is added to too many GitHub ids otherwise).  The
 goal is to have a for an email address a single GitHub id.  329 authors with 416 mail addresses are mapped.</p>
-<h3>Maintainer in opam</h3>
+<h3 id="maintainer-in-opam">Maintainer in opam</h3>
 <p>As mentioned, lots of packages contain a <code>maintainers</code> entry.  In
 <a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L40-L68"><code>maintainers</code></a>
 we extract the mail addresses of the <a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L70-L94">most recently released opam
@ -34,7 +34,7 @@ field (such as mirage and xapi-project ;).  We're open for suggestions to extend
 this massaging to the needs.  Additionally, the contact at ocamlpro mail address
 was used for all packages before the maintainers entry was introduced (based on
 a discussion with Louis Gesbert).  132 packages with empty maintainers.</p>
-<h3>Fitness</h3>
+<h3 id="fitness">Fitness</h3>
 <p>Combining these two data sources, we hoped to find a strict small set of whom to
 authorise for which package.  Turns out some people use different mail addresses
 for git commits and opam maintainer entries, which <a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L233-L269">are be easily
@ -62,14 +62,14 @@ mirage -&gt; MagnusS dbuenzli djs55 hannesm hnrgrgr jonludlam mato mor1 pgj pqwy
 ocsigen -&gt; balat benozol dbuenzli hhugo hnrgrgr jpdeplaix mfp pveber scjung slegrand45 smondet vasilisp
 xapi-project -&gt; dbuenzli djs55 euanh mcclurmc rdicosmo simonjbeaumont yomimono
 </code></pre>
-<h3>Alternative approach: GitHub urls</h3>
+<h3 id="alternative-approach-github-urls">Alternative approach: GitHub urls</h3>
 <p>An alternative approach (attempted earlier) working only for GitHub hosted projects, is to authorise
 <a href="https://github.com/hannesm/conex/blob/github/analysis/maintainer.ml#L37-L91">the use of the user part of the GitHub repository
 URL</a>.
 Results after filtering GitHub organisations are not yet satisfactory (but only
 56 packages with no maintainer, <a href="https://github.com/hannesm/opam-repository/tree/github">output repo</a>.  This approach
 completely ignores the manually written maintainer field.</p>
-<h3>Conclusion</h3>
+<h3 id="conclusion">Conclusion</h3>
 <p>Manually maintained metadata is easily out of date, and not very useful.  But
 combining automatically created metadata with manually, and some manual tweaking
 leads to reasonable data.</p>
--- a/Posts/Monitoring
+++ b/Posts/Monitoring
@ -1,14 +1,14 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>All your metrics belong to influx</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="All your metrics belong to influx" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>All your metrics belong to influx</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/monitoring" class="tag">monitoring</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2022-03-08 (last updated: 2023-05-16)</span><article><h1>Introduction to monitoring</h1>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>All your metrics belong to influx</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="All your metrics belong to influx" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>All your metrics belong to influx</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/monitoring" class="tag">monitoring</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2022-03-08 (last updated: 2023-05-16)</span><article><h1 id="introduction-to-monitoring">Introduction to monitoring</h1>
 <p>At <a href="https://robur.coop">robur</a> we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our <a href="https://builds.robur.coop">reproducible builds infrastructure</a>. Another part is, once deployed we want to observe what is going on.</p>
 <p>I first got into touch with monitoring - collecting and graphing metrics - with <a href="https://oss.oetiker.ch/mrtg/">MRTG</a> and <a href="https://munin-monitoring.org/">munin</a> - and the simple network management protocol <a href="https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol">SNMP</a>. From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.</p>
 <p>The rise of monitoring where graphs are done dynamically (such as <a href="https://grafana.com/">Grafana</a>) and can be programmed (with a query language) by the operator are very neat, it allows to put metrics in relation after they have been recorded - thus if there's a thesis why something went berserk, you can graph the collected data from the past and prove or disprove the thesis.</p>
-<h1>Monitoring a MirageOS unikernel</h1>
+<h1 id="monitoring-a-mirageos-unikernel">Monitoring a MirageOS unikernel</h1>
 <p>From the operational perspective, taking security into account - either the data should be authenticated and integrity-protected, or being transmitted on a private network. We chose the latter, there's a private network interface only for monitoring. Access to that network is only granted to the unikernels and metrics collector.</p>
 <p>For MirageOS unikernels, we use the <a href="https://github.com/mirage/metrics">metrics</a> library - which design shares the idea of <a href="https://erratique.ch/software/logs">logs</a> that only if there's a reporter registered, work is performed.  We use the Influx line protocol via TCP to report via <a href="https://www.influxdata.com/time-series-platform/telegraf/">Telegraf</a> to <a href="https://www.influxdata.com/">InfluxDB</a>. But due to the design of <a href="https://github.com/mirage/metrics">metrics</a>, other reporters can be developed and used -- prometheus, SNMP, your-other-favourite are all possible.</p>
 <p>Apart from monitoring metrics, we use the same network interface for logging via syslog. Since the logs library separates the log message generation (in the OCaml libraries) from the reporting, we developed <a href="https://github.com/hannesm/logs-syslog">logs-syslog</a>, which registers a log reporter sending each log message to a syslog sink.</p>
 <p>We developed a small library for metrics reporting of a MirageOS unikernel into the <a href="https://github.com/roburio/monitoring-experiments">monitoring-experiments</a> package - which also allows to dynamically adjust log level and disable or enable metrics sources.</p>
-<h2>Required components</h2>
+<h2 id="required-components">Required components</h2>
 <p>Install from your operating system the packages providing telegraf, influxdb, and grafana.</p>
 <p>Setup telegraf to contain a socket listener:</p>
 <pre><code>[[inputs.socket_listener]]
@ -18,7 +18,7 @@
 </code></pre>
 <p>Use a unikernel that reports to Influx (below the heading &quot;Unikernels (with metrics reported to Influx)&quot; on <a href="https://builds.robur.coop">builds.robur.coop</a>) and provide <code>--monitor=192.168.42.14</code> as boot parameter. Conventionally, these unikernels expect a second network interface (on the &quot;management&quot; bridge) where telegraf (and a syslog sink) are running. You'll need to pass <code>--net=management</code> and <code>--arg='--management-ipv4=192.168.42.x/24'</code> to albatross-client.</p>
 <p>Albatross provides a <code>albatross-influx</code> daemon that reports information from the host system about the unikernels to influx. Start it with <code>--influx=192.168.42.14</code>.</p>
-<h2>Adding monitoring to your unikernel</h2>
+<h2 id="adding-monitoring-to-your-unikernel">Adding monitoring to your unikernel</h2>
 <p>If you want to extend your own unikernel with metrics, follow along these lines.</p>
 <p>An example is the <a href="https://github.com/roburio/dns-primary-git">dns-primary-git</a> unikernel, where on the branch <code>future</code> we have a single commit ahead of main that adds monitoring. The difference is in the unikernel configuration and the main entry point. See the <a href="https://builds.robur.coop/job/dns-primary-git-monitoring/build/latest/">binary builts</a> in contrast to the <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/">non-monitoring builts</a>.</p>
 <p>In config, three new command line arguments are added: <code>--monitor=IP</code>, <code>--monitor-adjust=PORT</code> <code>--syslog=IP</code> and <code>--name=STRING</code>. In addition, the package <code>monitoring-experiments</code> is required. And a second network interface <code>management_stack</code> using the prefix <code>management</code> is required and passed to the unikernel. Since the syslog reporter requires a console (to report when logging fails), also a console is passed to the unikernel. Each reported metrics includes a tag <code>vm=&lt;name&gt;</code> that can be used to distinguish several unikernels reporting to the same InfluxDB.</p>
--- a/Posts/NGI
+++ b/Posts/NGI
@ -1,34 +1,34 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The road ahead for MirageOS in 2021</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The road ahead for MirageOS in 2021" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The road ahead for MirageOS in 2021</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a></div><span class="date">Published: 2021-01-25 (last updated: 2021-11-19)</span><article><h2>Introduction</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The road ahead for MirageOS in 2021</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The road ahead for MirageOS in 2021" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The road ahead for MirageOS in 2021</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a></div><span class="date">Published: 2021-01-25 (last updated: 2021-11-19)</span><article><h2 id="introduction">Introduction</h2>
 <p>2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations,  and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)</p>
 <p>I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.</p>
 <p>For 2021, we (at <a href="https://robur.coop">robur</a>) got a grant from the EU (via <a href="https://pointer.ngi.eu">NGI pointer</a>) for &quot;Deploying MirageOS&quot; (more details below), and another grant from <a href="https://ocaml-sf.org">OCaml software foundation</a> for securing the opam supply chain (using <a href="https://github.com/hannesm/conex">conex</a>). Some long-awaited releases for MirageOS libraries, namely a <a href="https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh">ssh implementation</a> and a rewrite of our <a href="https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/">git implementation</a> have already been published.</p>
 <p>With my MirageOS view, 2020 was a pretty successful year, where we managed to add more features, fixed lots of bugs, and paved the road ahead. I want to thank <a href="https://ocamllabs.io/">OCamlLabs</a> for funding work on MirageOS maintenance.</p>
-<h2>Recap 2020</h2>
+<h2 id="recap-2020">Recap 2020</h2>
 <p>Here is a very subjective random collection of accomplishments in 2020, where I was involved with some degree.</p>
-<h3>NetHSM</h3>
+<h3 id="nethsm">NetHSM</h3>
 <p><a href="https://www.nitrokey.com/products/nethsm">NetHSM</a> is a hardware security module in software. It is a product that uses MirageOS for security, and is based on the <a href="https://muen.sk">muen</a> separation kernel. We at <a href="https://robur.coop">robur</a> were heavily involved in this product. It already has been security audited by an external team. You can pre-order it from Nitrokey.</p>
-<h3>TLS 1.3</h3>
+<h3 id="tls-1.3">TLS 1.3</h3>
 <p>Dating back to 2016, at the <a href="https://www.ndss-symposium.org/ndss2016/tron-workshop-programme/">TRON</a> (TLS 1.3 Ready or NOt), we developed a first draft of a 1.3 implementation of <a href="https://github.com/mirleft/ocaml-tls">OCaml-TLS</a>. Finally in May 2020 we got our act together, including ECC (ECDH P256 from <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a>, X25519 from <a href="https://project-everest.github.io/">hacl</a>) and testing with <a href="https://github.com/tlsfuzzer/tlsfuzzer">tlsfuzzer</a>, and release tls 0.12.0 with TLS 1.3 support. Later we added <a href="https://github.com/mirleft/ocaml-tls/pull/414">ECC ciphersuites to TLS version 1.2</a>, implemented <a href="https://github.com/mirleft/ocaml-tls/pull/414">ChaCha20/Poly1305</a>, and fixed an <a href="https://github.com/mirleft/ocaml-tls/pull/424">interoperability issue with Go's implementation</a>.</p>
 <p><a href="https://github.com/mirage/mirage-crypto">Mirage-crypto</a> provides the underlying cryptographic primitives, initially released in March 2020 as a fork of <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a> -- huge thanks to <a href="https://github.com/pqwy">pqwy</a> for his great work. Mirage-crypto detects <a href="https://github.com/mirage/mirage-crypto/pull/53">CPU features at runtime</a> (thanks to <a href="https://github.com/Julow">Julow</a>) (<a href="https://github.com/mirage/mirage-crypto/pull/96">bugfix for bswap</a>), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports <a href="https://github.com/mirage/mirage-crypto/pull/39">compilation on Windows</a> (thanks to <a href="https://github.com/avsm">avsm</a>), <a href="https://github.com/mirage/mirage-crypto/pull/90">async entropy harvesting</a> (thanks to <a href="https://github.com/seliopou">seliopou</a>), <a href="https://github.com/mirage/mirage-crypto/pull/65">32 bit support</a>, <a href="https://github.com/mirage/mirage-crypto/pull/72">chacha20/poly1305</a> (thanks to <a href="https://github.com/abeaumont">abeaumont</a>), <a href="https://github.com/mirage/mirage-crypto/pull/84">cross-compilation</a> (thanks to <a href="https://github.com/EduardoRFS">EduardoRFS</a>) and <a href="https://github.com/mirage/mirage-crypto/pull/78">various</a> <a href="https://github.com/mirage/mirage-crypto/pull/81">bug</a> <a href="https://github.com/mirage/mirage-crypto/pull/83">fixes</a>, even <a href="https://github.com/mirage/mirage-crypto/pull/95">memory leak</a> (thanks to <a href="https://github.com/talex5">talex5</a> for reporting several of these issues), and <a href="https://github.com/mirage/mirage-crypto/pull/99">RSA</a> <a href="https://github.com/mirage/mirage-crypto/pull/100">interoperability</a> (thanks to <a href="https://github.com/psafont">psafont</a> for investigation and <a href="https://github.com/mattjbray">mattjbray</a> for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.</p>
-<h3>Qubes Firewall</h3>
+<h3 id="qubes-firewall">Qubes Firewall</h3>
 <p>The <a href="https://github.com/mirage/qubes-mirage-firewall/">MirageOS based Qubes firewall</a> is the most widely used MirageOS unikernel. And it got major updates: in May <a href="https://github.com/linse">Steffi</a> <a href="https://groups.google.com/g/qubes-users/c/Xzplmkjwa5Y">announced</a> her and <a href="https://github.com/yomimono">Mindy's</a> work on improving it for Qubes 4.0 - including <a href="https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x">dynamic firewall rules via QubesDB</a>. Thanks to <a href="https://prototypefund.de/project/portable-firewall-fuer-qubesos/">prototypefund</a> for sponsoring.</p>
 <p>In October 2020, we released <a href="https://mirage.io/blog/announcing-mirage-39-release">Mirage 3.9</a> with PVH virtualization mode (thanks to <a href="https://github.com/mato">mato</a>). There's still a <a href="https://github.com/mirage/qubes-mirage-firewall/issues/120">memory leak</a> to be investigated and fixed.</p>
-<h3>IPv6</h3>
+<h3 id="ipv6">IPv6</h3>
 <p>In December, with <a href="https://mirage.io/blog/announcing-mirage-310-release">Mirage 3.10</a> we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to <a href="https://github.com/nojb">nojb</a> for the initial code and <a href="https://github.com/MagnusS">MagnusS</a>.</p>
 <p>Turns out this blog, but also robur services, are now available via IPv6 :)</p>
-<h3>Albatross</h3>
+<h3 id="albatross">Albatross</h3>
 <p>Also in December, I pushed an initial release of <a href="https://github.com/roburio/albatross">albatross</a>, a unikernel orchestration system with remote access. <em>Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.</em></p>
 <p>Thanks to <a href="https://github.com/reynir">reynir</a> for statistics support on Linux and improvements of the systemd service scripts. Also thanks to <a href="https://github.com/cfcs">cfcs</a> for the initial Linux port.</p>
-<h3>CA certs</h3>
+<h3 id="ca-certs">CA certs</h3>
 <p>For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to <a href="https://github.com/emillon">emillon</a> for initial code, there are now <a href="https://github.com/mirage/ca-certs">ca-certs</a> and <a href="https://github.com/mirage/ca-certs-nss">ca-certs-nss</a> opam packages (see <a href="https://discuss.ocaml.org/t/ann-ca-certs-and-ca-certs-nss">release announcement</a>) which fills this gap.</p>
-<h2>Unikernels</h2>
+<h2 id="unikernels">Unikernels</h2>
 <p>I developed several useful unikernels in 2020, and also pushed <a href="https://mirage.io/wiki/gallery">a unikernel gallery</a> to the Mirage website:</p>
-<h3>Traceroute in MirageOS</h3>
+<h3 id="traceroute-in-mirageos">Traceroute in MirageOS</h3>
 <p>I already wrote about <a href="/Posts/Traceroute">traceroute</a> which traces the routing to a given remote host.</p>
-<h3>Unipi - static website hosting</h3>
+<h3 id="unipi---static-website-hosting">Unipi - static website hosting</h3>
 <p><a href="https://github.com/roburio/unipi">Unipi</a> is a static site webserver which retrieves the content from a remote git repository. Let's encrypt certificate provisioning and dynamic updates via a webhook to be executed for every push.</p>
-<h4>TLSTunnel - TLS demultiplexing</h4>
+<h4 id="tlstunnel---tls-demultiplexing">TLSTunnel - TLS demultiplexing</h4>
 <p>The physical machine this blog and other robur infrastructure runs on has been relocated from Sweden to Germany mid-December. Thanks to UPS! Fewer IPv4 addresses are available in the new data center, which motivated me to develop <a href="https://github.com/roburio/tlstunnel">tlstunnel</a>.</p>
 <p>The new behaviour is as follows (see the <code>monitoring</code> branch):</p>
 <ul>
@ -43,9 +43,9 @@
 <li>setting up a new service is very straightforward: only the new name needs to be registered with tlstunnel together with the TCP backend, and everything will just work
 </li>
 </ul>
-<h2>2021</h2>
+<h2 id="section">2021</h2>
 <p>The year started with a release of <a href="https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh">awa</a>, a SSH implementation in OCaml (thanks to <a href="https://github.com/haesbaert">haesbaert</a> for initial code). This was followed by a <a href="https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/">git 3.0 release</a> (thanks to <a href="https://github.com/dinosaure">dinosaure</a>).</p>
-<h3>Deploying MirageOS - NGI Pointer</h3>
+<h3 id="deploying-mirageos---ngi-pointer">Deploying MirageOS - NGI Pointer</h3>
 <p>For 2021 we at robur received funding from the EU (via <a href="https://pointer.ngi.eu/">NGI pointer</a>) for &quot;Deploying MirageOS&quot;, which boils down into three parts:</p>
 <ul>
 <li>reproducible binary releases of MirageOS unikernels,
@ -58,10 +58,10 @@
 <p>Of course this will all be available open source. Please get in touch via eMail (team aT robur dot coop) if you're eager to integrate MirageOS unikernels into your infrastructure.</p>
 <p>We discovered at an initial meeting with an infrastructure provider that a DNS resolver is of interest - even more now that dnsmasq suffered from <a href="https://www.jsof-tech.com/wp-content/uploads/2021/01/DNSpooq_Technical-Whitepaper.pdf">dnspooq</a>. We are already working on an <a href="https://github.com/mirage/ocaml-dns/pull/251">implementation of DNSSec</a>.</p>
 <p>MirageOS unikernels are binary reproducible, and <a href="https://github.com/rjbou/orb/pull/1">infrastructure tools are available</a>. We are working hard on a web interface (and REST API - think of it as &quot;Docker Hub for MirageOS unikernels&quot;), and more tooling to verify reproducibility.</p>
-<h3>Conex - securing the supply chain</h3>
+<h3 id="conex---securing-the-supply-chain">Conex - securing the supply chain</h3>
 <p>Another funding from the <a href="http://ocaml-sf.org/">OCSF</a> is to continue development and deploy <a href="https://github.com/hannesm/conex">conex</a> - to bring trust into opam-repository. This is a great combination with the reproducible build efforts, and will bring much more trust into retrieving OCaml packages and using MirageOS unikernels.</p>
-<h3>MirageOS 4.0</h3>
+<h3 id="mirageos-4.0">MirageOS 4.0</h3>
 <p>Mirage so far still uses ocamlbuild and ocamlfind for compiling the virtual machine binary. But the switch to dune is <a href="https://github.com/mirage/mirage/issues/1195">close</a>, a lot of effort has been done. This will make the developer experience of MirageOS much more smooth, with a per-unikernel monorepo workflow where you can push your changes to the individual libraries.</p>
-<h2>Footer</h2>
+<h2 id="footer">Footer</h2>
 <p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
 </article></div></div></main></body></html>
--- a/Posts/OCaml
+++ b/Posts/OCaml
@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Why OCaml</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Why OCaml" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Why OCaml</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-17 (last updated: 2021-11-19)</span><article><h2>Programming</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Why OCaml</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Why OCaml" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Why OCaml</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-17 (last updated: 2021-11-19)</span><article><h2 id="programming">Programming</h2>
 <p>For me, programming is fun.  I enjoy doing it, every single second.  All the way
 from designing over experimenting to debugging why it does not do what I want.
 In the end, the computer is dumb and executes only what you (or code from
@ -17,7 +17,7 @@ allows a developer to encode invariants, without the need to defer to runtime
 assertions.  Type systems differ in their expressive power (<a href="https://en.wikipedia.org/wiki/Dependent_type">dependent typing</a> are the hot research area at the moment).  Tooling depends purely
 on the community size, natural selection will prevail the useful tools
 (community size gives inertia to other factors: demand for libraries, package manager, activity on stack overflow, etc.).</p>
-<h2>Why OCaml?</h2>
+<h2 id="why-ocaml">Why OCaml?</h2>
 <p>As already mentioned in <a href="/Posts/About">other</a>
 <a href="/Posts/OperatingSystem">articles</a> here, it is a
 combination of sufficiently large community, runtime stability and performance, modularity,
@ -50,7 +50,7 @@ code might end up in an error state (common for parsers which process input
 from the network), return a variant type as value (<code>type ('a, 'b) result = Ok of 'a | Error of 'b</code>).
 That way, the caller has to handle
 both the success and failure case explicitly.</p>
-<h2>Where to start?</h2>
+<h2 id="where-to-start">Where to start?</h2>
 <p>The <a href="https://ocaml.org">OCaml website</a> contains a <a href="https://ocaml.org/learn/tutorials/">variety of
 tutorials</a> and examples, including
 <a href="https://ocaml.org/learn/tutorials/get_up_and_running.html">introductionary
@ -105,7 +105,7 @@ which watches lots of MirageOS-related repositories.</p>
 specification and
 implementation</a>.  I'm interested in feedback, either via
 <a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
-<h2>Other updates in the MirageOS ecosystem</h2>
+<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
 <ul>
 <li>Canopy now sends out appropriate <a href="https://github.com/Engil/Canopy/pull/23">content type</a> HTTP headers
 </li>
--- a/Posts/OpamMirror
+++ b/Posts/OpamMirror
@ -1,22 +1,22 @@
 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Mirroring the opam repository and all tarballs</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Mirroring the opam repository and all tarballs" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a><a href="/tags/opam" class="tag">opam</a></div><span class="date">Published: 2022-09-29 (last updated: 2022-10-11)</span><article><p>We at <a href="https://robur.coop">robur</a> developed <a href="https://git.robur.io/robur/opam-mirror">opam-mirror</a> in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).</p>
-<h1>What is opam and why should I care?</h1>
+<h1 id="what-is-opam-and-why-should-i-care">What is opam and why should I care?</h1>
 <p><a href="https://opam.ocaml.org">Opam</a> is the OCaml package manager (also used by other projects such as <a href="https://coq.inria.fr">coq</a>). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as <a href="https://github.com/ocaml/opam-repository">ocaml/opam-repository</a>, where authors of OCaml software can contribute (as pull request) their latest releases.</p>
 <p>When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state &quot;my package will not work with <a href="https://erratique.ch/software/cmdliner">cmdliner</a> in version 1.2.0&quot;).</p>
 <p>So, the opam-repository holds the metadata of lots of OCaml packages (around 4000 at the moment this article was written) with lots of versions (in total 25000) that have been released. It is used by the opam client to figure out which packages to install or upgrade (using a solver that takes the version bounds into consideration).</p>
 <p>Of course, opam can use other repositories (overlays) or forks thereof. So nothing stops you from using any other opam repository. The url to the source code of each package may be a tarball, or a git repository or other version control systems.</p>
 <p>The vast majority of opam packages released to the opam-repository include a link to the source tarball and a cryptographic hash of the tarball. This is crucial for security (under the assumption the opam-repository has been downloaded from a trustworthy source - check back later this year for updates on <a href="/Posts/Conex">conex</a>). At the moment, there are some weak spots in respect to security: md5 is still allowed, and the hash and the tarball are downloaded from the same server: anyone who is in control of that server can inject arbitrary malicious data. As outlined above, we're working on infrastructure which fixes the latter issue.</p>
-<h1>How does the opam client work?</h1>
+<h1 id="how-does-the-opam-client-work">How does the opam client work?</h1>
 <p>Opam, after initialisation, downloads the <code>index.tar.gz</code> from <code>https://opam.ocaml.org/index.tar.gz</code>, and uses this as the local opam universe. An <code>opam install cmdliner</code> will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: <code>https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e</code>.</p>
-<h1>How does the opam repository work?</h1>
+<h1 id="how-does-the-opam-repository-work">How does the opam repository work?</h1>
 <p>According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses <code>opam admin index</code> periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was <a href="https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/">a reported downtime</a>.</p>
 <p>Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS <a href="http://retreat.mirage.io">hack retreats</a> in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).</p>
-<h1>Re-developing &quot;opam admin create&quot; as MirageOS unikernel</h1>
+<h1 id="re-developing-opam-admin-create-as-mirageos-unikernel">Re-developing &quot;opam admin create&quot; as MirageOS unikernel</h1>
 <p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.io/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
 <p>Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.</p>
 <p>In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with <a href="https://github.com/mirage/ocaml-crunch">crunch</a>, or using a remote git repository), but recent development, e.g. of <a href="https://somerandomidiot.com/blog/2022/03/04/chamelon/">chamelon</a> sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.</p>
 <p>Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.</p>
-<h1>Code development and improvements</h1>
+<h1 id="code-development-and-improvements">Code development and improvements</h1>
 <p>Initially, our plan was to use <a href="https://github.com/mirage/ocaml-git">ocaml-git</a> for pulling the repository, <a href="https://github.com/yomimono/chamelon">chamelon</a> for persistent storage, and <a href="https://github.com/inhabitedtype/httpaf">httpaf</a> as web server. With <a href="https://github.com/mirage/ocaml-tar">ocaml-tar</a> recent support of <a href="https://github.com/mirage/ocaml-tar/pull/88">gzip</a> we should be all set, and done within a few days.</p>
 <p>There is already a gap in the above plan: which http client to use - in the best case something similar to our <a href="https://github.com/roburio/http-lwt-client">http-lwt-client</a> - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using <a href="https://github.com/roburio/happy-eyeballs">happy-eyeballs</a> to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.</p>
 <p>On the path (over the last month), we fixed file descriptor leaks (memory leaks) in <a href="https://github.com/dinosaure/paf-le-chien">paf</a> -- which is used as a runtime for httpaf and h2.</p>
@ -26,7 +26,7 @@
 <p>Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as <a href="https://github.com/mirage/ocaml-fat">fat</a> may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed <a href="https://github.com/reynir/mirage-block-partition">mirage-block-partition</a> to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed <a href="https://git.robur.io/reynir/oneffs">oneffs</a> which stores a single value on the block device.</p>
 <p>We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains <a href="https://github.com/mirage/mirage-kv/pull/28">recent enhancements</a> of the mirage-kv API, there is a <code>get_partial</code>, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).</p>
 <p>What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy <a href="https://github.com/roburio/tlstunnel">tlstunnel</a>) similar to how <a href="https://github.com/roburio/unipi">unipi</a> does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).</p>
-<h1>Conclusion</h1>
+<h1 id="conclusion">Conclusion</h1>
 <p>To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide <a href="https://builds.robur.coop/job/opam-mirror">reproducible binaries</a> for solo5-hvt. You can use our opam mirror with <code>opam repository set-url default https://opam.robur.coop</code> (revert to the other with <code>opam repository set-url default https://opam.ocaml.org</code>) or use it as a backup with <code>opam repository add robur --rank 2 https://opam.robur.coop</code>.</p>
 <p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
 </article></div></div></main></body></html>
--- a/Posts/OperatingSystem
+++ b/Posts/OperatingSystem
@ -1,6 +1,6 @@
 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Operating systems</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Operating systems" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Operating systems</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/operating system" class="tag">operating system</a><a href="/tags/mirageos" class="tag">mirageos</a></div><span class="date">Published: 2016-04-09 (last updated: 2021-11-19)</span><article><p>Sorry to be late with this entry, but I had to fix some issues.</p>
-<h2>What is an operating system?</h2>
+<h2 id="what-is-an-operating-system">What is an operating system?</h2>
 <p>Wikipedia says: &quot;An operating system (OS) is system software that manages
 computer hardware and software resources and provides common services for
 computer programs.&quot;  Great.  In other terms, it is an abstraction layer.
@ -39,7 +39,7 @@ the MMU).</p>
 <p>This ominous &quot;cloud&quot; uses hypervisors on huge amount of physical machines, and
 executes off-the-shelf operating systems as virtual machines on top.  Accounting
 is done by resource usage (time, bandwidth, storage).</p>
-<h2>From scratch</h2>
+<h2 id="from-scratch">From scratch</h2>
 <p>Ok, now we have hypervisors which already deals with memory and scheduling.  Why
 should we have the very same functionality again in the (general purpose) operating
 system running as virtual machine?</p>
@ -55,7 +55,7 @@ a strange urge to spend some time on Dylan, which is really weird...&quot;</p>
 <a href="https://github.com/dylan-hackers/network-night-vision/">TCP/IP</a> stack in
 Dylan), and as mentioned earlier I went into formal methods and mechanised
 proofs of full functional correctness properties.</p>
-<h3>MirageOS</h3>
+<h3 id="mirageos">MirageOS</h3>
 <p>At the end of 2013, David pointed me to
 <a href="https://mirage.io">MirageOS</a>, an operating system developed from scratch in the
 functional and statically typed language <a href="https://ocaml.org">OCaml</a>.  I've not
@ -131,7 +131,7 @@ access control.</p>
 <p>I hope I gave some insight into what the purpose of an operating systems is, and
 how MirageOS fits into the picture. I'm interested in feedback, either via
 <a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
-<h2>Other updates in the MirageOS ecosystem</h2>
+<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
 <ul>
 <li>this website is based on <a href="https://github.com/Engil/Canopy">Canopy</a>, the content is stored as markdown in a <a href="https://git.robur.io/hannes/hannes.robur.coop">git repository</a>
 </li>
--- a/Posts/Pinata
+++ b/Posts/Pinata
@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The Bitcoin Piñata - no candy for you</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The Bitcoin Piñata - no candy for you" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/bitcoin" class="tag">bitcoin</a></div><span class="date">Published: 2018-04-18 (last updated: 2021-11-19)</span><article><h2>History</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The Bitcoin Piñata - no candy for you</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The Bitcoin Piñata - no candy for you" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/bitcoin" class="tag">bitcoin</a></div><span class="date">Published: 2018-04-18 (last updated: 2021-11-19)</span><article><h2 id="history">History</h2>
 <p>On February 10th 2015 David Kaloper-Meršinjak and Hannes Mehnert
 <a href="https://mirage.io/announcing-bitcoin-pinata">launched</a> (read also <a href="http://amirchaudhry.com/bitcoin-pinata">Amir's
 description</a>) our <a href="https://en.wikipedia.org/wiki/Bug_bounty_program">bug bounty
@ -14,7 +14,7 @@ Piñata</a>.</p>
 <p>The 10 Bitcoin in the Piñata were fluctuating in price over time, at peak worth 165000€.</p>
 <p>From the start of the Piñata project, we published the <a href="https://github.com/mirleft/btc-pinata">source code</a>, the virtual machine image, and the versions of the used libraries in a git repository.  Everybody could develop their exploits locally before launching them against our Piñata.  The Piñata provides TLS endpoints, which require private keys and certificates.  These are generated by the Piñata at startup, and the secret for the Bitcoin wallet is provided as a command line argument.</p>
 <p>Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with <a href="https://github.com/solo5/solo5">solo5</a>, and in December 2017 it was migrated to native BHyve (<a href="/Posts/Solo5">using <code>ukvm-bin</code> and solo5</a>).  We also changed the Piñata code to accomodate for updates, such as the <a href="https://mirage.io/blog/announcing-mirage-30-release">MirageOS 3.0 release</a>, and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).</p>
-<h2>Motivation</h2>
+<h2 id="motivation">Motivation</h2>
 <p>We built the Piñata for many purposes: to attract security professionals to evaluate our <a href="https://mirage.io/blog/introducing-ocaml-tls">from-scratch developed TLS stack</a>, to gather empirical data for our <a href="https://usenix15.nqsb.io">Usenix Security 15 paper</a>, and as an improvement to current bug bounty programs.</p>
 <p>Most bug bounty programs require communication via forms and long wait times for
 human experts to evaluate the potential bug.  This evaluation is subjective,
@ -27,7 +27,7 @@ to read arbitrary memory.  Everyone can track transactions of the blockchain and
 see if the wallet still contains the bounty.</p>
 <p>Of course, the Piñata can't prove that our stack is secure, and it can't prove
 that the access to the wallet is actually inside.  But trust us, it is!</p>
-<h2>Observations</h2>
+<h2 id="observations">Observations</h2>
 <p>I still remember vividly the first nights in February 2015, being so nervous that I woke up every two hours and checked the blockchain.  Did the Piñata still have the Bitcoins?  I was familiar with the code of the Piñata and was afraid there might be a bug which allows to bypass authentication or leak the private key.  So far, this doesn't seem to be the case.</p>
 <p>In April 2016 we stumbled upon an <a href="/Posts/BadRecordMac">information disclosure in the virtual network
 device driver for Xen in MirageOS</a>.  Given enough
--- a/Posts/ReproducibleOPAM
+++ b/Posts/ReproducibleOPAM
@ -1,19 +1,19 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Reproducible MirageOS unikernel builds</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Reproducible MirageOS unikernel builds" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/package signing" class="tag">package signing</a></div><span class="date">Published: 2019-12-16 (last updated: 2021-11-19)</span><article><h2>Reproducible builds summit</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Reproducible MirageOS unikernel builds</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Reproducible MirageOS unikernel builds" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/package signing" class="tag">package signing</a></div><span class="date">Published: 2019-12-16 (last updated: 2021-11-19)</span><article><h2 id="reproducible-builds-summit">Reproducible builds summit</h2>
 <p>I'm just back from the <a href="https://reproducible-builds.org/events/Marrakesh2019/">Reproducible builds summit 2019</a>. In 2018, several people developing <a href="https://ocaml.org">OCaml</a> and <a href="https://opam.ocaml.org">opam</a> and <a href="https://mirage.io">MirageOS</a>, attended <a href="https://reproducible-builds.org/events/paris2018/">the Reproducible builds summit in Paris</a>. The notes from last year on <a href="https://reproducible-builds.org/events/paris2018/report/#Toc11410_331763073">opam reproducibility</a> and <a href="https://reproducible-builds.org/events/paris2018/report/#Toc11681_331763073">MirageOS reproducibility</a> are online. After last years workshop, Raja started developing the opam reproducibilty builder <a href="https://github.com/rjbou/orb">orb</a>, which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.</p>
-<h2>Goal</h2>
+<h2 id="goal">Goal</h2>
 <p>Stepping back a bit, first look on the <a href="https://reproducible-builds.org/">goal of reproducible builds</a>: when compiling source code multiple times, the produced binaries should be identical. It should be sufficient if the binaries are behaviourally equal, but this is pretty hard to check. It is much easier to check <strong>bit-wise identity of binaries</strong>, and relaxes the burden on the checker -- checking for reproducibility is reduced to computing the hash of the binaries. Let's stick to the bit-wise identical binary definition, which also means software developers have to avoid non-determinism during compilation in their toolchains, dependent libraries, and developed code.</p>
 <p>A <a href="https://reproducible-builds.org/docs/test-bench/">checklist</a> of potential things leading to non-determinism has been written up by the reproducible builds project. Examples include recording the build timestamp into the binary, ordering of code and embedded data. The reproducible builds project also developed <a href="https://packages.debian.org/sid/disorderfs">disorderfs</a> for testing reproducibility and <a href="https://diffoscope.org/">diffoscope</a> for comparing binaries with file-dependent readers, falling back to <code>objdump</code> and <code>hexdump</code>. A giant <a href="https://tests.reproducible-builds.org/">test infrastructure</a> with <a href="https://tests.reproducible-builds.org/debian/index_variations.html">lots of variations</a> between the builds, mostly using Debian, has been setup over the years.</p>
 <p>Reproducibility is a precondition for trustworthy binaries. See <a href="https://reproducible-builds.org/#why-does-it-matter">why does it matter</a>. If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. <strong>I will only ever publish binaries if they are reproducible</strong>.</p>
 <p>My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of <a href="https://mirage.io">MirageOS unikernels</a> -- a unikernel is a statically linked ELF binary to be run as Unix process or <a href="https://github.com/solo5/solo5">virtual machine</a>. MirageOS heavily uses <a href="https://ocaml.org">OCaml</a> and <a href="https://opam.ocaml.org">opam</a>, the OCaml package manager, and is an opam package itself. Thus, <em>checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package</em>.</p>
-<h2>Reproducible builds with opam</h2>
+<h2 id="reproducible-builds-with-opam">Reproducible builds with opam</h2>
 <p>Testing for reproducibility is achieved by taking the sources and compile them twice independently. Afterwards the equality of the resulting binaries can be checked. In trivial projects, the sources is just a single file, or originate from a single tarball. In OCaml, opam uses <a href="https://github.com/ocaml/opam-repository">a community repository</a> where OCaml developers publish their package releases to, but can also use custom repositores, and in addition pin packages to git remotes (url including branch or commit), or a directory on the local filesystem. Manually tracking and updating all dependent packages of a MirageOS unikernel is not feasible: our hello-world compiled for hvt (kvm/BHyve) already has 79 opam dependencies, including the OCaml compiler which is distribued as opam package. The unikernel serving this website depends on 175 opam packages.</p>
 <p>Conceptually there should be two tools, the <em>initial builder</em>, which takes the latest opam packages which do not conflict, and exports exact package versions used during the build, as well as hashes of binaries. The other tool is a <em>rebuilder</em>, which imports the export, conducts a build, and outputs the hashes of the produced binaries.</p>
 <p>Opam has the concept of a <code>switch</code>, which is an environment where a package set is installed. Switches are independent of each other, and can already be exported and imported. Unfortunately the export is incomplete: if a package includes additional patches as part of the repository -- sometimes needed for fixing releases where the actual author or maintainer of a package responds slowly -- these package neither the patches end up in the export. Also, if a package is pinned to a git branch, the branch appears in the export, but this may change over time by pushing more commits or even force-pushing to that branch. In <a href="https://github.com/ocaml/opam/pull/4040">PR #4040</a> (under discussion and review), also developed during the summit, I propose to embed the additional files as base64 encoded values in the opam file. To solve the latter issue, I modified the export mechanism to <a href="https://github.com/ocaml/opam/pull/4055">embed the git commit hash (PR #4055)</a>, and avoid sources from a local directory and which do not have a checksum.</p>
 <p>So the opam export contains the information required to gather the exact same sources and build instructions of the opam packages. If the opam repository would be self-contained (i.e. not depend on any other tools), this would be sufficient. But opam does not run in thin air, it requires some system utilities such as <code>/bin/sh</code>, <code>sed</code>, a GNU make, commonly <code>git</code>, a C compiler, a linker, an assembler. Since opam is available on various operating systems, the plugin <code>depext</code> handles host system dependencies, e.g. if your opam package requires <code>gmp</code> to be installed, this requires slightly different names depending on host system or distribution, take a look at <a href="https://github.com/ocaml/opam-repository/blob/master/packages/conf-gmp/conf-gmp.1/opam">conf-gmp</a>. This also means, opam has rather good information about both the opam dependencies and the host system dependencies for each package. Please note that the host system packages used during compilation are not yet recorded (i.e. which <code>gmp</code> package was installed and used during the build, only that a <code>gmp</code> package has to be installed). The base utilities mentioned above (C compiler, linker, shell) are also not recorded yet.</p>
 <p>Operating system information available in opam (such as architecture, distribution, version), which in some cases maps to exact base utilities, is recorded in the build-environment, a separate artifact. The environment variable <a href="https://reproducible-builds.org/specs/source-date-epoch/"><code>SOURCE_DATE_EPOCH</code></a>, used for communicating the same timestamp when software is required to record a timestamp into the resulting binary, is also captured in the build environment.</p>
 <p>Additional environment variables may be captured or used by opam packages to produce different output. To avoid this, both the initial builder and the rebuilder are run with minimal environment variables: only <code>PATH</code> (normalised to a whitelist of <code>/bin</code>, <code>/usr/bin</code>, <code>/usr/local/bin</code> and <code>/opt/bin</code>) and <code>HOME</code> are defined. Missing information at the moment includes CPU features: some libraries (gmp?, nocrypto) emit different code depending on the CPU feature.</p>
-<h2>Tooling</h2>
+<h2 id="tooling">Tooling</h2>
 <p><em>TL;DR: A <strong>build</strong> builds an opam package, and outputs <code>.opam-switch</code>, <code>.build-hashes.N</code>, and <code>.build-environment.N</code>. A <strong>rebuild</strong> uses these artifacts as input, builds the package and outputs another <code>.build-hashes.M</code> and <code>.build-environment.M</code>.</em></p>
 <p>The command-line utility <code>orb</code> can be installed and used:</p>
 <pre><code class="language-sh">$ opam pin add orb git+https://github.com/hannesm/orb.git#active
@ -22,7 +22,7 @@ $ orb build --twice --keep-build-dir --diffoscope &lt;your-favourite-opam-packag
 <p>It provides two subcommands <code>build</code> and <code>rebuild</code>. The <code>build</code> command takes a list of local opam <code>--repos</code> where to take opam packages from (defaults to <code>default</code>), a compiler (either a variant <code>--compiler=4.09.0+flambda</code>, a version <code>--compiler=4.06.0</code>, or a pin to a local development version <code>--compiler-pin=~/ocaml</code>), and optionally an existing switch <code>--use-switch</code>. It creates a switch, builds the packages, and emits the opam export, hashes of all files installed by these packages, and the build environment. The flags <code>--keep-build</code> retains the build products, opam's <code>--keep-build-dir</code> in addition temporary build products and generated source code. If <code>--twice</code> is provided, a rebuild (described next) is executed after the initial build.</p>
 <p>The <code>rebuild</code> command takes a directory with the opam export and build environment to build the opam package. It first compares the build-environment with the host system, sets the <code>SOURCE_DATE_EPOCH</code> and switch location accordingly and executes the import. Once the build is finished, it compares the hashes of the resulting files with the previous run. On divergence, if build directories were kept in the previous build, and if diffoscope is available and <code>--diffoscope</code> was provided, diffoscope is run on the diverging files. If <code>--keep-build-dir</code> was provided as well, <code>diff -ur</code> can be used to compare the temporary build and sources, including build logs.</p>
 <p>The builds are run in parallel, as opam does, this parallelism does not lead to different binaries in my experiments.</p>
-<h2>Results and discussion</h2>
+<h2 id="results-and-discussion">Results and discussion</h2>
 <p><strong>All MirageOS unikernels I have deployed are reproducible \o/</strong>. Also, several binaries such as <code>orb</code> itself, <code>opam</code>, <code>solo5-hvt</code>, and all <code>albatross</code> utilities are reproducible.</p>
 <p>The unikernel range from hello world, web servers (e.g. this blog, getting its data on startup via a git clone to memory), authoritative DNS servers, CalDAV server. They vary in size between 79 and 200 opam packages, resulting in 2MB - 16MB big ELF binaries (including debug symbols). The <a href="https://github.com/roburio/reproducible-unikernel-repo">unikernel opam repository</a> contains some reproducible unikernels used for testing. Some work-in-progress enhancements are needed to achieve this:</p>
 <p>At the moment, the opam package of a MirageOS unikernel is automatically generated by <code>mirage configure</code>, but only used for tracking opam dependencies. I worked on <a href="https://github.com/mirage/mirage/pull/1022">mirage PR #1022</a> to extend the generated opam package with build and install instructions.</p>
@ -31,7 +31,7 @@ $ orb build --twice --keep-build-dir --diffoscope &lt;your-favourite-opam-packag
 <p>In functoria, a tool used to configure MirageOS devices and their dependencies, can emit a list of opam packages which were required to build the unikernel. This uses <code>opam list --required-by --installed --rec &lt;pkgs&gt;</code>, which uses the cudf graph (<a href="https://github.com/mirage/functoria/pull/189#issuecomment-566696426">thanks to Raja for explanation</a>), that is during the rebuild dropping some packages. The <a href="https://github.com/mirage/functoria/pull/189">PR #189</a> avoids by not using the <code>--rec</code> argument, but manually computing the fixpoint.</p>
 <p>Certainly, the choice of environment variables, and whether to vary them (as <a href="https://tests.reproducible-builds.org/debian/index_variations.html">debian does</a>) or to not define them (or normalise) while building, is arguably. Since MirageOS does neither support time zone nor internationalisation, there is no need to prematurely solving this issue. On related note, even with different locale settings, MirageOS unikernels are reproducible apart from an <a href="https://github.com/backtracking/ocamlgraph/pull/90">issue in ocamlgraph #90</a> embedding the output of <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html"><code>date</code></a>, which is different depending on <code>LANG</code> and locale (<code>LC_*</code>) settings.</p>
 <p>Prior art in reproducible MirageOS unikernels is the <a href="https://github.com/mirage/qubes-mirage-firewall/">mirage-qubes-firewall</a>. Since <a href="https://github.com/mirage/qubes-mirage-firewall/commit/07ff3d61477383860216c69869a1ffee59145e45">early 2017</a> it is reproducible. Their approach is different by building in a docker container with the opam repository pinned to an exact git commit.</p>
-<h2>Further work</h2>
+<h2 id="further-work">Further work</h2>
 <p>I only tested a certain subset of opam packages and MirageOS unikernels, mainly on a single machine (my laptop) running FreeBSD, and am happy if others will test reproducibility of their OCaml programs with the tools provided. There could as well be CI machines rebuilding opam packages and reporting results to a central repository. I'm pretty sure there are more reproducibility issues in the opam ecosystem. I developed an <a href="https://github.com/roburio/reproducible-testing-repo">reproducible testing opam repository</a> with opam packages that do not depend on OCaml, mainly for further tooling development. Some tests were also conducted on a Debian system with the same result. The variations, apart from build time, were using a different user, and different locale settings.</p>
 <p>As mentioned above, more environment, such as the CPU features, and external system packages, should be captured in the build environment.</p>
 <p>When comparing OCaml libraries, some output files (cmt / cmti / cma / cmxa) are not deterministic, but contain minimal diverge where I was not able to spot the root cause. It would be great to fix this, likely in the OCaml compiler distribution. Since the final result, the binary I'm interested in, is not affected by non-identical intermediate build products, I hope someone (you?) is interested in improving on this side. OCaml bytecode output also seems to be non-deterministic. There is <a href="https://github.com/coq/coq/issues/11229">a discussion on the coq issue tracker</a> which may be related.</p>
--- a/Posts/Solo5
+++ b/Posts/Solo5
@ -7,14 +7,14 @@
 <li>Update (2017-02-23): no more extra remotes, Mirage3 is released!
 </li>
 </ul>
-<h2>What?</h2>
+<h2 id="what">What?</h2>
 <p>As described <a href="/Posts/OperatingSystem">earlier</a>, MirageOS is a library operating system developed in <a href="/Posts/OCaml">OCaml</a>.  The code size is already pretty small, deployments are so far either as a UNIX binary or as a Xen virtual machine.</p>
 <p>Xen is a hypervisor, providing concrete device drivers for the actual hardware of a physical machine, memory management, scheduling, etc.  The initial release of Xen was done in 2003, since then the code size and code complexity of Xen is growing.  It also has various different mechanisms for virtualisation, hardware assisted ones or purely software based ones, some where the guest operating system needs to cooperate others where it does not need to cooperate.</p>
 <p>Since 2005, Intel CPUs (as well as AMD CPUs) provide hardware assistance for virtualisation (the VT-x extension), since 2008 extended page tables (EPT) are around which allow a guest to safely access the MMU.  Those features gave rise to much smaller hypervisors, such as KVM (mainly Linux), <a href="http://bhyve.org">bhyve</a> (FreeBSD), <a href="http://xhyve.org">xhyve</a> (MacOSX), <a href="http://undeadly.org/cgi?action=article&amp;sid=20150831183826">vmm</a> (OpenBSD), which do not need to emulate the MMU and other things in software.  The boot sequence in those hypervisors uses kexec or multiboot, instead of doing all the 16 bit, 32 bit, 64 bit mode changes manually.</p>
 <p>MirageOS initially targeted only Xen, in 2015 there was a port to use rumpkernel (a modularised NetBSD), and 2016 <a href="https://github.com/Solo5/solo5">solo5</a> emerged where you can run MirageOS on.  Solo5 comes in two shapes, either as <code>ukvm</code> on top of KVM, or as a multiboot image using <code>virtio</code> interfaces (block and network, plus a serial console).  Solo5 is only ~1000 lines of code (plus dlmalloc), and ISC-licensed.</p>
 <p>A recent <a href="https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_williams.pdf">paper</a> describes the advantages of a tiny virtual machine monitor in detail, namely no more <a href="http://venom.crowdstrike.com/">venom</a> like security issues since there is no legacy hardware emulated.  Also, each virtual machine monitor can be customised to the unikernel running on top of it: if the unikernel does not need a block device, the monitor shouldn't contain code for it.  The idea is to have one customised monitor for each unikernel.</p>
 <p>While lots of people seem to like KVM and Linux, I still prefer FreeBSD, their jails, and nowadays bhyve.  I finally found some time, thanks to various cleanups to the solo5 code base, to finally look into porting solo5 to FreeBSD/bhyve.  It runs and can output to console.</p>
-<h2>How?</h2>
+<h2 id="how">How?</h2>
 <p>These instructions are still slightly bumpy.  If you've a FreeBSD with bhyve (I use FreeBSD-CURRENT), and OCaml and opam (&gt;=1.2.2) installed, it is pretty straightforward to get solo5 running.  First, I'd suggest to use a fresh opam switch in case you work on other OCaml projects: <code>opam switch -A 4.04.0 solo5</code> (followed by <code>eval `opam config env` </code> to setup some environment variables).</p>
 <p>You need some software from the ports: <code>devel/pkgconf</code>, <code>devel/gmake</code>, <code>devel/binutils</code>, and <code>sysutils/grub2-bhyve</code>.</p>
 <p>An <code>opam install mirage mirage-logs solo5-kernel-virtio mirage-bootvar-solo5 mirage-solo5</code> should provide you with a basic set of libraries.</p>
@ -48,7 +48,7 @@ Kernel done.
 Goodbye!
 </code></pre>
 <p>Network and TLS stack works as well (tested 30th October).</p>
-<h2>Open issues</h2>
+<h2 id="open-issues">Open issues</h2>
 <ul>
 <li>I'm not happy to require <code>ld</code> from the ports (but the one in base does not produce sensible binaries with <code>-z max-page-size=0x1000</code> <a href="https://github.com/Solo5/solo5/pull/56">related</a>)
 </li>
@ -57,10 +57,10 @@ Goodbye!
 <li>Debugging via gdb should be doable somehow, bhyve has <a href="https://wiki.freebsd.org/bhyve/DebuggingWithGdb">some support for gdb</a>, but it is unclear to me what I need to do to enter the debugger (busy looping in the VM and a gdb remote to the port opened by bhyve does not work).
 </li>
 </ul>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>I managed to get solo5 to work with bhyve.  I even use clang instead of gcc and don't need to link <code>libgcc.a</code>. :)  It is great to see further development in hypervisors and virtual machine monitors.  Especially thanks to <a href="https://lucina.net">Martin Lucina</a> for getting things sorted.</p>
 <p>I'm interested in feedback, either via
 <a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
-<h2>Other updates in the MirageOS ecosystem</h2>
+<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
 <p>There were some busy times, several pull requests are still waiting to get merged (e.g. some cosmetics in <a href="https://github.com/mirage/mirage/pull/544">mirage</a> as preconditions for treemaps and dependency diagrams), I <a href="https://github.com/mirage/mirage/pull/547">proposed</a> to use <code>sleep_ns : int64 -&gt; unit io</code> instead of the <code>sleep : float -&gt; unit io</code> (nobody wants floating point numbers); also an RFC for <a href="https://github.com/mirage/mirage/pull/551">random</a>, Matt Gray <a href="https://github.com/mirage/mirage/pull/548">proposed</a> to get rid of <code>CLOCK</code> (and have a <code>PCLOCK</code> and a <code>MCLOCK</code> instead).  Soon there will be a major MirageOS release which breaks all the previous unikernels! :)</p>
 </article></div></div></main></body></html>
--- a/Posts/Summer2019
+++ b/Posts/Summer2019
@ -1,12 +1,12 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Summer 2019</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Summer 2019" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Summer 2019</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/package signing" class="tag">package signing</a><a href="/tags/tls" class="tag">tls</a><a href="/tags/monitoring" class="tag">monitoring</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2019-07-08 (last updated: 2021-11-19)</span><article><h2>Working at <a href="https://robur.io">robur</a></h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Summer 2019</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Summer 2019" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Summer 2019</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/package signing" class="tag">package signing</a><a href="/tags/tls" class="tag">tls</a><a href="/tags/monitoring" class="tag">monitoring</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2019-07-08 (last updated: 2021-11-19)</span><article><h2 id="working-at-robur">Working at <a href="https://robur.io">robur</a></h2>
 <p>As announced <a href="/Posts/DNS">previously</a>, I started to work at robur early 2018. We're a collective of five people, distributed around Europe and the US, with the goal to deploy MirageOS unikernels. We do this by developing bespoke MirageOS unikernels which provide useful services, and deploy them for ourselves. We also develop new libraries and enhance existing ones and other components of MirageOS. Example unikernels include <a href="https://robur.io">our website</a> which uses <a href="https://github.com/Engil/Canopy">Canopy</a>, a <a href="https://robur.io/Our%20Work/Projects#CalDAV-Server">CalDAV server that stores entries in a git remote</a>, and <a href="https://github.com/roburio/unikernels">DNS servers</a> (the latter two are further described below).</p>
 <p>Robur is part of the non-profit company <a href="https://techcultivation.org">Center for the Cultivation of Technology</a>, who are managing the legal and administrative sides for us. We're ourselves responsible to acquire funding to pay ourselves reasonable salaries. We received funding for CalDAV from <a href="https://prototypefund.de">prototypefund</a> and further funding from <a href="https://tarides.com">Tarides</a>, for TLS 1.3 from <a href="http://ocamllabs.io/">OCaml Labs</a>; security-audited an OCaml codebase, and received <a href="https://robur.io/Donate">donations</a>, also in the form of Bitcoins. We're looking for further funded collaborations and also contracting, mail us at <code>team@robur.io</code>. Please <a href="https://robur.io/Donate">donate</a> (tax-deductible in EU), so we can accomplish our goal of putting robust and sustainable MirageOS unikernels into production, replacing insecure legacy system that emit tons of CO<span style="vertical-align: baseline; position: relative;bottom: -0.4em;">2</span>.</p>
-<h2>Deploying MirageOS unikernels</h2>
+<h2 id="deploying-mirageos-unikernels">Deploying MirageOS unikernels</h2>
 <p>While several examples are running since years (the <a href="https://mirage.io">MirageOS website</a>, <a href="http://ownme.ipredator.se">Bitcoin Piñata</a>, <a href="https://tls.nqsb.io">TLS demo server</a>, etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.</p>
 <p>Service deployment is complex: you have to consider its configuration, exfiltration of logs and metrics, provisioning with valid key material (TLS certificate, hmac shared secret) and authenticators (CA certificate, ssh key fingerprint). Instead of requiring millions lines of code during orchestration (such as Kubernetes), creating the images (docker), or provisioning (ansible), why not minimise the required configuration and dependencies?</p>
 <p><a href="/Posts/VMM">Earlier in this blog I introduced Albatross</a>, which serves in an enhanced version as our deployment platform on a physical machine (running 15 unikernels at the moment), I won't discuss more detail thereof in this article.</p>
-<h2>CalDAV</h2>
+<h2 id="caldav">CalDAV</h2>
 <p><a href="https://linse.me/">Steffi</a> and I developed in 2018 a CalDAV server. Since November 2018 we have a test installation for robur, initially running as a Unix process on a virtual machine and persisting data to files on the disk. Mid-June 2019 we migrated it to a MirageOS unikernel, thanks to great efforts in <a href="https://github.com/mirage/ocaml-git">git</a> and <a href="https://github.com/mirage/irmin">irmin</a>, unikernels can push to a remote git repository. We <a href="https://github.com/haesbaert/awa-ssh/pull/8">extended the ssh library</a> with a ssh client and <a href="https://github.com/mirage/ocaml-git/pull/362">use this in git</a>. This also means our CalDAV server is completely immutable (does not carry state across reboots, apart from the data in the remote repository) and does not have persistent state in the form of a block device. Its configuration is mainly done at compile time by the selection of libraries (syslog, monitoring, ...), and boot arguments passed to the unikernel at startup.</p>
 <p>We monitored the resource usage when migrating our CalDAV server from Unix process to a MirageOS unikernel. The unikernel size is just below 10MB. The workload is some clients communicating with the server on a regular basis. We use <a href="https://grafana.com/">Grafana</a> with a <a href="https://www.influxdata.com/">influx</a> time series database to monitor virtual machines. Data is collected on the host system (<code>rusage</code> sysctl, <code>kinfo_mem</code> sysctl, <code>ifdata</code> sysctl, <code>vm_get_stats</code> BHyve statistics), and our unikernels these days emit further metrics (mostly counters: gc statistics, malloc statistics, tcp sessions, http requests and status codes).</p>
 <p><a href="/static/img/crobur-june-2019.png"><img src="/static/img/crobur-june-2019.png" width="700" /></a></p>
@ -14,18 +14,18 @@
 <p>A MirageOS unikernel, apart from a smaller attack surface, indeed uses fewer resources and actually emits less CO<span style="vertical-align: baseline; position: relative;bottom: -0.4em;">2</span> than the same service on a Unix virtual machine. So we're doing something good for the environment! :)</p>
 <p>Our calendar server contains at the moment 63 events, the git repository had around 500 commits in the past month: nearly all of them from the CalDAV server itself when a client modified data via CalDAV, and two manual commits: the initial data imported from the file system, and one commit for fixing a bug of the encoder in our <a href="https://github.com/roburio/icalendar/pull/2">icalendar library</a>.</p>
 <p>Our CalDAV implementation is very basic, scheduling, adding attendees (which requires sending out eMail), is not supported. But it works well for us, we have individual calendars and a shared one which everyone can write to. On the client side we use macOS and iOS iCalendar, Android DAVdroid, and Thunderbird. If you like to try our CalDAV server, have a look <a href="https://github.com/roburio/caldav/tree/future/README.md">at our installation instructions</a>.  Please <a href="https://github.com/roburio/caldav/issues">report issues</a> if you find issues or struggle with the installation.</p>
-<h2>DNS</h2>
+<h2 id="dns">DNS</h2>
 <p>There has been more work on our DNS implementation, now <a href="https://github.com/mirage/ocaml-dns">here</a>. We included a DNS client library, and some <a href="https://github.com/roburio/unikernels/tree/future">example unikernels</a> are available. They as well require our <a href="https://github.com/roburio/git-ssh-dns-mirage3-repo">opam repository overlay</a>. Please report issues if you run into trouble while experimenting with that.</p>
 <p>Most prominently is <code>primary-git</code>, a unikernel which acts as a primary authoritative DNS server (UDP and TCP). On startup, it fetches a remote git repository that contains zone files and shared hmac secrets. The zones are served, and secondary servers are notified with the respective serial numbers of the zones, authenticated using TSIG with the shared secrets. The primary server provides dynamic in-protocol updates of DNS resource records (<code>nsupdate</code>), and after successful authentication pushes the change to the remote git. To change the zone, you can just edit the zonefile and push to the git remote - with the proper pre- and post-commit-hooks an authenticated notify is send to the primary server which then pulls the git remote.</p>
 <p>Another noteworthy unikernel is <code>letsencrypt</code>, which acts as a secondary server, and whenever a TLSA record with custom type (0xFF) and a DER-encoded certificate signing request is observed, it requests a signature from letsencrypt by solving the DNS challenge. The certificate is pushed to the DNS server as TLSA record as well. The DNS implementation provides <code>ocertify</code> and <code>dns-mirage-certify</code> which use the above mechanism to retrieve valid let's encrypt certificates. The caller (unikernel or Unix command-line utility) either takes a private key directly or generates one from a (provided) seed and generates a certificate signing request. It then looks in DNS for a certificate which is still valid and matches the public key and the hostname. If such a certificate is not present, the certificate signing request is pushed to DNS (via the nsupdate protocol), authenticated using TSIG with a given secret. This way our public facing unikernels (website, this blog, TLS demo server, ..) block until they got a certificate via DNS on startup - we avoid embedding of the certificate into the unikernel image.</p>
-<h2>Monitoring</h2>
+<h2 id="monitoring">Monitoring</h2>
 <p>We like to gather statistics about the resource usage of our unikernels to find potential bottlenecks and observe memory leaks ;) The base for the setup is the <a href="https://github.com/mirage/metrics">metrics</a> library, which is similarly in design to the <a href="https://erratique.ch/software/logs">logs</a> library: libraries use the core to gather metrics. A different aspect is the reporter, which is globally registered and responsible for exfiltrating the data via their favourite protocol. If no reporter is registered, the work overhead is negligible.</p>
 <p><a href="/static/img/crobur-june-2019-unikernel.png"><img src="/static/img/crobur-june-2019-unikernel.png" width="700" /></a></p>
 <p>This is a dashboard which combines both statistics gathered from the host system and various metrics from the MirageOS unikernel. The <code>monitoring</code> branch of our opam repository overlay is used together with <a href="https://github.com/hannesm/monitoring-experiments">monitoring-experiments</a>. The logs errors counter (middle right) was the icalendar parser which tried to parse its badly emitted ics (the bug is now fixed, the dashboard is from last month).</p>
-<h2>OCaml libraries</h2>
+<h2 id="ocaml-libraries">OCaml libraries</h2>
 <p>The <a href="https://github.com/hannesm/domain-name">domain-name</a> library was developed to handle RFC 1035 domain names and host names. It initially was part of the DNS code, but is now freestanding to be used in other core libraries (such as ipaddr) with a small dependency footprint.</p>
 <p>The <a href="https://github.com/hannesm/gmap">GADT map</a> is a normal OCaml Map structure, but takes key-dependent value types by using a GADT. This library also was part of DNS, but is more broadly useful, we already use it in our icalendar (the data format for calendar entries in CalDAV) library, our <a href="https://git.robur.io/?p=openvpn.git;a=summary">OpenVPN</a> configuration parser uses it as well, and also <a href="https://github.com/mirleft/ocaml-x509/pull/115">x509</a> - which got reworked quite a bit recently (release pending), and there's preliminary PKCS12 support (which deserves its own article). <a href="https://github.com/hannesm/ocaml-tls">TLS 1.3</a> is available on a branch, but is not yet merged. More work is underway, hopefully with sufficient time to write more articles about it.</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>More projects are happening as we speak, it takes time to upstream all the changes, such as monitoring, new core libraries, getting our DNS implementation released, pushing Conex into production, more features such as DNSSec, ...</p>
 <p>I'm interested in feedback, either via <strike><a href="https://twitter.com/h4nnes">twitter</a></strike> <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
 </article></div></div></main></body></html>
--- a/Posts/Syslog
+++ b/Posts/Syslog
@ -2,7 +2,7 @@
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Exfiltrating log data using syslog</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Exfiltrating log data using syslog" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a><a href="/tags/logging" class="tag">logging</a></div><span class="date">Published: 2016-11-05 (last updated: 2021-11-19)</span><article><p>It has been a while since my last entry... I've been busy working on too many
 projects in parallel, and was also travelling on several continents.  I hope to
 get back to a biweekly cycle.</p>
-<h2>What is syslog?</h2>
+<h2 id="what-is-syslog">What is syslog?</h2>
 <p>According to <a href="https://en.wikipedia.org/wiki/Syslog">Wikipedia</a>, syslog is a
 standard for message logging.  Syslog permits separation of the software which
 generates, stores, reports, and analyses the message.  A syslog message contains
@ -59,7 +59,7 @@ into your <code>/var/log/messages</code>.</p>
 <p>This is a good first step, but we want more: on the one side integration into
 MirageOS, and a more reliable log stream (what about authentication and
 encryption?).  I'll cover both topics in the rest of this article.</p>
-<h3>MirageOS integration</h3>
+<h3 id="mirageos-integration">MirageOS integration</h3>
 <p>Since Mirage3, syslog is integrated (see
 <a href="http://docs.mirage.io/mirage/Mirage/index.html#type-syslog_config">documentation</a>).
 Some additions to your <code>config.ml</code> are needed, see <a href="https://github.com/hannesm/ns.nqsb.io/blob/master/config.ml">ns
@ -77,7 +77,7 @@ let () =
    foreign ~deps:[abstract logger]
    ...
 </code></pre>
-<h3>Reliable syslog</h3>
+<h3 id="reliable-syslog">Reliable syslog</h3>
 <p>The old BSD syslog RFC is obsoleted by <a href="https://tools.ietf.org/html/rfc5424">RFC
 5424</a>, which describes a new wire format,
 and also a transport over TCP, and <a href="https://tools.ietf.org/html/rfc5425">TLS</a> in
@ -119,7 +119,7 @@ interface</a>
 (also
 <a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_mirage_tls.html">TLS</a>).
 The code size is below 500 lines in total.</p>
-<h3>MirageOS syslog in production</h3>
+<h3 id="mirageos-syslog-in-production">MirageOS syslog in production</h3>
 <p>As collector I use syslog-ng, which is capable of receiving both the new and the
 old syslog messages on all three transports.  The configuration snippet for a
 BSD syslog TLS collector is as following:</p>
--- a/Posts/Traceroute
+++ b/Posts/Traceroute
@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Traceroute</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Traceroute" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Traceroute</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2020-06-24 (last updated: 2021-11-19)</span><article><h2>Traceroute</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Traceroute</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Traceroute" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Traceroute</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2020-06-24 (last updated: 2021-11-19)</span><article><h2 id="traceroute">Traceroute</h2>
 <p>Is a diagnostic utility which displays the route and measures transit delays of
 packets across an Internet protocol (IP) network.</p>
 <pre><code class="language-bash">$ doas solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --host=198.167.222.207
--- a/Posts/VMM
+++ b/Posts/VMM
@ -1,6 +1,6 @@
 <!DOCTYPE html>
 <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Albatross - provisioning, deploying, managing, and monitoring virtual machines</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Albatross - provisioning, deploying, managing, and monitoring virtual machines" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a><a href="/tags/provisioning" class="tag">provisioning</a></div><span class="date">Published: 2017-07-10 (last updated: 2023-05-16)</span><article><p>EDIT (2023-05-16): Please take a look at <a href="/Posts/Albatross">the updated article</a>.</p>
-<h2>How to deploy unikernels?</h2>
+<h2 id="how-to-deploy-unikernels">How to deploy unikernels?</h2>
 <p>MirageOS has a pretty good story on how to compose your OCaml libraries into a
 virtual machine image.  The <code>mirage</code> command line utility contains all the
 knowledge about which backend requires which library. This enables it to write a
@ -37,7 +37,7 @@ waits for (mutually!) authenticated network connections, and provides the
 desired commands; to create a new virtual machine, to acquire a block device of
 a given size, to destroy a virtual machine, to stream the console output of a
 virtual machine.</p>
-<h2>System design</h2>
+<h2 id="system-design">System design</h2>
 <p>The system bears minimalistic characteristics.  The single interface to the
 outside world is a TLS stream over TCP.  Internally, there is a family of
 processes, one of which has superuser privileges, communicating via unix domain
@ -102,7 +102,7 @@ are destroyed.</p>
 <p>The maximum size of a virtual machine image embedded into a X.509 certificate
 transferred over TLS is 2 ^ 24 - 1 bytes, roughly 16 MB.  If this turns out to
 be not sufficient, compression may help.  Or staging of deployment.</p>
-<h2>An example</h2>
+<h2 id="an-example">An example</h2>
 <p>Instructions on how to setup <code>vmmd</code> and the certificate authority are in the
 README file of the <a href="https://github.com/hannesm/albatross"><code>albatross</code> git repository</a>.  Here
 is some (stripped) terminal output:</p>
@ -190,7 +190,7 @@ virtual machine is started or not:</p>
 <pre><code class="language-bash">&gt; vmm_client cacert.pem hello.bundle hello.key localhost:1025
 success VM started
 </code></pre>
-<h2>Sharing is caring</h2>
+<h2 id="sharing-is-caring">Sharing is caring</h2>
 <p>Deploying unikernels is now easier for myself on my physical machine. That's
 fine.  Another aspect comes <em>for free</em> by reusing X.509: further delegation (and
 limiting thereof).  Within a delegation certificate, the basic constraints
@ -207,7 +207,7 @@ and vmmd will only start up to 2 virtual machines using 2GB of memory in total
 issued delegation (using a revocation certificate described above) to free up
 some resources for herself.  I don't need to interact when Alice or Dan share
 their delegated resources further.</p>
-<h2>Security</h2>
+<h2 id="security">Security</h2>
 <p>There are several security properties preserved by <code>vmmd</code>, such as the virtual
 machine image is never transmitted in clear.  Only properly authenticated
 clients can create, destroy, gather statistics of <em>their</em> virtual machines.</p>
@ -244,7 +244,7 @@ stored in a memory-backed file system.  A virtual machine with a lots of disk
 operation may only delay or starve revocation list updates - if this turns out
 to be a problem, the solution may be to use separate physical block devices for
 the revocation lists and virtual block devices for clients.</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>I showed a minimalistic system to provision, deploy, and manage virtual machine
 images.  It also allows to delegate resources (CPU, disk, ..) further.  I'm
 pretty satisfied with the security properties of the system.</p>
--- a/Posts/X50907
+++ b/Posts/X50907
@ -1,7 +1,7 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>X509 0.7</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="X509 0.7" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>X509 0.7</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/tls" class="tag">tls</a></div><span class="date">Published: 2019-08-15 (last updated: 2021-11-19)</span><article><h2>Cryptographic material</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>X509 0.7</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="X509 0.7" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>X509 0.7</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/tls" class="tag">tls</a></div><span class="date">Published: 2019-08-15 (last updated: 2021-11-19)</span><article><h2 id="cryptographic-material">Cryptographic material</h2>
 <p>Once a private and public key pair is generated (doesn't matter whether it is plain RSA, DSA, ECC on any curve), this is fine from a scientific point of view, and can already be used for authenticating and encrypting. From a practical point of view, the public parts need to be exchanged and verified (usually a fingerprint or hash thereof). This leads to the struggle how to encode this cryptographic material, and how to embed an identity (or multiple), capabilities, and other information into it. <a href="https://en.wikipedia.org/wiki/X.509">X.509</a> is a standard to solve this encoding and embedding, and provides more functionality, such as establishing chains of trust and revocation of invalidated or compromised material. X.509 uses certificates, which contain the public key, and additional information (in a extensible key-value store), and are signed by an issuer, either the private key corresponding to the public key - a so-called self-signed certificate - or by a different private key, an authority one step up the chain. A rather long, but very good introduction to certificates by Mike Malone is <a href="https://smallstep.com/blog/everything-pki.html">available here</a>.</p>
-<h2>OCaml ecosystem evolving</h2>
+<h2 id="ocaml-ecosystem-evolving">OCaml ecosystem evolving</h2>
 <p>More than 5 years ago David Kaloper and I <a href="https://mirage.io/blog/introducing-x509">released the initial ocaml-x509</a> package as part of our <a href="https://nqsb.io">TLS stack</a>, which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in <a href="https://tools.ietf.org/html/rfc6125">RFC 5280</a>). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's <a href="https://github.com/mirleft/ocaml-asn1-combinators">asn1-combinators</a> library changed much over time.</p>
 <p>The OCaml ecosystem evolved over the years, which lead to some changes:</p>
 <ul>
@ -28,27 +28,27 @@
 <li>Usage of the <a href="https://github.com/mirage/alcotest">alcotest</a> unit testing framework (instead of oUnit).
 </li>
 </ul>
-<h2>More use cases for X.509</h2>
+<h2 id="more-use-cases-for-x.509">More use cases for X.509</h2>
 <p>Initially, we designed and used ocaml-x509 for providing TLS server endpoints and validation in TLS clients - mostly on the public web, where each operating system ships a set of ~100 trust anchors to validate any web server certificate against. But once you have a X.509 implementation, every authentication problem can be solved by applying it.</p>
-<h3>Authentication with path building</h3>
+<h3 id="authentication-with-path-building">Authentication with path building</h3>
 <p>It turns out that the trust anchor sets are not equal across operating systems and versions, thus some web servers serve sets, instead of chains, of certificates - as described in <a href="https://tools.ietf.org/html/rfc4158">RFC 4158</a>, where the client implementation needs to build valid paths and accept a connection if any path can be validated. The path building was initially in 0.5.2 slightly wrong, but fixed quickly in <a href="https://github.com/mirleft/ocaml-x509/commit/1a1476308d24bdcc49d45c4cd9ef539ca57461d2">0.5.3</a>.</p>
-<h3>Fingerprint authentication</h3>
+<h3 id="fingerprint-authentication">Fingerprint authentication</h3>
 <p>The chain of trust validation is useful for the open web, where you as software developer don't know to which remote endpoint your software will ever connect to - as long as the remote has a certificate signed (via intermediates) by any of the trust anchors. In the early days, before <a href="https://letsencrypt.org/">let's encrypt</a> was launched and embedded as trust anchors (or cross-signed by already deployed trust anchors), operators needed to pay for a certificate - a business model where some CAs did not bother to check the authenticity of a certificate signing request, and thus random people owning valid certificates for microsoft.com or google.com.</p>
 <p>Instead of using the set of trust anchors, the fingerprint of the server certificate, or preferably the fingerprint of the public key of the certificate, can be used for authentication, as optionally done since some years in <a href="https://github.com/hannesm/jackline/commit/a1e6f3159be1e45e6b690845e1b29366c41239a2">jackline</a>, an XMPP client. Support for this certificate / public key pinning was added in x509 0.2.1 / 0.5.0.</p>
-<h3>Certificate signing requests</h3>
+<h3 id="certificate-signing-requests">Certificate signing requests</h3>
 <p>Until x509 0.4.0 there was no support for generating certificate signing requests (CSR), as defined in PKCS 10, which are self-signed blobs containing a public key, an identity, and possibly extensions. Such as CSR is sent to the certificate authority, and after validation of ownership of the identity and paying a fee, the certificate is issued. Let's encrypt specified the ACME protocol which automates the proof of ownership: they provide a HTTP API for requesting a challenge, providing the response (the proof of ownership) via HTTP or DNS, and then allow the submission of a CSR and downloading the signed certificate. The ocaml-x509 library provides operations for creating such a CSR, and also for signing a CSR to generate a certificate.</p>
 <p>Mindy developed the command-line utility <a href="https://github.com/yomimono/ocaml-certify/">certify</a> which uses these operations from the ocaml-x509 library and acts as a swiss-army knife purely in OCaml for these required operations.</p>
 <p>Maker developed a <a href="https://github.com/mmaker/ocaml-letsencrypt">let's encrypt library</a> which implements the above mentioned ACME protocol for provisioning CSR to certificates, also using our ocaml-x509 library.</p>
 <p>To complete the required certificate authority functionality, in x509 0.6.0 certificate revocation lists, both validation and signing, was implemented.</p>
-<h3>Deploying unikernels</h3>
+<h3 id="deploying-unikernels">Deploying unikernels</h3>
 <p>As <a href="/Posts/VMM">described in another post</a>, I developed <a href="https://github.com/hannesm/albatross">albatross</a>, an orchestration system for MirageOS unikernels. This uses ASN.1 for internal socket communication and allows remote management via a TLS connection which is mutually authenticated with a X.509 client certificate. To encrypt the X.509 client certificate, first a TLS handshake where the server authenticates itself to the client is established, and over that connection another TLS handshake is established where the client certificate is requested. Note that this mechanism can be dropped with TLS 1.3, since there the certificates are transmitted over an already encrypted channel.</p>
 <p>The client certificate already contains the command to execute remotely - as a custom extension, being it &quot;show me the console output&quot;, or &quot;destroy the unikernel with name = YYY&quot;, or &quot;deploy the included unikernel image&quot;. The advantage is that the commands are already authenticated, and there is no need for developing an ad-hoc protocol on top of the TLS session. The resource limits, assigned by the authority, are also part of the certificate chain - i.e. the number of unikernels, access to network bridges, available accumulated memory, accumulated size for block devices, are constrained by the certificate chain presented to the server, and currently running unikernels. The names of the chain are used for access control - if Alice and Bob have intermediate certificates from the same CA, neither Alice may manage Bob's unikernels, nor Bob may manage Alice's unikernels. I'm using albatross since 2.5 years in production on two physical machines with ~20 unikernels total (multiple users, multiple administrative domains), and it works stable and is much nicer to deal with than <code>scp</code> and custom hacked shell scripts.</p>
-<h2>Why 0.7?</h2>
+<h2 id="why-0.7">Why 0.7?</h2>
 <p>There are still some missing pieces in our ocaml-x509 implementation, namely modern ECC certificates (depending on elliptic curve primitives not yet available in OCaml), RSA-PSS signing (should be straightforward), PKCS 12 (there is a <a href="https://github.com/mirleft/ocaml-x509/pull/114">pull request</a>, but this should wait until asn1-combinators supports the <code>ANY defined BY</code> construct to cleanup the code), ...
 Once these features are supported, the library should likely be named PKCS since it supports more than X.509, and released as 1.0.</p>
 <p>The 0.7 release series moved a lot of modules and function names around, thus it is a major breaking release. By using a map instead of lists for extensions, GeneralName, ..., the API was further revised - invariants that each extension key (an ASN.1 object identifier) may occur at most once are now enforced. By not leaking exceptions through the public interface, the API is easier to use safely - see <a href="https://github.com/mmaker/ocaml-letsencrypt/commit/dc53518f46310f384c9526b1d96a8e8f815a09c7">let's encrypt</a>, <a href="https://git.robur.io/?p=openvpn.git;a=commitdiff;h=929c53116c1438ba1214f53df7506d32da566ccc">openvpn</a>, <a href="https://github.com/yomimono/ocaml-certify/pull/17">certify</a>, <a href="https://github.com/mirleft/ocaml-tls/pull/394">tls</a>, <a href="https://github.com/mirage/capnp-rpc/pull/158">capnp</a>, <a href="https://github.com/hannesm/albatross/commit/50ed6a8d1ead169b3e322aaccb469e870ad72acc">albatross</a>.</p>
 <p>I intended in 0.7.0 to have much more precise types, esp. for the SubjectAlternativeName (SAN) extension that uses a GeneralName, but it turns out the GeneralName is as well used for NameConstraints (NC) in a different way -- IP in SAN is an IPv4 or IPv6 address, in CN it is the IP/netmask; DNS is a domain name in SAN, in CN it is a name starting with a leading dot (i.e. &quot;.example.com&quot;), which is not a valid domain name. In 0.7.1, based on a bug report, I had to revert these variants and use less precise types.</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>The work on X.509 was sponsored by <a href="http://ocamllabs.io/">OCaml Labs</a>. You can support our work at robur by a <a href="https://robur.io/Donate">donation</a>, which we will use to work on our OCaml and MirageOS projects. You can also reach out to us to realize commercial products.</p>
 <p>I'm interested in feedback, either via <strike><a href="https://twitter.com/h4nnes">twitter</a></strike> <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
 </article></div></div></main></body></html>
--- a/Posts/nqsbWebsite
+++ b/Posts/nqsbWebsite
@ -1,12 +1,12 @@
 <!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Fitting the things together</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Fitting the things together" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Fitting the things together</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/http" class="tag">http</a><a href="/tags/tls" class="tag">tls</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2016-04-24 (last updated: 2021-11-19)</span><article><h2>Task</h2>
+<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Fitting the things together</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Fitting the things together" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Fitting the things together</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/http" class="tag">http</a><a href="/tags/tls" class="tag">tls</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2016-04-24 (last updated: 2021-11-19)</span><article><h2 id="task">Task</h2>
 <p>Our task is to build a small unikernel which provides a project website.  On our way we will wade through various layers using code examples.  The website itself contains a few paragraphs of text, some link lists, and our published papers in pdf form.</p>
 <p><em>Spoiler alert</em> final result can be seen <a href="https://nqsb.io">here</a>, the full code <a href="https://github.com/mirleft/nqsb.io">here</a>.</p>
-<h2>A first idea</h2>
+<h2 id="a-first-idea">A first idea</h2>
 <p>We could go all the way to use <a href="https://github.com/mirage/ocaml-conduit">conduit</a> for wrapping connections, and <a href="https://github.com/mirage/mirage-http">mirage-http</a> (using <a href="https://github.com/mirage/ocaml-cohttp">cohttp</a>, a very lightweight HTTP server).  We'd just need to write routing code which in the end reads from a virtual file system, and some HTML and CSS for the actual site.</p>
 <p>Turns out, the conduit library is already 1.7 MB in size and depends on 34 libraries, cohttp is another 3.7 MB and 40 dependent libraries.
 Both libraries are actively developed, combined there were 25 releases within the last year.</p>
-<h2>Plan</h2>
+<h2 id="plan">Plan</h2>
 <p>Let me state our demands more clearly:</p>
 <ul>
 <li>easy to maintain
@ -17,7 +17,7 @@ Both libraries are actively developed, combined there were 25 releases within th
 </li>
 </ul>
 <p>To achieve easy maintenance we keep build and run time dependencies small, use a single virtual machine image to ease deployment.  We try to develop only little new code.  Our general approach to performance is to do as little work as we can on each request, and precompute at compile time or once at startup as much as we can.</p>
-<h3>HTML code</h3>
+<h3 id="html-code">HTML code</h3>
 <p>From the <a href="http://ocsigen.org/tyxml/">tyxml description</a>: &quot;Tyxml provides a set of combinators to build Html5 and Svg documents. These combinators use the OCaml type-system to ensure the validity of the generated Html5 and Svg.&quot;  A <a href="http://ocsigen.org/tyxml/dev/manual/intro">tutorial</a> is available.</p>
 <p>You can plug elements (or attributes) inside each other only if the HTML specification allows this (no <code>&lt;body&gt;</code> inside of a <code>&lt;body&gt;</code>).  An example that can be rendered to a <code>div</code> with <code>pcdata</code> inside.</p>
 <p>If you use <code>utop</code> (as interactive read-eval-print-loop), you first need to load tyxml by <code>#require &quot;tyxml&quot;</code>.</p>
@ -51,7 +51,7 @@ Our full page source (CSS embedding is done via a string, no fancy types there (
    (body [ mycontent ]) ;
  Cstruct.of_string @@ Buffer.contents buf
 </code></pre>
-<h3>Binary data</h3>
+<h3 id="binary-data">Binary data</h3>
 <p>There are various ways how to embed binary data into MirageOS:</p>
 <ul>
 <li>connect an external (FAT) disk image; upside: works for large data, independent, can be shared with other systems; downside: an extra file to distribute onto the production machine, lots of code (block storage and file system access) which can contain directory traversals and other issues
@ -86,7 +86,7 @@ let start kv =
 </code></pre>
 <p>The funny <code>&gt;&gt;=</code> syntax notes that something is doing input/output, which might be blocking or interrupted or failing.  It composes effects using an imperative style (semicolon in other languages, another term is monadic bind).  The <code>Page.render</code> function is described above, and is pure, thus no <code>&gt;&gt;=</code>.</p>
 <p>We now have all the resources we wanted available inside our MirageOS unikernel.  There is some cost during configuration (converting binary into code), and startup (concatenating lists, lookups, rendering HTML into string representation).</p>
-<h3>Building a HTTP response</h3>
+<h3 id="building-a-http-response">Building a HTTP response</h3>
 <p>HTTP consists of headers and data, we already have the data.  A HTTP header contains of an initial status line (<code>HTTP/1.1 200 OK</code>), and a list of keys-values, each of the form <code>key + &quot;: &quot; + value + &quot;\r\n&quot;</code> (<code>+</code> is string concatenation).   The header is separated with <code>&quot;\r\n\r\n&quot;</code> from the data:</p>
 <pre><code class="language-OCaml">let http_header ~status xs =
  let headers = List.map (fun (k, v) -&gt; k ^ &quot;: &quot; ^ v) xs in
@ -98,7 +98,7 @@ let header content_type =
 </code></pre>
 <p>We also know statically (at compile time) which headers to send: <code>content-type</code> should be <code>text/html</code> for our main page, and <code>application/pdf</code> for the pdf files.  The status code <code>200</code> is used in HTTP to signal that the request is successful.  We can combine the headers and the data during startup, because our single communication channel is HTTP, and thus we don't need access to the data or headers separately (support for HTTP caching etc. is out of scope).</p>
 <p>We are now finished with the response side of HTTP, and can emit three different resources.  Now, we need to handle incoming HTTP requests and dispatch them to the resource.  Let's first to a brief detour to HTTPS (and thus TLS).</p>
-<h3>Security via HTTPS</h3>
+<h3 id="security-via-https">Security via HTTPS</h3>
 <p>Transport layer security is a protocol on top of TCP providing an end-to-end encrypted and authenticated channel.  In our setting, our web server has a certificate and a private key to authenticate itself to the clients.</p>
 <p>A certificate is a token containing a public key, a name, a validity period, and a signature from the authority which issued the certificate.  The authority is crucial here: this infrastructure only works if the client trusts the public key of the authority (and thus can verify their signature on our certificate).  I used <a href="https://letsencrypt.org/">let's encrypt</a> (actually the <a href="https://github.com/lukas2511/letsencrypt.sh/">letsencrypt.sh</a> client (would be great to have one natively in OCaml) to get a signed certificate, which is widely accepted by web browsers.</p>
 <p>The MirageOS interface for TLS is that it takes a <a href="https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L108"><code>FLOW</code></a> (byte stream, e.g. TCP) and provides a <code>FLOW</code>.  Libraries can be written to be agnostic whether they use a TCP stream or a TLS session to carry data.</p>
@ -117,14 +117,14 @@ let header content_type =
 </code></pre>
 <p>The <code>server_of_flow</code> is provided by the <a href="https://mirleft.github.io/ocaml-tls/Tls_mirage.Make.html#VALserver_of_flow">Mirage TLS layer</a>.  It can fail, if the client and server do not speak a common protocol suite, ciphersuite, or if one of the sides behaves non-protocol compliant.</p>
 <p>To wrap up, we managed to listen on the HTTPS port and establish TLS sessions for each incoming request.  We have our resources available, and now need to dispatch the request onto the resource.</p>
-<h3>HTTP request handling</h3>
+<h3 id="http-request-handling">HTTP request handling</h3>
 <p>HTTP is a string based protocol, the first line of the request contains the method and resource which the client wants to access, <code>GET / HTTP/1.1</code>.  We could read a single line from the client, and cut the part between <code>GET</code> and <code>HTTP/1.1</code> to deliver the resource.</p>
 <p>This would either need some (brittle?) string processing, or a full-blown HTTP library on our side.  &quot;I'm sorry Dave, I can't do that&quot;.  There is no way we'll do string processing on data received from the network for this.</p>
 <p>Looking a bit deeper into TLS, there is a specification for <a href="https://tools.ietf.org/html/rfc3546#section-3.1">server name indication</a> from 2003.  The main purpose is to run on a single IP(v4) address multiple TLS services.  The client indicates in the TLS handshake what the server name is it wants to talk to.  This extension looks <a href="https://en.wikipedia.org/wiki/Server_Name_Indication">in wikipedia</a> widely enough deployed.</p>
 <p>During the TLS handshake there is already some server name information exposed, and we have a very small set of available resources.  Thanks to let's encrypt, generating certificates is easy and free of cost.</p>
 <p>And, if we're down to a single resource, we can use the same technique used by <a href="https://github.com/pqwy">David</a> in
 the <a href="http://ownme.ipredator.se">BTC Piñata</a>: just send the resource back without waiting for a request.</p>
-<h3>Putting it all together</h3>
+<h3 id="putting-it-all-together">Putting it all together</h3>
 <p>What we need is a hostname for each resource, and certificates and private keys for them, or a single certificate with all hostnames as alternative names.</p>
 <p>Our TLS library supports to select a certificate chain based on the requested name (look <a href="https://mirleft.github.io/ocaml-tls/Config.html#TYPEown_cert">here</a>).  The following snippet is a setup to use the <code>nqsb.io</code> certificate chain by default (if no SNI is provided, or none matches), and also have a <code>usenix15</code> and a <code>tron</code> certificate chain.</p>
 <pre><code class="language-OCaml">let start stack keys kv =
@ -171,7 +171,7 @@ the <a href="http://ownme.ipredator.se">BTC Piñata</a>: just send the resource
  S.listen stack
 </code></pre>
 <p>That's it, the <a href="https://nqsb.io">nqsb.io</a> contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a <code>301 Moved permanently</code> HTTP status code).</p>
-<h2>Conclusion</h2>
+<h2 id="conclusion">Conclusion</h2>
 <p>A comparison using Firefox builtin network diagnostics shows that the waiting before receiving data is minimal (3ms, even spotted 0ms).</p>
 <p><a href="/static/img/performance-nqsbio.png"><img src="/static/img/performance-nqsbio.png" title="latency of our website" width="50%" /></a></p>
 <p>We do not render HTML for each request, we do not splice data together, we <em>don't even read the client request</em>.  And I'm sure we can improve the performance even more by profiling.</p>
@ -180,7 +180,7 @@ the <a href="http://ownme.ipredator.se">BTC Piñata</a>: just send the resource
 <p>For a start in MirageOS unikernels, look into our <a href="https://github.com/mirage/mirage-skeleton">mirage-skeleton</a> project, and into the <a href="https://github.com/mattgray/devwinter2016/">/dev/winter</a> presentation by Matt Gray.</p>
 <p>I'm interested in feedback, either via
 <a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
-<h2>Other updates in the MirageOS ecosystem</h2>
+<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
 <ul>
 <li><a href="https://github.com/Engil/Canopy">Canopy</a> improvements: <a href="https://github.com/Engil/Canopy/pull/26">no bower anymore</a>,  <a href="https://github.com/Engil/Canopy/pull/27">HTTP caching support (via etags)</a>, <a href="https://github.com/Engil/Canopy/pull/31">listings now include dates</a>, <a href="https://github.com/Engil/Canopy/pull/32">dates are now in big-endian (y-m-d)</a>
 </li>
--- a/202
+++ b/202
@ -1,8 +1,8 @@
-<feed xmlns="http://www.w3.org/2005/Atom"><link href="https://hannes.robur.coop/atom" rel="self"/><id>urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156</id><title type="text">full stack engineer</title><updated>2023-05-16T17:21:47-00:00</updated><entry><summary type="text">&lt;p&gt;fleet management for MirageOS unikernels using a mutually authenticated TLS handshake&lt;/p&gt;
+<feed xmlns="http://www.w3.org/2005/Atom"><link href="https://hannes.robur.coop/atom" rel="self"/><id>urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156</id><title type="text">full stack engineer</title><updated>2023-05-16T17:21:47-00:00</updated><entry><summary type="html">&lt;p&gt;fleet management for MirageOS unikernels using a mutually authenticated TLS handshake&lt;/p&gt;
 </summary><published>2022-11-17T12:41:11-00:00</published><link href="/Posts/Albatross" rel="alternate"/><content type="html">&lt;p&gt;EDIT (2023-05-16): Updated with albatross release version 2.0.0.&lt;/p&gt;
-&lt;h2&gt;Deploying MirageOS unikernels&lt;/h2&gt;
+&lt;h2 id=&quot;deploying-mirageos-unikernels&quot;&gt;Deploying MirageOS unikernels&lt;/h2&gt;
 &lt;p&gt;More than five years ago, I posted &lt;a href=&quot;/Posts/VMM&quot;&gt;how to deploy MirageOS unikernels&lt;/a&gt;. My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described &lt;a href=&quot;/Posts/Deploy&quot;&gt;how to deploy reproducible unikernels&lt;/a&gt;.&lt;/p&gt;
-&lt;h2&gt;Albatross&lt;/h2&gt;
+&lt;h2 id=&quot;albatross&quot;&gt;Albatross&lt;/h2&gt;
 &lt;p&gt;In recent months we worked hard on the underlying infrastructure: &lt;a href=&quot;https://github.com/roburio/albatross&quot;&gt;albatross&lt;/a&gt;. Albatross is the orchestration system for MirageOS unikernels that use solo5 with &lt;a href=&quot;https://github.com/Solo5/solo5/blob/master/docs/architecture.md&quot;&gt;hvt or spt tender&lt;/a&gt;. It deals with three tasks:&lt;/p&gt;
 &lt;ul&gt;
 &lt;li&gt;unikernel creation (destroyal, restart)
@ -13,50 +13,50 @@
 &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;An addition to the above is dealing with multiple tenants on the same machine: remote management of your unikernel fleet via TLS, and resource policies.&lt;/p&gt;
-&lt;h2&gt;History&lt;/h2&gt;
+&lt;h2 id=&quot;history&quot;&gt;History&lt;/h2&gt;
 &lt;p&gt;The initial commit of albatross was in May 2017. Back then it replaced the shell scripts and manual &lt;code&gt;scp&lt;/code&gt; of unikernel images to the server. Over time it evolved and adapted to new environments. Initially a solo5 unikernel would only know of a single network interface, these days there can be multiple distinguished by name. Initially there was no support for block devices. Only FreeBSD was supported in the early days. Nowadays we built daily packages for Debian, Ubuntu, FreeBSD, and have support for NixOS, and the client side is supported on macOS as well.&lt;/p&gt;
-&lt;h3&gt;ASN.1&lt;/h3&gt;
+&lt;h3 id=&quot;asn.1&quot;&gt;ASN.1&lt;/h3&gt;
 &lt;p&gt;The communication format between the albatross daemons and clients was changed multiple times. I'm glad that albatross uses ASN.1 as communication format, which makes extension with optional fields easy, and also allows &amp;quot;choice&amp;quot; (the sum type) to be not tagged (the binary is the same as no choice type), thus adding choice to an existing grammar, and preserving the old in the default (untagged) case is a decent solution.&lt;/p&gt;
 &lt;p&gt;So, if you care about backward and forward compatibility, as we do, since we may be in control of which albatross servers are deployed on our machine, but not what albatross versions the clients are using -- it may be wise to look into ASN.1. Recent efforts (json with schema, ...) may solve similar issues, but ASN.1 is as well very tiny in size.&lt;/p&gt;
-&lt;h2&gt;What resources does a unikernel need?&lt;/h2&gt;
+&lt;h2 id=&quot;what-resources-does-a-unikernel-need&quot;&gt;What resources does a unikernel need?&lt;/h2&gt;
 &lt;p&gt;A unikernel is just an operating system for a single service, there can't be much it can need.&lt;/p&gt;
-&lt;h3&gt;Name&lt;/h3&gt;
+&lt;h3 id=&quot;name&quot;&gt;Name&lt;/h3&gt;
 &lt;p&gt;So, first of all a unikernel has a name, or a handle. This is useful for reporting statistics, but also to specify which console output you're interested in. The name is a string with printable ASCII characters (and dash '-' and dot '.'), with a length up to 64 characters - so yes, you can use an UUID if you like.&lt;/p&gt;
-&lt;h3&gt;Memory&lt;/h3&gt;
+&lt;h3 id=&quot;memory&quot;&gt;Memory&lt;/h3&gt;
 &lt;p&gt;Another resource is the amount of memory assigned to the unikernel. This is specified in megabyte (as solo5 does), with the range being 10 (below not even a hello world wants to start) to 1024.&lt;/p&gt;
-&lt;h3&gt;Arguments&lt;/h3&gt;
+&lt;h3 id=&quot;arguments&quot;&gt;Arguments&lt;/h3&gt;
 &lt;p&gt;Of course, you can pass via albatross boot parameters to the unikernel. Albatross doesn't impose any restrictions here, but the lower levels may.&lt;/p&gt;
-&lt;h3&gt;CPU&lt;/h3&gt;
+&lt;h3 id=&quot;cpu&quot;&gt;CPU&lt;/h3&gt;
 &lt;p&gt;Due to multiple tenants, and side channel attacks, it looked right at the beginning like a good idea to restrict each unikernel to a specific CPU. This way, one tenant may use CPU 5, and another CPU 9 - and they'll not starve each other (best to make sure that these CPUs are in different packages). So, albatross takes a number as the CPU, and executes the solo5 tender within &lt;code&gt;taskset&lt;/code&gt;/&lt;code&gt;cpuset&lt;/code&gt;.&lt;/p&gt;
-&lt;h3&gt;Fail behaviour&lt;/h3&gt;
+&lt;h3 id=&quot;fail-behaviour&quot;&gt;Fail behaviour&lt;/h3&gt;
 &lt;p&gt;In normal operations, exceptional behaviour may occur. I have to admit that I've seen MirageOS unikernels that suffer from not freeing all the memory they have allocated. To avoid having to get up at 4 AM just to start the unikernel that went out of memory, there's the possibility to restart the unikernel when it exited. You can even specify on which exit codes it should be restarted (the exit code is the only piece of information we have from the outside what caused the exit). This feature was implemented in October 2019, and has been very precious since then. :)&lt;/p&gt;
-&lt;h3&gt;Network&lt;/h3&gt;
+&lt;h3 id=&quot;network&quot;&gt;Network&lt;/h3&gt;
 &lt;p&gt;This becomes a bit more complex: a MirageOS unikernel can have network interfaces, and solo5 specifies a so-called manifest with a list of these (name and type, and type is so far always basic). Then, on the actual server there are bridges (virtual switches) configured. Now, these may have the same name, or may need to be mapped. And of course, the unikernel expects a tap interface that is connected to such a bridge, not the bridge itself. Thus, albatross creates tap devices, attaches these to the respective bridges, and takes care about cleaning them up on teardown. The albatross client verifies that for each network interface in the manifest, there is a command-line argument specified (&lt;code&gt;--net service:my_bridge&lt;/code&gt; or just &lt;code&gt;--net service&lt;/code&gt; if the bridge is named service). The tap interface name is not really of interest to the user, and will not be exposed.&lt;/p&gt;
-&lt;h3&gt;Block devices&lt;/h3&gt;
+&lt;h3 id=&quot;block-devices&quot;&gt;Block devices&lt;/h3&gt;
 &lt;p&gt;On the host system, it's just a file, and passed to the unikernel. There's the need to be able to create one, dump it, and ensure that each file is only used by one unikernel. That's all that is there.&lt;/p&gt;
-&lt;h2&gt;Metrics&lt;/h2&gt;
+&lt;h2 id=&quot;metrics&quot;&gt;Metrics&lt;/h2&gt;
 &lt;p&gt;Everyone likes graphs, over time, showing how much traffic or CPU or memory or whatever has been used by your service. Some of these statistics are only available in the host system, and it is also crucial for development purposes to compare whether the bytes sent in the unikernel sum up to the same on the host system's tap interface.&lt;/p&gt;
 &lt;p&gt;The albatross-stats daemon collects metrics from three sources: network interfaces, getrusage (of a child process), VMM debug counters (to count VM exits etc.). Since the recent 1.5.3, albatross-stats now connects at startup to the albatross-daemon and then retrieves the information which unikernels are up and running, and starts periodically collecting data in memory.&lt;/p&gt;
 &lt;p&gt;Other clients, being it a dump on your console window, a write into an rrd file (good old MRTG times), or a push to influx, can use the stats data to correlate and better analyse what is happening on the grand scale of things. This helped a lot by running several unikernels with different opam package sets to figure out which opam packages leave their hands on memory over time.&lt;/p&gt;
 &lt;p&gt;As a side note, if you make the unikernel name also available in the unikernel, it can tag its own metrics with the same identifier, and you can correlate high-level events (such as amount of HTTP requests) with low-level things &amp;quot;allocated more memory&amp;quot; or &amp;quot;consumed a lot of CPU&amp;quot;.&lt;/p&gt;
-&lt;h2&gt;Console&lt;/h2&gt;
+&lt;h2 id=&quot;console&quot;&gt;Console&lt;/h2&gt;
 &lt;p&gt;There's not much to say about the console, just that the albatross-console daemon is running with low privileges, and reading from a FIFO that the unikernel writes to. It never writes anything to disk, but keeps the last 1000 lines in memory, available from a client asking for it.&lt;/p&gt;
-&lt;h2&gt;The daemons&lt;/h2&gt;
+&lt;h2 id=&quot;the-daemons&quot;&gt;The daemons&lt;/h2&gt;
 &lt;p&gt;So, the main albatross-daemon runs with superuser privileges to create virtual machines, and opens a unix domain socket where the clients and other daemons are connecting to. The other daemons are executed with normal user privileges, and never write anything to disk.&lt;/p&gt;
 &lt;p&gt;The albatross-daemon keeps state about the running unikernels, and if it is restarted, the unikernels are started again. Maybe worth to mention that this lead sometimes to headaches (due to data being dumped to disk, and the old format should always be supported), but was also a huge relief to not have to care about creating all the unikernels just because albatross-daemon was killed.&lt;/p&gt;
-&lt;h2&gt;Remote management&lt;/h2&gt;
+&lt;h2 id=&quot;remote-management&quot;&gt;Remote management&lt;/h2&gt;
 &lt;p&gt;There's one more daemon program: albatross-tls-endpoint. It accepts clients via a remote TCP connection, and establish a mutual-authenticated TLS handshake. When done, the command is forwarded to the respective Unix domain socket, and the reply is sent back to the client.&lt;/p&gt;
 &lt;p&gt;The daemon itself has a X.509 certificate to authenticate, but the client is requested to show its certificate chain as well. This by now requires TLS 1.3, so the client certificates are sent over the encrypted channel.&lt;/p&gt;
 &lt;p&gt;A step back, X.509 certificate contains a public key and a signature from one level up. When the server knows about the root (or certificate authority (CA)) certificate, and following the chain can verify that the leaf certificate is valid. Additionally, a X.509 certificate is a ASN.1 structure with some fixed fields, but also contains extensions, a key-value store where the keys are object identifiers, and the values are key-dependent data. Also note that this key-value store is cryptographically signed.&lt;/p&gt;
 &lt;p&gt;Albatross uses the object identifier, assigned to Camelus Dromedarius (MirageOS - 1.3.6.1.4.1.49836.42) to encode the command to be executed. This means that once the TLS handshake is established, the command to be executed is already transferred.&lt;/p&gt;
 &lt;p&gt;In the leaf certificate, there may be the &amp;quot;create unikernel&amp;quot; command with the unikernel image, it's boot parameters, and other resources. Or a &amp;quot;read the console of my unikernel&amp;quot;. In the intermediate certificates (from root to leaf), resource policies are encoded (this path may only have X unikernels running with a total of Y MB memory, and Z MB of block storage, using CPUs A and B, accessing bridges C and D). From the root downwards these policies may only decrease. When a unikernel should be created (or other commands are executed), the policies are verified to hold. If they do not, an error is reported.&lt;/p&gt;
-&lt;h2&gt;Fleet management&lt;/h2&gt;
+&lt;h2 id=&quot;fleet-management&quot;&gt;Fleet management&lt;/h2&gt;
 &lt;p&gt;Of course it is very fine to create your locally compiled unikernel to your albatross server, go for it. But in terms of &amp;quot;what is actually running here?&amp;quot; and &amp;quot;does this unikernel need to be updated because some opam package had a security issues?&amp;quot;, this is not optimal.&lt;/p&gt;
 &lt;p&gt;Since we provide &lt;a href=&quot;https://builds.robur.coop&quot;&gt;daily reproducible builds&lt;/a&gt; with the current HEAD of the main opam-repository, and these unikernels have no configuration embedded (but take everything as boot parameters), we just deploy them. They come with the information what opam packages contributed to the binary, which environment variables were set, and which system packages were installed with which versions.&lt;/p&gt;
 &lt;p&gt;The whole result of reproducible builds for us means: we have a hash of a unikernel image that we can lookup in our build infrastructure, and take a look whether there is a newer image for the same job. And if there is, we provide a diff between the packages contributed to the currently running unikernel and the new image. That is what the albatross-client update command is all about.&lt;/p&gt;
 &lt;p&gt;Of course, your mileage may vary and you want automated deployments where each git commit triggers recompilation and redeployment. The downside would be that sometimes only dependencies are updated and you've to cope with that.&lt;/p&gt;
 &lt;p&gt;There is a client &lt;code&gt;albatross-client&lt;/code&gt;, depending on arguments either connects to a local Unix domain socket, or to a remote albatross instance via TCP and TLS, or outputs a certificate signing request for later usage. Data, such as the unikernel ELF image, is compressed in certificates.&lt;/p&gt;
-&lt;h2&gt;Installation&lt;/h2&gt;
+&lt;h2 id=&quot;installation&quot;&gt;Installation&lt;/h2&gt;
 &lt;p&gt;For Debian and Ubuntu systems, we provide package repositories. Browse the dists folder for one matching your distribution, and add it to &lt;code&gt;/etc/apt/sources.list&lt;/code&gt;:&lt;/p&gt;
 &lt;pre&gt;&lt;code&gt;$ wget -q -O /etc/apt/trusted.gpg.d/apt.robur.coop.gpg https://apt.robur.coop/gpg.pub
 $ echo &amp;quot;deb https://apt.robur.coop ubuntu-20.04 main&amp;quot; &amp;gt;&amp;gt; /etc/apt/sources.list # replace ubuntu-20.04 with e.g. debian-11 on a debian buster machine
@ -77,28 +77,28 @@ $ pkg install solo5 albatross
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;Please ensure to have at least version 2.0.0 of albatross installed.&lt;/p&gt;
 &lt;p&gt;For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (&lt;code&gt;opam install solo5 albatross&lt;/code&gt;). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.&lt;/p&gt;
-&lt;h2&gt;Conclusion&lt;/h2&gt;
+&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
 &lt;p&gt;After five years of development and operating albatross, feel free to get it and try it out. Or read the code, discuss issues and shortcomings with us - either at the issue tracker or via eMail.&lt;/p&gt;
 &lt;p&gt;Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donations&lt;/a&gt; for doing our work - everyone can contribute.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:1f354218-e8c3-5136-a2ca-c88f3c2878d8</id><title type="text">Deploying reproducible unikernels with albatross</title><updated>2023-05-16T17:21:47-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;Re-developing an opam cache from scratch, as a MirageOS unikernel&lt;/p&gt;
+</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:1f354218-e8c3-5136-a2ca-c88f3c2878d8</id><title type="text">Deploying reproducible unikernels with albatross</title><updated>2023-05-16T17:21:47-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;Re-developing an opam cache from scratch, as a MirageOS unikernel&lt;/p&gt;
 </summary><published>2022-09-29T13:04:14-00:00</published><link href="/Posts/OpamMirror" rel="alternate"/><content type="html">&lt;p&gt;We at &lt;a href=&quot;https://robur.coop&quot;&gt;robur&lt;/a&gt; developed &lt;a href=&quot;https://git.robur.io/robur/opam-mirror&quot;&gt;opam-mirror&lt;/a&gt; in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).&lt;/p&gt;
-&lt;h1&gt;What is opam and why should I care?&lt;/h1&gt;
+&lt;h1 id=&quot;what-is-opam-and-why-should-i-care&quot;&gt;What is opam and why should I care?&lt;/h1&gt;
 &lt;p&gt;&lt;a href=&quot;https://opam.ocaml.org&quot;&gt;Opam&lt;/a&gt; is the OCaml package manager (also used by other projects such as &lt;a href=&quot;https://coq.inria.fr&quot;&gt;coq&lt;/a&gt;). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as &lt;a href=&quot;https://github.com/ocaml/opam-repository&quot;&gt;ocaml/opam-repository&lt;/a&gt;, where authors of OCaml software can contribute (as pull request) their latest releases.&lt;/p&gt;
 &lt;p&gt;When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state &amp;quot;my package will not work with &lt;a href=&quot;https://erratique.ch/software/cmdliner&quot;&gt;cmdliner&lt;/a&gt; in version 1.2.0&amp;quot;).&lt;/p&gt;
 &lt;p&gt;So, the opam-repository holds the metadata of lots of OCaml packages (around 4000 at the moment this article was written) with lots of versions (in total 25000) that have been released. It is used by the opam client to figure out which packages to install or upgrade (using a solver that takes the version bounds into consideration).&lt;/p&gt;
 &lt;p&gt;Of course, opam can use other repositories (overlays) or forks thereof. So nothing stops you from using any other opam repository. The url to the source code of each package may be a tarball, or a git repository or other version control systems.&lt;/p&gt;
 &lt;p&gt;The vast majority of opam packages released to the opam-repository include a link to the source tarball and a cryptographic hash of the tarball. This is crucial for security (under the assumption the opam-repository has been downloaded from a trustworthy source - check back later this year for updates on &lt;a href=&quot;/Posts/Conex&quot;&gt;conex&lt;/a&gt;). At the moment, there are some weak spots in respect to security: md5 is still allowed, and the hash and the tarball are downloaded from the same server: anyone who is in control of that server can inject arbitrary malicious data. As outlined above, we're working on infrastructure which fixes the latter issue.&lt;/p&gt;
-&lt;h1&gt;How does the opam client work?&lt;/h1&gt;
+&lt;h1 id=&quot;how-does-the-opam-client-work&quot;&gt;How does the opam client work?&lt;/h1&gt;
 &lt;p&gt;Opam, after initialisation, downloads the &lt;code&gt;index.tar.gz&lt;/code&gt; from &lt;code&gt;https://opam.ocaml.org/index.tar.gz&lt;/code&gt;, and uses this as the local opam universe. An &lt;code&gt;opam install cmdliner&lt;/code&gt; will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: &lt;code&gt;https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e&lt;/code&gt;.&lt;/p&gt;
-&lt;h1&gt;How does the opam repository work?&lt;/h1&gt;
+&lt;h1 id=&quot;how-does-the-opam-repository-work&quot;&gt;How does the opam repository work?&lt;/h1&gt;
 &lt;p&gt;According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses &lt;code&gt;opam admin index&lt;/code&gt; periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was &lt;a href=&quot;https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/&quot;&gt;a reported downtime&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS &lt;a href=&quot;http://retreat.mirage.io&quot;&gt;hack retreats&lt;/a&gt; in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).&lt;/p&gt;
-&lt;h1&gt;Re-developing &amp;quot;opam admin create&amp;quot; as MirageOS unikernel&lt;/h1&gt;
+&lt;h1 id=&quot;re-developing-opam-admin-create-as-mirageos-unikernel&quot;&gt;Re-developing &amp;quot;opam admin create&amp;quot; as MirageOS unikernel&lt;/h1&gt;
 &lt;p&gt;The need for a local opam cache at our &lt;a href=&quot;https://builds.robur.coop&quot;&gt;reproducible build infrastructure&lt;/a&gt; and the retreats, we decided to develop &lt;a href=&quot;https://git.robur.io/robur/opam-mirror&quot;&gt;opam-mirror&lt;/a&gt; as a &lt;a href=&quot;https://mirage.io&quot;&gt;MirageOS unikernel&lt;/a&gt;. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the &lt;code&gt;opam admin index&lt;/code&gt; is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.&lt;/p&gt;
 &lt;p&gt;Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.&lt;/p&gt;
 &lt;p&gt;In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with &lt;a href=&quot;https://github.com/mirage/ocaml-crunch&quot;&gt;crunch&lt;/a&gt;, or using a remote git repository), but recent development, e.g. of &lt;a href=&quot;https://somerandomidiot.com/blog/2022/03/04/chamelon/&quot;&gt;chamelon&lt;/a&gt; sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.&lt;/p&gt;
 &lt;p&gt;Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.&lt;/p&gt;
-&lt;h1&gt;Code development and improvements&lt;/h1&gt;
+&lt;h1 id=&quot;code-development-and-improvements&quot;&gt;Code development and improvements&lt;/h1&gt;
 &lt;p&gt;Initially, our plan was to use &lt;a href=&quot;https://github.com/mirage/ocaml-git&quot;&gt;ocaml-git&lt;/a&gt; for pulling the repository, &lt;a href=&quot;https://github.com/yomimono/chamelon&quot;&gt;chamelon&lt;/a&gt; for persistent storage, and &lt;a href=&quot;https://github.com/inhabitedtype/httpaf&quot;&gt;httpaf&lt;/a&gt; as web server. With &lt;a href=&quot;https://github.com/mirage/ocaml-tar&quot;&gt;ocaml-tar&lt;/a&gt; recent support of &lt;a href=&quot;https://github.com/mirage/ocaml-tar/pull/88&quot;&gt;gzip&lt;/a&gt; we should be all set, and done within a few days.&lt;/p&gt;
 &lt;p&gt;There is already a gap in the above plan: which http client to use - in the best case something similar to our &lt;a href=&quot;https://github.com/roburio/http-lwt-client&quot;&gt;http-lwt-client&lt;/a&gt; - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using &lt;a href=&quot;https://github.com/roburio/happy-eyeballs&quot;&gt;happy-eyeballs&lt;/a&gt; to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.&lt;/p&gt;
 &lt;p&gt;On the path (over the last month), we fixed file descriptor leaks (memory leaks) in &lt;a href=&quot;https://github.com/dinosaure/paf-le-chien&quot;&gt;paf&lt;/a&gt; -- which is used as a runtime for httpaf and h2.&lt;/p&gt;
@ -108,20 +108,20 @@ $ pkg install solo5 albatross
 &lt;p&gt;Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as &lt;a href=&quot;https://github.com/mirage/ocaml-fat&quot;&gt;fat&lt;/a&gt; may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed &lt;a href=&quot;https://github.com/reynir/mirage-block-partition&quot;&gt;mirage-block-partition&lt;/a&gt; to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed &lt;a href=&quot;https://git.robur.io/reynir/oneffs&quot;&gt;oneffs&lt;/a&gt; which stores a single value on the block device.&lt;/p&gt;
 &lt;p&gt;We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains &lt;a href=&quot;https://github.com/mirage/mirage-kv/pull/28&quot;&gt;recent enhancements&lt;/a&gt; of the mirage-kv API, there is a &lt;code&gt;get_partial&lt;/code&gt;, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).&lt;/p&gt;
 &lt;p&gt;What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy &lt;a href=&quot;https://github.com/roburio/tlstunnel&quot;&gt;tlstunnel&lt;/a&gt;) similar to how &lt;a href=&quot;https://github.com/roburio/unipi&quot;&gt;unipi&lt;/a&gt; does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).&lt;/p&gt;
-&lt;h1&gt;Conclusion&lt;/h1&gt;
+&lt;h1 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h1&gt;
 &lt;p&gt;To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide &lt;a href=&quot;https://builds.robur.coop/job/opam-mirror&quot;&gt;reproducible binaries&lt;/a&gt; for solo5-hvt. You can use our opam mirror with &lt;code&gt;opam repository set-url default https://opam.robur.coop&lt;/code&gt; (revert to the other with &lt;code&gt;opam repository set-url default https://opam.ocaml.org&lt;/code&gt;) or use it as a backup with &lt;code&gt;opam repository add robur --rank 2 https://opam.robur.coop&lt;/code&gt;.&lt;/p&gt;
 &lt;p&gt;Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donations&lt;/a&gt; for doing our work - everyone can contribute.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/opam" term="opam"/><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:0dbd251f-32c7-57bd-8e8f-7392c0833a09</id><title type="text">Mirroring the opam repository and all tarballs</title><updated>2022-10-11T12:14:07-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;How to monitor your MirageOS unikernel with albatross and monitoring-experiments&lt;/p&gt;
-</summary><published>2022-03-08T11:26:31-00:00</published><link href="/Posts/Monitoring" rel="alternate"/><content type="html">&lt;h1&gt;Introduction to monitoring&lt;/h1&gt;
+</content><category scheme="https://hannes.robur.coop/tags/opam" term="opam"/><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:0dbd251f-32c7-57bd-8e8f-7392c0833a09</id><title type="text">Mirroring the opam repository and all tarballs</title><updated>2022-10-11T12:14:07-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;How to monitor your MirageOS unikernel with albatross and monitoring-experiments&lt;/p&gt;
+</summary><published>2022-03-08T11:26:31-00:00</published><link href="/Posts/Monitoring" rel="alternate"/><content type="html">&lt;h1 id=&quot;introduction-to-monitoring&quot;&gt;Introduction to monitoring&lt;/h1&gt;
 &lt;p&gt;At &lt;a href=&quot;https://robur.coop&quot;&gt;robur&lt;/a&gt; we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our &lt;a href=&quot;https://builds.robur.coop&quot;&gt;reproducible builds infrastructure&lt;/a&gt;. Another part is, once deployed we want to observe what is going on.&lt;/p&gt;
 &lt;p&gt;I first got into touch with monitoring - collecting and graphing metrics - with &lt;a href=&quot;https://oss.oetiker.ch/mrtg/&quot;&gt;MRTG&lt;/a&gt; and &lt;a href=&quot;https://munin-monitoring.org/&quot;&gt;munin&lt;/a&gt; - and the simple network management protocol &lt;a href=&quot;https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol&quot;&gt;SNMP&lt;/a&gt;. From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.&lt;/p&gt;
 &lt;p&gt;The rise of monitoring where graphs are done dynamically (such as &lt;a href=&quot;https://grafana.com/&quot;&gt;Grafana&lt;/a&gt;) and can be programmed (with a query language) by the operator are very neat, it allows to put metrics in relation after they have been recorded - thus if there's a thesis why something went berserk, you can graph the collected data from the past and prove or disprove the thesis.&lt;/p&gt;
-&lt;h1&gt;Monitoring a MirageOS unikernel&lt;/h1&gt;
+&lt;h1 id=&quot;monitoring-a-mirageos-unikernel&quot;&gt;Monitoring a MirageOS unikernel&lt;/h1&gt;
 &lt;p&gt;From the operational perspective, taking security into account - either the data should be authenticated and integrity-protected, or being transmitted on a private network. We chose the latter, there's a private network interface only for monitoring. Access to that network is only granted to the unikernels and metrics collector.&lt;/p&gt;
 &lt;p&gt;For MirageOS unikernels, we use the &lt;a href=&quot;https://github.com/mirage/metrics&quot;&gt;metrics&lt;/a&gt; library - which design shares the idea of &lt;a href=&quot;https://erratique.ch/software/logs&quot;&gt;logs&lt;/a&gt; that only if there's a reporter registered, work is performed.  We use the Influx line protocol via TCP to report via &lt;a href=&quot;https://www.influxdata.com/time-series-platform/telegraf/&quot;&gt;Telegraf&lt;/a&gt; to &lt;a href=&quot;https://www.influxdata.com/&quot;&gt;InfluxDB&lt;/a&gt;. But due to the design of &lt;a href=&quot;https://github.com/mirage/metrics&quot;&gt;metrics&lt;/a&gt;, other reporters can be developed and used -- prometheus, SNMP, your-other-favourite are all possible.&lt;/p&gt;
 &lt;p&gt;Apart from monitoring metrics, we use the same network interface for logging via syslog. Since the logs library separates the log message generation (in the OCaml libraries) from the reporting, we developed &lt;a href=&quot;https://github.com/hannesm/logs-syslog&quot;&gt;logs-syslog&lt;/a&gt;, which registers a log reporter sending each log message to a syslog sink.&lt;/p&gt;
 &lt;p&gt;We developed a small library for metrics reporting of a MirageOS unikernel into the &lt;a href=&quot;https://github.com/roburio/monitoring-experiments&quot;&gt;monitoring-experiments&lt;/a&gt; package - which also allows to dynamically adjust log level and disable or enable metrics sources.&lt;/p&gt;
-&lt;h2&gt;Required components&lt;/h2&gt;
+&lt;h2 id=&quot;required-components&quot;&gt;Required components&lt;/h2&gt;
 &lt;p&gt;Install from your operating system the packages providing telegraf, influxdb, and grafana.&lt;/p&gt;
 &lt;p&gt;Setup telegraf to contain a socket listener:&lt;/p&gt;
 &lt;pre&gt;&lt;code&gt;[[inputs.socket_listener]]
@ -131,7 +131,7 @@ $ pkg install solo5 albatross
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;Use a unikernel that reports to Influx (below the heading &amp;quot;Unikernels (with metrics reported to Influx)&amp;quot; on &lt;a href=&quot;https://builds.robur.coop&quot;&gt;builds.robur.coop&lt;/a&gt;) and provide &lt;code&gt;--monitor=192.168.42.14&lt;/code&gt; as boot parameter. Conventionally, these unikernels expect a second network interface (on the &amp;quot;management&amp;quot; bridge) where telegraf (and a syslog sink) are running. You'll need to pass &lt;code&gt;--net=management&lt;/code&gt; and &lt;code&gt;--arg='--management-ipv4=192.168.42.x/24'&lt;/code&gt; to albatross-client.&lt;/p&gt;
 &lt;p&gt;Albatross provides a &lt;code&gt;albatross-influx&lt;/code&gt; daemon that reports information from the host system about the unikernels to influx. Start it with &lt;code&gt;--influx=192.168.42.14&lt;/code&gt;.&lt;/p&gt;
-&lt;h2&gt;Adding monitoring to your unikernel&lt;/h2&gt;
+&lt;h2 id=&quot;adding-monitoring-to-your-unikernel&quot;&gt;Adding monitoring to your unikernel&lt;/h2&gt;
 &lt;p&gt;If you want to extend your own unikernel with metrics, follow along these lines.&lt;/p&gt;
 &lt;p&gt;An example is the &lt;a href=&quot;https://github.com/roburio/dns-primary-git&quot;&gt;dns-primary-git&lt;/a&gt; unikernel, where on the branch &lt;code&gt;future&lt;/code&gt; we have a single commit ahead of main that adds monitoring. The difference is in the unikernel configuration and the main entry point. See the &lt;a href=&quot;https://builds.robur.coop/job/dns-primary-git-monitoring/build/latest/&quot;&gt;binary builts&lt;/a&gt; in contrast to the &lt;a href=&quot;https://builds.robur.coop/job/dns-primary-git/build/latest/&quot;&gt;non-monitoring builts&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;In config, three new command line arguments are added: &lt;code&gt;--monitor=IP&lt;/code&gt;, &lt;code&gt;--monitor-adjust=PORT&lt;/code&gt; &lt;code&gt;--syslog=IP&lt;/code&gt; and &lt;code&gt;--name=STRING&lt;/code&gt;. In addition, the package &lt;code&gt;monitoring-experiments&lt;/code&gt; is required. And a second network interface &lt;code&gt;management_stack&lt;/code&gt; using the prefix &lt;code&gt;management&lt;/code&gt; is required and passed to the unikernel. Since the syslog reporter requires a console (to report when logging fails), also a console is passed to the unikernel. Each reported metrics includes a tag &lt;code&gt;vm=&amp;lt;name&amp;gt;&lt;/code&gt; that can be used to distinguish several unikernels reporting to the same InfluxDB.&lt;/p&gt;
@ -225,19 +225,19 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
 &lt;p&gt;With this, your unikernel will report metrics using the influx protocol to 192.168.42.14 on port 8094 (every 10 seconds), and syslog messages via UDP to 192.168.0.10 (port 514). You should see your InfluxDB getting filled and syslog server receiving messages.&lt;/p&gt;
 &lt;p&gt;When you configure &lt;a href=&quot;https://grafana.com/docs/grafana/latest/getting-started/getting-started-influxdb/&quot;&gt;Grafana to use InfluxDB&lt;/a&gt;, you'll be able to see the data in the data sources.&lt;/p&gt;
 &lt;p&gt;Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/monitoring" term="monitoring"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:b8f1fa5b-d8dd-5a54-a9e4-064b9dcd053e</id><title type="text">All your metrics belong to influx</title><updated>2023-05-16T17:21:47-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder&lt;/p&gt;
-</summary><published>2021-06-30T13:13:37-00:00</published><link href="/Posts/Deploy" rel="alternate"/><content type="html">&lt;h2&gt;Introduction&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/monitoring" term="monitoring"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:b8f1fa5b-d8dd-5a54-a9e4-064b9dcd053e</id><title type="text">All your metrics belong to influx</title><updated>2023-05-16T17:21:47-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder&lt;/p&gt;
+</summary><published>2021-06-30T13:13:37-00:00</published><link href="/Posts/Deploy" rel="alternate"/><content type="html">&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
 &lt;p&gt;MirageOS development focus has been a lot on tooling and the developer experience, but to accomplish &lt;a href=&quot;https://robur.coop&quot;&gt;our&lt;/a&gt; goal to &amp;quot;get MirageOS into production&amp;quot;, we need to lower the barrier. This means for us to release binary unikernels. As described &lt;a href=&quot;/Posts/NGI&quot;&gt;earlier&lt;/a&gt;, we received a grant for &amp;quot;Deploying MirageOS&amp;quot; from &lt;a href=&quot;https://pointer.ngi.eu&quot;&gt;NGI Pointer&lt;/a&gt; to work on the required infrastructure. This is joint work with &lt;a href=&quot;https://reynir.dk/&quot;&gt;Reynir&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;We provide at &lt;a href=&quot;https://builds.robur.coop&quot;&gt;builds.robur.coop&lt;/a&gt; binary unikernel images (and supplementary software). Doing binary releases of MirageOS unikernels is challenging in two aspects: firstly to be useful for everyone, a binary unikernel should not contain any configuration (such as private keys, certificates, etc.). Secondly, the binaries should be &lt;a href=&quot;https://reproducible-builds.org&quot;&gt;reproducible&lt;/a&gt;. This is crucial for security; everyone can reproduce the exact same binary and verify that our build service did only use the sources. No malware or backdoors included.&lt;/p&gt;
 &lt;p&gt;This post describes how you can deploy MirageOS unikernels without compiling it from source, then dives into the two issues outlined above - configuration and reproducibility - and finally describes how to setup your own reproducible build infrastructure for MirageOS, and how to bootstrap it.&lt;/p&gt;
-&lt;h2&gt;Deploying MirageOS unikernels from binary&lt;/h2&gt;
+&lt;h2 id=&quot;deploying-mirageos-unikernels-from-binary&quot;&gt;Deploying MirageOS unikernels from binary&lt;/h2&gt;
 &lt;p&gt;To execute a MirageOS unikernel, apart from a hypervisor (Xen/KVM/Muen), a tender (responsible for allocating host system resources and passing these to the unikernel) is needed. Using virtio, this is conventionally done with qemu on Linux, but its code size (and attack surface) is huge. For MirageOS, we develop &lt;a href=&quot;https://github.com/solo5/solo5&quot;&gt;Solo5&lt;/a&gt;, a minimal tender. It supports &lt;em&gt;hvt&lt;/em&gt; - hardware virtualization (Linux KVM, FreeBSD BHyve, OpenBSD VMM), &lt;em&gt;spt&lt;/em&gt; - sandboxed process (a tight seccomp ruleset (only a handful of system calls allowed, no hardware virtualization needed), Linux only). Apart from that, &lt;a href=&quot;https://muen.sk&quot;&gt;&lt;em&gt;muen&lt;/em&gt;&lt;/a&gt; (a hypervisor developed in Ada), &lt;em&gt;virtio&lt;/em&gt; (for some cloud deployments), and &lt;em&gt;xen&lt;/em&gt; (PVHv2 or Qubes 4.0) - &lt;a href=&quot;https://github.com/Solo5/solo5/blob/master/docs/building.md&quot;&gt;read more&lt;/a&gt;. We deploy our unikernels as hvt with FreeBSD BHyve as hypervisor.&lt;/p&gt;
 &lt;p&gt;On &lt;a href=&quot;https://builds.robur.coop&quot;&gt;builds.robur.coop&lt;/a&gt;, next to the unikernel images, &lt;a href=&quot;https://builds.robur.coop/job/solo5-hvt/&quot;&gt;&lt;em&gt;solo5-hvt&lt;/em&gt; packages&lt;/a&gt; are provided - download the binary and install it. A &lt;a href=&quot;https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/solo5&quot;&gt;NixOS package&lt;/a&gt; is already available - please note that &lt;a href=&quot;https://github.com/Solo5/solo5/pull/494&quot;&gt;soon&lt;/a&gt; packaging will be much easier (and we will work on packages merged into distributions).&lt;/p&gt;
 &lt;p&gt;When the tender is installed, download a unikernel image (e.g. the &lt;a href=&quot;https://builds.robur.coop/job/traceroute/build/latest/&quot;&gt;traceroute&lt;/a&gt; described in &lt;a href=&quot;/Posts/Traceroute&quot;&gt;an earlier post&lt;/a&gt;), and execute it:&lt;/p&gt;
 &lt;pre&gt;&lt;code&gt;$ solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;If you plan to orchestrate MirageOS unikernels, you may be interested in &lt;a href=&quot;https://github.com/roburio/albatross&quot;&gt;albatross&lt;/a&gt; - we provide &lt;a href=&quot;https://builds.robur.coop/job/albatross/&quot;&gt;binary packages as well for albatross&lt;/a&gt;. An upcoming post will go into further details of how to setup albatross.&lt;/p&gt;
-&lt;h2&gt;MirageOS configuration&lt;/h2&gt;
+&lt;h2 id=&quot;mirageos-configuration&quot;&gt;MirageOS configuration&lt;/h2&gt;
 &lt;p&gt;A MirageOS unikernel has a specific purpose - composed of OCaml libraries - selected at compile time, which allows to only embed the required pieces. This reduces the attack surface drastically. At the same time, to be widely useful to multiple organisations, no configuration data must be embedded into the unikernel.&lt;/p&gt;
 &lt;p&gt;Early MirageOS unikernels such as &lt;a href=&quot;https://github.com/mirage/mirage-www&quot;&gt;mirage-www&lt;/a&gt; embed content (blog posts, ..) and TLS certificates and private keys in the binary (using &lt;a href=&quot;https://github.com/mirage/ocaml-crunch&quot;&gt;crunch&lt;/a&gt;). The &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall&quot;&gt;Qubes firewall&lt;/a&gt; (read the &lt;a href=&quot;http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/&quot;&gt;blog post by Thomas&lt;/a&gt; for more information) used to include the firewall rules until &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall/releases/tag/v0.6&quot;&gt;v0.6&lt;/a&gt; in the binary, since &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall/tree/v0.7&quot;&gt;v0.7&lt;/a&gt; the rules are read dynamically from QubesDB. This is big usability improvement.&lt;/p&gt;
 &lt;p&gt;We have several possibilities to provide configuration information in MirageOS, on the one hand via boot parameters (can be pre-filled at development time, and further refined at configuration time, but those passed at boot time take precedence). Boot parameters have a length limitation.&lt;/p&gt;
@ -245,7 +245,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
 &lt;p&gt;Several other unikernels, such as &lt;a href=&quot;https://github.com/Engil/Canopy&quot;&gt;this website&lt;/a&gt; and &lt;a href=&quot;https://github.com/roburio/caldav&quot;&gt;our CalDAV server&lt;/a&gt;, store the content in a remote git repository. The git URI and credentials (private key seed, host key fingerprint) are passed via boot parameter.&lt;/p&gt;
 &lt;p&gt;Finally, another option that we take advantage of is to introduce a post-link step that rewrites the binary to embed configuration. The tool &lt;a href=&quot;https://github.com/dinosaure/caravan&quot;&gt;caravan&lt;/a&gt; developed by Romain that does this rewrite is used by our &lt;a href=&quot;https://github.com/roburio/openvpn/tree/robur/mirage-router&quot;&gt;openvpn router&lt;/a&gt; (&lt;a href=&quot;https://builds.robur.coop/job/openvpn-router/build/latest/&quot;&gt;binary&lt;/a&gt;).&lt;/p&gt;
 &lt;p&gt;In the future, some configuration information - such as monitoring system, syslog sink, IP addresses - may be done via DHCP on one of the private network interfaces - this would mean that the DHCP server has some global configuration option, and the unikernels no longer require that many boot parameters. Another option we want to investigate is where the tender shares a file as read-only memory-mapped region from the host system to the guest system - but this is tricky considering all targets above (especially virtio and muen).&lt;/p&gt;
-&lt;h2&gt;Behind the scenes: reproducible builds&lt;/h2&gt;
+&lt;h2 id=&quot;behind-the-scenes-reproducible-builds&quot;&gt;Behind the scenes: reproducible builds&lt;/h2&gt;
 &lt;p&gt;To provide a high level of assurance and trust, if you distribute binaries in 2021, you should have a recipe how they can be reproduced in a bit-by-bit identical way. This way, different organisations can run builders and rebuilders, and a user can decide to only use a binary if it has been reproduced by multiple organisations in different jurisdictions using different physical machines - to avoid malware being embedded in the binary.&lt;/p&gt;
 &lt;p&gt;For a reproduction to be successful, you need to collect the checksums of all sources that contributed to the built, together with other things (host system packages, environment variables, etc.). Of course, you can record the entire OS and sources as a tarball (or file system snapshot) and distribute that - but this may be suboptimal in terms of bandwidth requirements.&lt;/p&gt;
 &lt;p&gt;With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the &lt;code&gt;opam switch export&lt;/code&gt; includes &lt;a href=&quot;https://github.com/ocaml/opam/pull/4040&quot;&gt;extra-files (patches)&lt;/a&gt; and &lt;a href=&quot;https://github.com/ocaml/opam/pull/4055&quot;&gt;records the VCS version&lt;/a&gt;. Based on this functionality, &lt;a href=&quot;https://github.com/roburio/orb&quot;&gt;orb&lt;/a&gt;, an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are &lt;code&gt;PATH&lt;/code&gt; (sanitized with an allow list, &lt;code&gt;/bin&lt;/code&gt;, &lt;code&gt;/sbin&lt;/code&gt;, with &lt;code&gt;/usr&lt;/code&gt;, &lt;code&gt;/usr/local&lt;/code&gt;, and &lt;code&gt;/opt&lt;/code&gt; prefixes), and &lt;code&gt;HOME&lt;/code&gt;. To enable Debian's &lt;code&gt;apt&lt;/code&gt; to install packages, &lt;code&gt;DEBIAN_FRONTEND&lt;/code&gt; is set to &lt;code&gt;noninteractive&lt;/code&gt;. The &lt;code&gt;SWITCH_PATH&lt;/code&gt; is recorded to allow orb to use the same path during a rebuild. The &lt;code&gt;SOURCE_DATE_EPOCH&lt;/code&gt; is set to enable tools that record a timestamp to use a static one. The &lt;code&gt;OS*&lt;/code&gt; variables are only used for recording the host OS and version.&lt;/p&gt;
@ -278,15 +278,15 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
 &lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.&lt;/p&gt;
-&lt;h2&gt;Conclusion&lt;/h2&gt;
+&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
 &lt;p&gt;Thanks to NGI funding we now have reproducible MirageOS binary builds available at &lt;a href=&quot;https://builds.robur.coop&quot;&gt;builds.robur.coop&lt;/a&gt;. The underlying infrastructure is reproducible, available for multiple platforms (Ubuntu using docker, FreeBSD using jails), and can be easily bootstrapped from source (once you have OCaml and opam working, getting builder and orb should be easy). All components are open source software, mostly with permissive licenses.&lt;/p&gt;
 &lt;p&gt;We also have an index over sha-256 checksum of binaries - in the case you find a running unikernel image where you forgot which exact packages were used, you can do a reverse lookup.&lt;/p&gt;
 &lt;p&gt;We are aware that the web interface can be improved (PRs welcome). We will also work on the rebuilder setup and run some rebuilds.&lt;/p&gt;
 &lt;p&gt;Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:331831d8-6093-5dd7-9164-445afff953cb</id><title type="text">Deploying binary MirageOS unikernels</title><updated>2021-11-15T11:17:23-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.&lt;/p&gt;
-</summary><published>2021-04-23T13:33:06-00:00</published><link href="/Posts/EC" rel="alternate"/><content type="html">&lt;h2&gt;Introduction&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:331831d8-6093-5dd7-9164-445afff953cb</id><title type="text">Deploying binary MirageOS unikernels</title><updated>2021-11-15T11:17:23-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.&lt;/p&gt;
+</summary><published>2021-04-23T13:33:06-00:00</published><link href="/Posts/EC" rel="alternate"/><content type="html">&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
 &lt;p&gt;Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/&quot;&gt;fiat-crypto&lt;/a&gt; stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.&lt;/p&gt;
-&lt;h2&gt;Elliptic curve cryptography&lt;/h2&gt;
+&lt;h2 id=&quot;elliptic-curve-cryptography&quot;&gt;Elliptic curve cryptography&lt;/h2&gt;
 &lt;p&gt;&lt;a href=&quot;https://mirage.io/blog/tls-1-3-mirageos&quot;&gt;Since May 2020&lt;/a&gt;, our &lt;a href=&quot;https://usenix15.nqsb.io&quot;&gt;OCaml-TLS&lt;/a&gt; stack supports TLS 1.3 (since tls version 0.12.0 on opam).&lt;/p&gt;
 &lt;p&gt;TLS 1.3 requires elliptic curve cryptography - which was not available in &lt;a href=&quot;https://github.com/mirage/mirage-crypto&quot;&gt;mirage-crypto&lt;/a&gt; (the maintained fork of &lt;a href=&quot;https://github.com/mirleft/ocaml-nocrypto&quot;&gt;nocrypto&lt;/a&gt;).&lt;/p&gt;
 &lt;p&gt;There are two major uses of elliptic curves: &lt;a href=&quot;https://en.wikipedia.org/wiki/Elliptic-curve_Diffie%E2%80%93Hellman&quot;&gt;key exchange (ECDH)&lt;/a&gt; for establishing a shared secret over an insecure channel, and &lt;a href=&quot;https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm&quot;&gt;digital signature (ECDSA)&lt;/a&gt; for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)&lt;/p&gt;
@ -294,70 +294,70 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
 &lt;p&gt;In addition, to use the code in MirageOS, it should be boring C code: no heap allocations, only using a very small amount of C library functions -- the code needs to be compiled in an environment with &lt;a href=&quot;https://github.com/mirage/ocaml-freestanding/tree/v0.6.4/nolibc&quot;&gt;nolibc&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Two projects started in semantics, to solve the issue from the grounds up: &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/&quot;&gt;fiat-crypto&lt;/a&gt; and &lt;a href=&quot;https://github.com/project-everest/hacl-star/&quot;&gt;hacl-star&lt;/a&gt;: their approach is to use a proof system (&lt;a href=&quot;https://coq.inria.fr&quot;&gt;Coq&lt;/a&gt; or &lt;a href=&quot;https://www.fstar-lang.org/&quot;&gt;F*&lt;/a&gt; to verify that the code executes in constant time, not depending on data input. Both projects provide as output of their proof systems C code.&lt;/p&gt;
 &lt;p&gt;For our initial TLS 1.3 stack, &lt;a href=&quot;https://github.com/pascutto/&quot;&gt;Clément&lt;/a&gt;, &lt;a href=&quot;https://github.com/NathanReb/&quot;&gt;Nathan&lt;/a&gt; and &lt;a href=&quot;https://github.com/emillon/&quot;&gt;Etienne&lt;/a&gt; developed &lt;a href=&quot;https://github.com/mirage/fiat&quot;&gt;fiat-p256&lt;/a&gt; and &lt;a href=&quot;https://github.com/mirage/hacl&quot;&gt;hacl_x5519&lt;/a&gt;. Both were one-shot interfaces for a narrow use case (ECDH for NIST P-256 and X25519), worked well for their purpose, and allowed to gather some experience from the development side.&lt;/p&gt;
-&lt;h3&gt;Changed requirements&lt;/h3&gt;
+&lt;h3 id=&quot;changed-requirements&quot;&gt;Changed requirements&lt;/h3&gt;
 &lt;p&gt;Revisiting our cryptography stack with the elliptic curve perspective had several reasons, on the one side the customer project &lt;a href=&quot;https://www.nitrokey.com/products/nethsm&quot;&gt;NetHSM&lt;/a&gt; asked for feasibility of ECDSA/EdDSA for various elliptic curves, on the other side &lt;a href=&quot;https://github.com/mirage/ocaml-dns/pull/251&quot;&gt;DNSSec&lt;/a&gt; uses elliptic curve cryptography (ECDSA), and also &lt;a href=&quot;https://www.wireguard.com/&quot;&gt;wireguard&lt;/a&gt; relies on elliptic curve cryptography. The number of X.509 certificates using elliptic curves is increasing, and we don't want to leave our TLS stack in a state where it can barely talk to a growing number of services on the Internet.&lt;/p&gt;
 &lt;p&gt;Looking at &lt;a href=&quot;https://github.com/project-everest/hacl-star/&quot;&gt;&lt;em&gt;hacl-star&lt;/em&gt;&lt;/a&gt;, their &lt;a href=&quot;https://hacl-star.github.io/Supported.html&quot;&gt;support&lt;/a&gt; is limited to P-256 and Curve25519, any new curve requires writing F*. Another issue with hacl-star is C code quality: their C code does neither &lt;a href=&quot;https://github.com/mirage/hacl/issues/46&quot;&gt;compile with older C compilers (found on Oracle Linux 7 / CentOS 7)&lt;/a&gt;, nor when enabling all warnings (&amp;gt; 150 are generated). We consider the C compiler as useful resource to figure out undefined behaviour (and other problems), and when shipping C code we ensure that it compiles with &lt;code&gt;-Wall -Wextra -Wpedantic --std=c99 -Werror&lt;/code&gt;. The hacl project &lt;a href=&quot;https://github.com/mirage/hacl/tree/master/src/kremlin&quot;&gt;ships&lt;/a&gt; a bunch of header files and helper functions to work on all platforms, which is a clunky &lt;code&gt;ifdef&lt;/code&gt; desert. The hacl approach is to generate a whole algorithm solution: from arithmetic primitives, group operations, up to cryptographic protocol - everything included.&lt;/p&gt;
 &lt;p&gt;In contrast, &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/&quot;&gt;&lt;em&gt;fiat-crypto&lt;/em&gt;&lt;/a&gt; is a Coq development, which as part of compilation (proof verification) generates executables (via OCaml code extraction from Coq). These executables are used to generate modular arithmetic (as C code) given a curve description. The &lt;a href=&quot;https://github.com/mirage/mirage-crypto/tree/main/ec/native&quot;&gt;generated C code&lt;/a&gt; is highly portable, independent of platform (word size is taken as input) - it only requires a &lt;code&gt;&amp;lt;stdint.h&amp;gt;&lt;/code&gt;, and compiles with all warnings enabled (once &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/pull/906&quot;&gt;a minor PR&lt;/a&gt; got merged). Supporting a new curve is simple: generate the arithmetic code using fiat-crypto with the new curve description. The downside is that group operations and protocol needs to implemented elsewhere (and is not part of the proven code) - gladly this is pretty straightforward to do, especially in high-level languages.&lt;/p&gt;
-&lt;h3&gt;Working with fiat-crypto&lt;/h3&gt;
+&lt;h3 id=&quot;working-with-fiat-crypto&quot;&gt;Working with fiat-crypto&lt;/h3&gt;
 &lt;p&gt;As mentioned, our initial &lt;a href=&quot;https://github.com/mirage/fiat&quot;&gt;fiat-p256&lt;/a&gt; binding provided ECDH for the NIST P-256 curve. Also, BoringSSL uses fiat-crypto for ECDH, and developed the code for group operations and cryptographic protocol on top of it.&lt;/p&gt;
 &lt;p&gt;The work needed was (a) ECDSA support and (b) supporting more curves (let's focus on NIST curves). For ECDSA, the algorithm requires modular arithmetics in the field of the group order (in addition to the prime). We generate these primitives with fiat-crypto (named &lt;code&gt;npYYY_AA&lt;/code&gt;) - that required &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/commit/e31a36d5f1b20134e67ccc5339d88f0ff3cb0f86&quot;&gt;a small fix in decoding hex&lt;/a&gt;. Fiat-crypto also provides inversion &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/pull/670&quot;&gt;since late October 2020&lt;/a&gt;, &lt;a href=&quot;https://eprint.iacr.org/2021/549&quot;&gt;paper&lt;/a&gt; - which allowed to reduce our code base taken from BoringSSL. The ECDSA protocol was easy to implement in OCaml using the generated arithmetics.&lt;/p&gt;
 &lt;p&gt;Addressing the issue of more curves was also easy to achieve, the C code (group operations) are macros that are instantiated for each curve - the OCaml code are functors that are applied with each curve description.&lt;/p&gt;
 &lt;p&gt;Thanks to the test vectors (as structured data) from &lt;a href=&quot;https://github.com/google/wycheproof/&quot;&gt;wycheproof&lt;/a&gt; (and again thanks to Etienne, Nathan, and Clément for their OCaml code decodin them), I feel confident that our elliptic curve code works as desired.&lt;/p&gt;
 &lt;p&gt;What was left is X25519 and Ed25519 - dropping the hacl dependency entirely felt appealing (less C code to maintain from fewer projects). This turned out to require more C code, which we took from BoringSSL. It may be desirable to reduce the imported C code, or to wait until a project on top of fiat-crypto which provides proven cryptographic protocols is in a usable state.&lt;/p&gt;
 &lt;p&gt;To avoid performance degradation, I distilled some &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/107#issuecomment-799701703&quot;&gt;X25519 benchmarks&lt;/a&gt;, turns out the fiat-crypto and hacl performance is very similar.&lt;/p&gt;
-&lt;h3&gt;Achievements&lt;/h3&gt;
+&lt;h3 id=&quot;achievements&quot;&gt;Achievements&lt;/h3&gt;
 &lt;p&gt;The new opam package &lt;a href=&quot;https://mirage.github.io/mirage-crypto/doc/mirage-crypto-ec/Mirage_crypto_ec/index.html&quot;&gt;mirage-crypto-ec&lt;/a&gt; is released, which includes the C code generated by fiat-crypto (including &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/pull/670&quot;&gt;inversion&lt;/a&gt;), &lt;a href=&quot;https://github.com/mirage/mirage-crypto/blob/main/ec/native/point_operations.h&quot;&gt;point operations&lt;/a&gt; from BoringSSL, and some &lt;a href=&quot;https://github.com/mirage/mirage-crypto/blob/main/ec/mirage_crypto_ec.ml&quot;&gt;OCaml code&lt;/a&gt; for invoking these functions and doing bounds checks, and whether points are on the curve. The OCaml code are some functors that take the curve description (consisting of parameters, C function names, byte length of value) and provide Diffie-Hellman (Dh) and digital signature algorithm (Dsa) modules. The nonce for ECDSA is computed deterministically, as suggested by &lt;a href=&quot;https://tools.ietf.org/html/rfc6979&quot;&gt;RFC 6979&lt;/a&gt;, to avoid private key leakage.&lt;/p&gt;
 &lt;p&gt;The code has been developed in &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/101&quot;&gt;NIST curves&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/106&quot;&gt;removing blinding&lt;/a&gt; (since we use operations that are verified to be constant-time), &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/108&quot;&gt;added missing length checks&lt;/a&gt; (reported by &lt;a href=&quot;https://github.com/greg42&quot;&gt;Greg&lt;/a&gt;), &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/107&quot;&gt;curve25519&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/117&quot;&gt;a fix for signatures that do not span the entire byte size (discovered while adapting X.509)&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/118&quot;&gt;fix X25519 when the input has offset &amp;lt;&amp;gt; 0&lt;/a&gt;. It works on x86 and arm, both 32 and 64 bit (checked by CI). The development was partially sponsored by Nitrokey.&lt;/p&gt;
 &lt;p&gt;What is left to do, apart from further security reviews, is &lt;a href=&quot;https://github.com/mirage/mirage-crypto/issues/109&quot;&gt;performance improvements&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage-crypto/issues/112&quot;&gt;Ed448/X448 support&lt;/a&gt;, and &lt;a href=&quot;https://github.com/mirage/mirage-crypto/issues/105&quot;&gt;investigating deterministic k for P521&lt;/a&gt;. Pull requests are welcome.&lt;/p&gt;
 &lt;p&gt;When you use the code, and encounter any issues, please &lt;a href=&quot;https://github.com/mirage/mirage-crypto/issues&quot;&gt;report them&lt;/a&gt;.&lt;/p&gt;
-&lt;h2&gt;Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed&lt;/h2&gt;
+&lt;h2 id=&quot;layer-up---x.509-now-with-ecdsa-eddsa-and-pkcs-12-support-and-a-long-standing-issue-fixed&quot;&gt;Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed&lt;/h2&gt;
 &lt;p&gt;With the sign and verify primitives, the next step is to interoperate with other tools that generate and use these public and private keys. This consists of serialisation to and deserialisation from common data formats (ASN.1 DER and PEM encoding), and support for handling X.509 certificates with elliptic curve keys. Since X.509 0.12.0, it supports EC private and public keys, including certificate validation and issuance.&lt;/p&gt;
 &lt;p&gt;Releasing X.509 also included to go through the issue tracker and attempt to solve the existing issues. This time, the &lt;a href=&quot;https://github.com/mirleft/ocaml-x509/issues/69&quot;&gt;&amp;quot;country name is encoded as UTF8String, while RFC demands PrintableString&amp;quot;&lt;/a&gt; filed more than 5 years ago by &lt;a href=&quot;https://github.com/reynir&quot;&gt;Reynir&lt;/a&gt;, re-reported by &lt;a href=&quot;https://github.com/paurkedal&quot;&gt;Petter&lt;/a&gt; in early 2017, and again by &lt;a href=&quot;https://github.com/NightBlues&quot;&gt;Vadim&lt;/a&gt; in late 2020, &lt;a href=&quot;https://github.com/mirleft/ocaml-x509/pull/140&quot;&gt;was fixed by Vadim&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Another long-standing pull request was support for &lt;a href=&quot;https://tools.ietf.org/html/rfc7292&quot;&gt;PKCS 12&lt;/a&gt;, the archive format for certificate and private key bundles. This has &lt;a href=&quot;https://github.com/mirleft/ocaml-x509/pull/114&quot;&gt;been developed and merged&lt;/a&gt;. PKCS 12 is a widely used and old format (e.g. when importing / exporting cryptographic material in your browser, used by OpenVPN, ...). Its specification uses RC2 and 3DES (see &lt;a href=&quot;https://unmitigatedrisk.com/?p=654&quot;&gt;this nice article&lt;/a&gt;), which are the default algorithms used by &lt;code&gt;openssl pkcs12&lt;/code&gt;.&lt;/p&gt;
-&lt;h2&gt;One more layer up - TLS&lt;/h2&gt;
+&lt;h2 id=&quot;one-more-layer-up---tls&quot;&gt;One more layer up - TLS&lt;/h2&gt;
 &lt;p&gt;In TLS we are finally able to use ECDSA (and EdDSA) certificates and private keys, this resulted in slightly more complex configuration - the constraints between supported groups, signature algorithms, ciphersuite, and certificates are intricate:&lt;/p&gt;
 &lt;p&gt;The ciphersuite (in TLS before 1.3) specifies which key exchange mechanism to use, but also which signature algorithm to use (RSA/ECDSA). The supported groups client hello extension specifies which elliptic curves are supported by the client. The signature algorithm hello extension (TLS 1.2 and above) specifies the signature algorithm. In the end, at load time the TLS configuration is validated and groups, ciphersuites, and signature algorithms are condensed depending on configured server certificates. At session initiation time, once the client reports what it supports, these parameters are further cut down to eventually find some suitable cryptographic parameters for this session.&lt;/p&gt;
 &lt;p&gt;From the user perspective, earlier the certificate bundle and private key was a pair of &lt;code&gt;X509.Certificate.t list&lt;/code&gt; and &lt;code&gt;Mirage_crypto_pk.Rsa.priv&lt;/code&gt;, now the second part is a &lt;code&gt;X509.Private_key.t&lt;/code&gt; - all provided constructors have been updates (notably &lt;code&gt;X509_lwt.private_of_pems&lt;/code&gt; and &lt;code&gt;Tls_mirage.X509.certificate&lt;/code&gt;).&lt;/p&gt;
-&lt;h2&gt;Finally, conduit and mirage&lt;/h2&gt;
+&lt;h2 id=&quot;finally-conduit-and-mirage&quot;&gt;Finally, conduit and mirage&lt;/h2&gt;
 &lt;p&gt;Thanks to &lt;a href=&quot;https://github.com/dinosaure&quot;&gt;Romain&lt;/a&gt;, conduit* 4.0.0 was released which supports the modified API of X.509 and TLS. Romain also developed patches and released mirage 3.10.3 which supports the above mentioned work.&lt;/p&gt;
-&lt;h2&gt;Conclusion&lt;/h2&gt;
+&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
 &lt;p&gt;Elliptic curve cryptography is now available in OCaml using verified cryptographic primitives from the fiat-crypto project - &lt;code&gt;opam install mirage-crypto-ec&lt;/code&gt;. X.509 since 0.12.0 and TLS since 0.13.0 and MirageOS since 3.10.3 support this new development which gives rise to smaller EC keys. Our old bindings, fiat-p256 and hacl_x25519 have been archived and will no longer be maintained.&lt;/p&gt;
 &lt;p&gt;Thanks to everyone involved on this journey: reporting issues, sponsoring parts of the work, helping with integration, developing initial prototypes, and keep motivating me to continue this until the release is done.&lt;/p&gt;
 &lt;p&gt;In the future, it may be possible to remove zarith and gmp from the dependency chain, and provide EC-only TLS servers and clients for MirageOS. The benefit will be much less C code (libgmp-freestanding.a is 1.5MB in size) in our trusted code base.&lt;/p&gt;
 &lt;p&gt;Another potential project that is very close now is a certificate authority developed in MirageOS - now that EC keys, PKCS 12, revocation lists, ... are implemented.&lt;/p&gt;
-&lt;h2&gt;Footer&lt;/h2&gt;
+&lt;h2 id=&quot;footer&quot;&gt;Footer&lt;/h2&gt;
 &lt;p&gt;If you want to support our work on MirageOS unikernels, please &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donate to robur&lt;/a&gt;. I'm interested in feedback, either via &lt;a href=&quot;https://twitter.com/h4nnes&quot;&gt;twitter&lt;/a&gt;, &lt;a href=&quot;https://mastodon.social/@hannesm&quot;&gt;hannesm@mastodon.social&lt;/a&gt; or via eMail.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/tls" term="tls"/><category scheme="https://hannes.robur.coop/tags/security" term="security"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:16427713-5da1-50cd-b17c-ca5b5cca431d</id><title type="text">Cryptography updates in OCaml and MirageOS</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;Home office, MirageOS unikernels, 2020 recap, 2021 tbd&lt;/p&gt;
-</summary><published>2021-01-25T12:45:54-00:00</published><link href="/Posts/NGI" rel="alternate"/><content type="html">&lt;h2&gt;Introduction&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/tls" term="tls"/><category scheme="https://hannes.robur.coop/tags/security" term="security"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:16427713-5da1-50cd-b17c-ca5b5cca431d</id><title type="text">Cryptography updates in OCaml and MirageOS</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;Home office, MirageOS unikernels, 2020 recap, 2021 tbd&lt;/p&gt;
+</summary><published>2021-01-25T12:45:54-00:00</published><link href="/Posts/NGI" rel="alternate"/><content type="html">&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
 &lt;p&gt;2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations,  and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)&lt;/p&gt;
 &lt;p&gt;I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.&lt;/p&gt;
 &lt;p&gt;For 2021, we (at &lt;a href=&quot;https://robur.coop&quot;&gt;robur&lt;/a&gt;) got a grant from the EU (via &lt;a href=&quot;https://pointer.ngi.eu&quot;&gt;NGI pointer&lt;/a&gt;) for &amp;quot;Deploying MirageOS&amp;quot; (more details below), and another grant from &lt;a href=&quot;https://ocaml-sf.org&quot;&gt;OCaml software foundation&lt;/a&gt; for securing the opam supply chain (using &lt;a href=&quot;https://github.com/hannesm/conex&quot;&gt;conex&lt;/a&gt;). Some long-awaited releases for MirageOS libraries, namely a &lt;a href=&quot;https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh&quot;&gt;ssh implementation&lt;/a&gt; and a rewrite of our &lt;a href=&quot;https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/&quot;&gt;git implementation&lt;/a&gt; have already been published.&lt;/p&gt;
 &lt;p&gt;With my MirageOS view, 2020 was a pretty successful year, where we managed to add more features, fixed lots of bugs, and paved the road ahead. I want to thank &lt;a href=&quot;https://ocamllabs.io/&quot;&gt;OCamlLabs&lt;/a&gt; for funding work on MirageOS maintenance.&lt;/p&gt;
-&lt;h2&gt;Recap 2020&lt;/h2&gt;
+&lt;h2 id=&quot;recap-2020&quot;&gt;Recap 2020&lt;/h2&gt;
 &lt;p&gt;Here is a very subjective random collection of accomplishments in 2020, where I was involved with some degree.&lt;/p&gt;
-&lt;h3&gt;NetHSM&lt;/h3&gt;
+&lt;h3 id=&quot;nethsm&quot;&gt;NetHSM&lt;/h3&gt;
 &lt;p&gt;&lt;a href=&quot;https://www.nitrokey.com/products/nethsm&quot;&gt;NetHSM&lt;/a&gt; is a hardware security module in software. It is a product that uses MirageOS for security, and is based on the &lt;a href=&quot;https://muen.sk&quot;&gt;muen&lt;/a&gt; separation kernel. We at &lt;a href=&quot;https://robur.coop&quot;&gt;robur&lt;/a&gt; were heavily involved in this product. It already has been security audited by an external team. You can pre-order it from Nitrokey.&lt;/p&gt;
-&lt;h3&gt;TLS 1.3&lt;/h3&gt;
+&lt;h3 id=&quot;tls-1.3&quot;&gt;TLS 1.3&lt;/h3&gt;
 &lt;p&gt;Dating back to 2016, at the &lt;a href=&quot;https://www.ndss-symposium.org/ndss2016/tron-workshop-programme/&quot;&gt;TRON&lt;/a&gt; (TLS 1.3 Ready or NOt), we developed a first draft of a 1.3 implementation of &lt;a href=&quot;https://github.com/mirleft/ocaml-tls&quot;&gt;OCaml-TLS&lt;/a&gt;. Finally in May 2020 we got our act together, including ECC (ECDH P256 from &lt;a href=&quot;https://github.com/mit-plv/fiat-crypto/&quot;&gt;fiat-crypto&lt;/a&gt;, X25519 from &lt;a href=&quot;https://project-everest.github.io/&quot;&gt;hacl&lt;/a&gt;) and testing with &lt;a href=&quot;https://github.com/tlsfuzzer/tlsfuzzer&quot;&gt;tlsfuzzer&lt;/a&gt;, and release tls 0.12.0 with TLS 1.3 support. Later we added &lt;a href=&quot;https://github.com/mirleft/ocaml-tls/pull/414&quot;&gt;ECC ciphersuites to TLS version 1.2&lt;/a&gt;, implemented &lt;a href=&quot;https://github.com/mirleft/ocaml-tls/pull/414&quot;&gt;ChaCha20/Poly1305&lt;/a&gt;, and fixed an &lt;a href=&quot;https://github.com/mirleft/ocaml-tls/pull/424&quot;&gt;interoperability issue with Go's implementation&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;&lt;a href=&quot;https://github.com/mirage/mirage-crypto&quot;&gt;Mirage-crypto&lt;/a&gt; provides the underlying cryptographic primitives, initially released in March 2020 as a fork of &lt;a href=&quot;https://github.com/mirleft/ocaml-nocrypto&quot;&gt;nocrypto&lt;/a&gt; -- huge thanks to &lt;a href=&quot;https://github.com/pqwy&quot;&gt;pqwy&lt;/a&gt; for his great work. Mirage-crypto detects &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/53&quot;&gt;CPU features at runtime&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/Julow&quot;&gt;Julow&lt;/a&gt;) (&lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/96&quot;&gt;bugfix for bswap&lt;/a&gt;), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/39&quot;&gt;compilation on Windows&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/avsm&quot;&gt;avsm&lt;/a&gt;), &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/90&quot;&gt;async entropy harvesting&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/seliopou&quot;&gt;seliopou&lt;/a&gt;), &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/65&quot;&gt;32 bit support&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/72&quot;&gt;chacha20/poly1305&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/abeaumont&quot;&gt;abeaumont&lt;/a&gt;), &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/84&quot;&gt;cross-compilation&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/EduardoRFS&quot;&gt;EduardoRFS&lt;/a&gt;) and &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/78&quot;&gt;various&lt;/a&gt; &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/81&quot;&gt;bug&lt;/a&gt; &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/83&quot;&gt;fixes&lt;/a&gt;, even &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/95&quot;&gt;memory leak&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/talex5&quot;&gt;talex5&lt;/a&gt; for reporting several of these issues), and &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/99&quot;&gt;RSA&lt;/a&gt; &lt;a href=&quot;https://github.com/mirage/mirage-crypto/pull/100&quot;&gt;interoperability&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/psafont&quot;&gt;psafont&lt;/a&gt; for investigation and &lt;a href=&quot;https://github.com/mattjbray&quot;&gt;mattjbray&lt;/a&gt; for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.&lt;/p&gt;
-&lt;h3&gt;Qubes Firewall&lt;/h3&gt;
+&lt;h3 id=&quot;qubes-firewall&quot;&gt;Qubes Firewall&lt;/h3&gt;
 &lt;p&gt;The &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall/&quot;&gt;MirageOS based Qubes firewall&lt;/a&gt; is the most widely used MirageOS unikernel. And it got major updates: in May &lt;a href=&quot;https://github.com/linse&quot;&gt;Steffi&lt;/a&gt; &lt;a href=&quot;https://groups.google.com/g/qubes-users/c/Xzplmkjwa5Y&quot;&gt;announced&lt;/a&gt; her and &lt;a href=&quot;https://github.com/yomimono&quot;&gt;Mindy's&lt;/a&gt; work on improving it for Qubes 4.0 - including &lt;a href=&quot;https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x&quot;&gt;dynamic firewall rules via QubesDB&lt;/a&gt;. Thanks to &lt;a href=&quot;https://prototypefund.de/project/portable-firewall-fuer-qubesos/&quot;&gt;prototypefund&lt;/a&gt; for sponsoring.&lt;/p&gt;
 &lt;p&gt;In October 2020, we released &lt;a href=&quot;https://mirage.io/blog/announcing-mirage-39-release&quot;&gt;Mirage 3.9&lt;/a&gt; with PVH virtualization mode (thanks to &lt;a href=&quot;https://github.com/mato&quot;&gt;mato&lt;/a&gt;). There's still a &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall/issues/120&quot;&gt;memory leak&lt;/a&gt; to be investigated and fixed.&lt;/p&gt;
-&lt;h3&gt;IPv6&lt;/h3&gt;
+&lt;h3 id=&quot;ipv6&quot;&gt;IPv6&lt;/h3&gt;
 &lt;p&gt;In December, with &lt;a href=&quot;https://mirage.io/blog/announcing-mirage-310-release&quot;&gt;Mirage 3.10&lt;/a&gt; we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to &lt;a href=&quot;https://github.com/nojb&quot;&gt;nojb&lt;/a&gt; for the initial code and &lt;a href=&quot;https://github.com/MagnusS&quot;&gt;MagnusS&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Turns out this blog, but also robur services, are now available via IPv6 :)&lt;/p&gt;
-&lt;h3&gt;Albatross&lt;/h3&gt;
+&lt;h3 id=&quot;albatross&quot;&gt;Albatross&lt;/h3&gt;
 &lt;p&gt;Also in December, I pushed an initial release of &lt;a href=&quot;https://github.com/roburio/albatross&quot;&gt;albatross&lt;/a&gt;, a unikernel orchestration system with remote access. &lt;em&gt;Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.&lt;/em&gt;&lt;/p&gt;
 &lt;p&gt;Thanks to &lt;a href=&quot;https://github.com/reynir&quot;&gt;reynir&lt;/a&gt; for statistics support on Linux and improvements of the systemd service scripts. Also thanks to &lt;a href=&quot;https://github.com/cfcs&quot;&gt;cfcs&lt;/a&gt; for the initial Linux port.&lt;/p&gt;
-&lt;h3&gt;CA certs&lt;/h3&gt;
+&lt;h3 id=&quot;ca-certs&quot;&gt;CA certs&lt;/h3&gt;
 &lt;p&gt;For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to &lt;a href=&quot;https://github.com/emillon&quot;&gt;emillon&lt;/a&gt; for initial code, there are now &lt;a href=&quot;https://github.com/mirage/ca-certs&quot;&gt;ca-certs&lt;/a&gt; and &lt;a href=&quot;https://github.com/mirage/ca-certs-nss&quot;&gt;ca-certs-nss&lt;/a&gt; opam packages (see &lt;a href=&quot;https://discuss.ocaml.org/t/ann-ca-certs-and-ca-certs-nss&quot;&gt;release announcement&lt;/a&gt;) which fills this gap.&lt;/p&gt;
-&lt;h2&gt;Unikernels&lt;/h2&gt;
+&lt;h2 id=&quot;unikernels&quot;&gt;Unikernels&lt;/h2&gt;
 &lt;p&gt;I developed several useful unikernels in 2020, and also pushed &lt;a href=&quot;https://mirage.io/wiki/gallery&quot;&gt;a unikernel gallery&lt;/a&gt; to the Mirage website:&lt;/p&gt;
-&lt;h3&gt;Traceroute in MirageOS&lt;/h3&gt;
+&lt;h3 id=&quot;traceroute-in-mirageos&quot;&gt;Traceroute in MirageOS&lt;/h3&gt;
 &lt;p&gt;I already wrote about &lt;a href=&quot;/Posts/Traceroute&quot;&gt;traceroute&lt;/a&gt; which traces the routing to a given remote host.&lt;/p&gt;
-&lt;h3&gt;Unipi - static website hosting&lt;/h3&gt;
+&lt;h3 id=&quot;unipi---static-website-hosting&quot;&gt;Unipi - static website hosting&lt;/h3&gt;
 &lt;p&gt;&lt;a href=&quot;https://github.com/roburio/unipi&quot;&gt;Unipi&lt;/a&gt; is a static site webserver which retrieves the content from a remote git repository. Let's encrypt certificate provisioning and dynamic updates via a webhook to be executed for every push.&lt;/p&gt;
-&lt;h4&gt;TLSTunnel - TLS demultiplexing&lt;/h4&gt;
+&lt;h4 id=&quot;tlstunnel---tls-demultiplexing&quot;&gt;TLSTunnel - TLS demultiplexing&lt;/h4&gt;
 &lt;p&gt;The physical machine this blog and other robur infrastructure runs on has been relocated from Sweden to Germany mid-December. Thanks to UPS! Fewer IPv4 addresses are available in the new data center, which motivated me to develop &lt;a href=&quot;https://github.com/roburio/tlstunnel&quot;&gt;tlstunnel&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;The new behaviour is as follows (see the &lt;code&gt;monitoring&lt;/code&gt; branch):&lt;/p&gt;
 &lt;ul&gt;
@ -372,9 +372,9 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
 &lt;li&gt;setting up a new service is very straightforward: only the new name needs to be registered with tlstunnel together with the TCP backend, and everything will just work
 &lt;/li&gt;
 &lt;/ul&gt;
-&lt;h2&gt;2021&lt;/h2&gt;
+&lt;h2 id=&quot;section&quot;&gt;2021&lt;/h2&gt;
 &lt;p&gt;The year started with a release of &lt;a href=&quot;https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh&quot;&gt;awa&lt;/a&gt;, a SSH implementation in OCaml (thanks to &lt;a href=&quot;https://github.com/haesbaert&quot;&gt;haesbaert&lt;/a&gt; for initial code). This was followed by a &lt;a href=&quot;https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/&quot;&gt;git 3.0 release&lt;/a&gt; (thanks to &lt;a href=&quot;https://github.com/dinosaure&quot;&gt;dinosaure&lt;/a&gt;).&lt;/p&gt;
-&lt;h3&gt;Deploying MirageOS - NGI Pointer&lt;/h3&gt;
+&lt;h3 id=&quot;deploying-mirageos---ngi-pointer&quot;&gt;Deploying MirageOS - NGI Pointer&lt;/h3&gt;
 &lt;p&gt;For 2021 we at robur received funding from the EU (via &lt;a href=&quot;https://pointer.ngi.eu/&quot;&gt;NGI pointer&lt;/a&gt;) for &amp;quot;Deploying MirageOS&amp;quot;, which boils down into three parts:&lt;/p&gt;
 &lt;ul&gt;
 &lt;li&gt;reproducible binary releases of MirageOS unikernels,
@ -387,14 +387,14 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
 &lt;p&gt;Of course this will all be available open source. Please get in touch via eMail (team aT robur dot coop) if you're eager to integrate MirageOS unikernels into your infrastructure.&lt;/p&gt;
 &lt;p&gt;We discovered at an initial meeting with an infrastructure provider that a DNS resolver is of interest - even more now that dnsmasq suffered from &lt;a href=&quot;https://www.jsof-tech.com/wp-content/uploads/2021/01/DNSpooq_Technical-Whitepaper.pdf&quot;&gt;dnspooq&lt;/a&gt;. We are already working on an &lt;a href=&quot;https://github.com/mirage/ocaml-dns/pull/251&quot;&gt;implementation of DNSSec&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;MirageOS unikernels are binary reproducible, and &lt;a href=&quot;https://github.com/rjbou/orb/pull/1&quot;&gt;infrastructure tools are available&lt;/a&gt;. We are working hard on a web interface (and REST API - think of it as &amp;quot;Docker Hub for MirageOS unikernels&amp;quot;), and more tooling to verify reproducibility.&lt;/p&gt;
-&lt;h3&gt;Conex - securing the supply chain&lt;/h3&gt;
+&lt;h3 id=&quot;conex---securing-the-supply-chain&quot;&gt;Conex - securing the supply chain&lt;/h3&gt;
 &lt;p&gt;Another funding from the &lt;a href=&quot;http://ocaml-sf.org/&quot;&gt;OCSF&lt;/a&gt; is to continue development and deploy &lt;a href=&quot;https://github.com/hannesm/conex&quot;&gt;conex&lt;/a&gt; - to bring trust into opam-repository. This is a great combination with the reproducible build efforts, and will bring much more trust into retrieving OCaml packages and using MirageOS unikernels.&lt;/p&gt;
-&lt;h3&gt;MirageOS 4.0&lt;/h3&gt;
+&lt;h3 id=&quot;mirageos-4.0&quot;&gt;MirageOS 4.0&lt;/h3&gt;
 &lt;p&gt;Mirage so far still uses ocamlbuild and ocamlfind for compiling the virtual machine binary. But the switch to dune is &lt;a href=&quot;https://github.com/mirage/mirage/issues/1195&quot;&gt;close&lt;/a&gt;, a lot of effort has been done. This will make the developer experience of MirageOS much more smooth, with a per-unikernel monorepo workflow where you can push your changes to the individual libraries.&lt;/p&gt;
-&lt;h2&gt;Footer&lt;/h2&gt;
+&lt;h2 id=&quot;footer&quot;&gt;Footer&lt;/h2&gt;
 &lt;p&gt;If you want to support our work on MirageOS unikernels, please &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donate to robur&lt;/a&gt;. I'm interested in feedback, either via &lt;a href=&quot;https://twitter.com/h4nnes&quot;&gt;twitter&lt;/a&gt;, &lt;a href=&quot;https://mastodon.social/@hannesm&quot;&gt;hannesm@mastodon.social&lt;/a&gt; or via eMail.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:bc7675a5-47d0-5ce1-970c-01ed07fdf404</id><title type="text">The road ahead for MirageOS in 2021</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;A MirageOS unikernel which traces the path between itself and a remote host.&lt;/p&gt;
-</summary><published>2020-06-24T10:38:10-00:00</published><link href="/Posts/Traceroute" rel="alternate"/><content type="html">&lt;h2&gt;Traceroute&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:bc7675a5-47d0-5ce1-970c-01ed07fdf404</id><title type="text">The road ahead for MirageOS in 2021</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;A MirageOS unikernel which traces the path between itself and a remote host.&lt;/p&gt;
+</summary><published>2020-06-24T10:38:10-00:00</published><link href="/Posts/Traceroute" rel="alternate"/><content type="html">&lt;h2 id=&quot;traceroute&quot;&gt;Traceroute&lt;/h2&gt;
 &lt;p&gt;Is a diagnostic utility which displays the route and measures transit delays of
 packets across an Internet protocol (IP) network.&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-bash&quot;&gt;$ doas solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --host=198.167.222.207
@ -738,16 +738,16 @@ $ solo5-hvt --net:service=tap0 -- traceroute.hvt ...
 &lt;p&gt;If you develop enhancements you'd like to share, please sent a pull request to the git repository.&lt;/p&gt;
 &lt;p&gt;Motivation for this traceroute unikernel was while talking with &lt;a href=&quot;https://twitter.com/networkservice&quot;&gt;Aaron&lt;/a&gt; and &lt;a href=&quot;https://github.com/phaer&quot;&gt;Paul&lt;/a&gt;, who contributed several patches to the IP stack which pass the ttl through.&lt;/p&gt;
 &lt;p&gt;If you want to support our work on MirageOS unikernels, please &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donate to robur&lt;/a&gt;. I'm interested in feedback, either via &lt;a href=&quot;https://twitter.com/h4nnes&quot;&gt;twitter&lt;/a&gt;, &lt;a href=&quot;https://mastodon.social/@hannesm&quot;&gt;hannesm@mastodon.social&lt;/a&gt; or via eMail.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/protocol" term="protocol"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:ed3036f6-83d2-5e80-b3da-4ccbedb5ae9e</id><title type="text">Traceroute</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.&lt;/p&gt;
-</summary><published>2019-12-23T21:30:53-00:00</published><link href="/Posts/DnsServer" rel="alternate"/><content type="html">&lt;h2&gt;Goal&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/protocol" term="protocol"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:ed3036f6-83d2-5e80-b3da-4ccbedb5ae9e</id><title type="text">Traceroute</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.&lt;/p&gt;
+</summary><published>2019-12-23T21:30:53-00:00</published><link href="/Posts/DnsServer" rel="alternate"/><content type="html">&lt;h2 id=&quot;goal&quot;&gt;Goal&lt;/h2&gt;
 &lt;p&gt;Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as &lt;code&gt;nqsb.io&lt;/code&gt; and &lt;code&gt;robur.coop&lt;/code&gt;. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as &lt;a href=&quot;/Posts/DNS&quot;&gt;µDNS&lt;/a&gt;.&lt;/p&gt;
-&lt;h2&gt;Prerequisites&lt;/h2&gt;
+&lt;h2 id=&quot;prerequisites&quot;&gt;Prerequisites&lt;/h2&gt;
 &lt;p&gt;You need to own a domain, and be able to delegate the name service to your own servers.
 You also need two spare public IPv4 addresses (in different /24 networks) for your name servers.
 A git server or remote repository reachable via git over ssh.
 Servers which support &lt;a href=&quot;https://github.com/solo5/solo5&quot;&gt;solo5&lt;/a&gt; guests, and have the corresponding tender installed.
 A computer with &lt;a href=&quot;https://opam.ocaml.org&quot;&gt;opam&lt;/a&gt; (&amp;gt;= 2.0.0) installed.&lt;/p&gt;
-&lt;h2&gt;Data preparation&lt;/h2&gt;
+&lt;h2 id=&quot;data-preparation&quot;&gt;Data preparation&lt;/h2&gt;
 &lt;p&gt;Figure out a way to get the DNS entries of your domain in a &lt;a href=&quot;https://tools.ietf.org/html/rfc1034&quot;&gt;&amp;quot;master file format&amp;quot;&lt;/a&gt;, i.e. what bind uses.&lt;/p&gt;
 &lt;p&gt;This is a master file for the &lt;code&gt;mirage&lt;/code&gt; domain, defining &lt;code&gt;$ORIGIN&lt;/code&gt; to avoid typing the domain name after each hostname (use &lt;code&gt;@&lt;/code&gt; if you need the domain name only; if you need to refer to a hostname in a different domain end it with a dot (&lt;code&gt;.&lt;/code&gt;), i.e. &lt;code&gt;ns2.foo.com.&lt;/code&gt;). The default time to live &lt;code&gt;$TTL&lt;/code&gt; is an hour (3600 seconds).
 The zone contains a &lt;a href=&quot;https://tools.ietf.org/html/rfc1035#section-3.3.13&quot;&gt;start of authority (&lt;code&gt;SOA&lt;/code&gt;) record&lt;/a&gt; containing the nameserver, hostmaster, serial, refresh, retry, expiry, and minimum.
@ -761,7 +761,7 @@ ns1     A       127.0.0.1
 www	A	1.1.1.1
 git-repo&amp;gt; git add mirage &amp;amp;&amp;amp; git commit -m initial &amp;amp;&amp;amp; git push
 &lt;/code&gt;&lt;/pre&gt;
-&lt;h2&gt;Installation&lt;/h2&gt;
+&lt;h2 id=&quot;installation&quot;&gt;Installation&lt;/h2&gt;
 &lt;p&gt;On your development machine, you need to install various OCaml packages. You don't need privileged access if common tools (C compiler, make, libgmp) are already installed. You have &lt;code&gt;opam&lt;/code&gt; installed.&lt;/p&gt;
 &lt;p&gt;Let's create a fresh &lt;code&gt;switch&lt;/code&gt; for the DNS journey:&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;$ opam init
@ -771,7 +771,7 @@ $ opam switch create udns 4.14.1
 $ eval `opam env` #sets some environment variables
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;The last command set environment variables in your current shell session, please use the same shell for the commands following (or run &lt;code&gt;eval $(opam env)&lt;/code&gt; in another shell and proceed in there - the output of &lt;code&gt;opam switch&lt;/code&gt; sohuld point to &lt;code&gt;udns&lt;/code&gt;).&lt;/p&gt;
-&lt;h3&gt;Validation of our zonefile&lt;/h3&gt;
+&lt;h3 id=&quot;validation-of-our-zonefile&quot;&gt;Validation of our zonefile&lt;/h3&gt;
 &lt;p&gt;First let's check that OCaml-DNS can parse our zonefile:&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;$ opam install dns-cli #installs ~/.opam/udns/bin/ozone and other binaries
 $ ozone &amp;lt;git-repo&amp;gt;/mirage # see ozone --help
@ -779,7 +779,7 @@ successfully checked zone
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;Great. Error reporting is not great, but line numbers are indicated (&lt;code&gt;ozone: zone parse problem at line 3: syntax error&lt;/code&gt;), &lt;a href=&quot;https://github.com/mirage/ocaml-dns/tree/v4.2.0/zone&quot;&gt;lexer and parser are lex/yacc style&lt;/a&gt; (PRs welcome).&lt;/p&gt;
 &lt;p&gt;FWIW, &lt;code&gt;ozone&lt;/code&gt; accepts &lt;code&gt;--old &amp;lt;filename&amp;gt;&lt;/code&gt; to check whether an update from the old zone to the new is fine. This can be used as &lt;a href=&quot;https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks&quot;&gt;pre-commit hook&lt;/a&gt; in your git repository to avoid bad parse states in your name servers.&lt;/p&gt;
-&lt;h3&gt;Getting the primary up&lt;/h3&gt;
+&lt;h3 id=&quot;getting-the-primary-up&quot;&gt;Getting the primary up&lt;/h3&gt;
 &lt;p&gt;The next step is to compile the primary server and run it to serve the domain data. Since the git-via-ssh client is not yet released, we need to add a custom opam repository to this switch.&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;# get the `mirage` application via opam
 $ opam install lwt mirage
@ -810,7 +810,7 @@ $ dig any mirage @127.0.0.1
 # a DNS packet printout with all records available for mirage
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;That's exciting, the DNS server serving answers from a remote git repository.&lt;/p&gt;
-&lt;h3&gt;Securing the git access with ssh&lt;/h3&gt;
+&lt;h3 id=&quot;securing-the-git-access-with-ssh&quot;&gt;Securing the git access with ssh&lt;/h3&gt;
 &lt;p&gt;Let's authenticate the access by using ssh, so we feel ready to push data there as well. The primary-git unikernel already includes an experimental &lt;a href=&quot;https://github.com/haesbaert/awa-ssh&quot;&gt;ssh client&lt;/a&gt;, all we need to do is setting up credentials - in the following a RSA keypair and the server fingerprint.&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;# collect the RSA host key fingerprint
 $ ssh-keyscan &amp;lt;git-server&amp;gt; &amp;gt; /tmp/git-server-public-keys
@ -831,7 +831,7 @@ $ ./primary-git --authenticator=SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBa
 # started up, you can try the host and dig commands from above if you like
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;To wrap up, we now have a primary authoritative name server for our zone running as Unix process, which clones a remote git repository via ssh on startup and then serves it.&lt;/p&gt;
-&lt;h3&gt;Authenticated data updates&lt;/h3&gt;
+&lt;h3 id=&quot;authenticated-data-updates&quot;&gt;Authenticated data updates&lt;/h3&gt;
 &lt;p&gt;Our remote git repository is the source of truth, if you need to add a DNS entry to the zone, you git pull, edit the zone file, remember to increase the serial in the SOA line, run &lt;code&gt;ozone&lt;/code&gt;, git commit and push to the repository.&lt;/p&gt;
 &lt;p&gt;So, the &lt;code&gt;primary-git&lt;/code&gt; needs to be informed of git pushes. This requires a communication channel from the git server (or somewhere else, e.g. your laptop) to the DNS server. I prefer in-protocol solutions over adding yet another protocol stack, no way my DNS server will talk HTTP REST.&lt;/p&gt;
 &lt;p&gt;The DNS protocol has an extension for &lt;a href=&quot;https://tools.ietf.org/html/rfc1996&quot;&gt;notifications of zone changes&lt;/a&gt; (as a DNS packet), usually used between the primary and secondary servers. The &lt;code&gt;primary-git&lt;/code&gt; accepts these notify requests (i.e. bends the standard slightly), and upon receival pulls the remote git repository, and serves the fresh zone files. Since a git pull may be rather excessive in terms of CPU cycles and network bandwidth, only authenticated notifications are accepted.&lt;/p&gt;
@ -857,7 +857,7 @@ $ onotify 127.0.0.1 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31
 # further changes to the hmac secrets don't require a restart anymore, a notify packet is sufficient :D
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;Ok, this onotify command line could be setup as a git post-commit hook, or run manually after each manual git push.&lt;/p&gt;
-&lt;h3&gt;Secondary&lt;/h3&gt;
+&lt;h3 id=&quot;secondary&quot;&gt;Secondary&lt;/h3&gt;
 &lt;p&gt;It's time to figure out how to integrate the secondary name server. An already existing bind or something else that accepts notifications and issues zone transfers with hmac-sha256 secrets should work out of the box. If you encounter interoperability issues, please get in touch with me.&lt;/p&gt;
 &lt;p&gt;The &lt;code&gt;secondary&lt;/code&gt; unikernel is available from another git repository:&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;# get the secondary sources
@ -870,7 +870,7 @@ $ cd dns-secondary
 $ make
 $ ./dist/secondary
 &lt;/code&gt;&lt;/pre&gt;
-&lt;h3&gt;IP addresses and routing&lt;/h3&gt;
+&lt;h3 id=&quot;ip-addresses-and-routing&quot;&gt;IP addresses and routing&lt;/h3&gt;
 &lt;p&gt;Both primary and secondary serve the data on the DNS port (53) on UDP and TCP. To run both on the same machine and bind them to different IP addresses, we'll use a layer 2 network (ethernet frames) with a host system software switch (bridge interface &lt;code&gt;service&lt;/code&gt;), the unikernels as virtual machines (or seccomp-sandboxed) via the &lt;a href=&quot;https://github.com/solo5/solo5&quot;&gt;solo5&lt;/a&gt; backend. Using xen is possible as well. As IP address range we'll use 10.0.42.0/24, and the host system uses the 10.0.42.1.&lt;/p&gt;
 &lt;p&gt;The primary git needs connectivity to the remote git repository, thus on a laptop in a private network we need network address translation (NAT) from the bridge where the unikernels speak to the Internet where the git repository resides.&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;# on FreeBSD:
@ -896,7 +896,7 @@ tap1
 # add them to the bridge
 $ ifconfig service addm tap0 addm tap1
 &lt;/code&gt;&lt;/pre&gt;
-&lt;h3&gt;Primary and secondary setup&lt;/h3&gt;
+&lt;h3 id=&quot;primary-and-secondary-setup&quot;&gt;Primary and secondary setup&lt;/h3&gt;
 &lt;p&gt;Let's update our zone slightly to reflect the IP changes.&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;git-repo&amp;gt; cat mirage
 $ORIGIN mirage.
@ -948,7 +948,7 @@ $ dig foo.mirage @10.0.42.2 # primary
 $ dig foo.mirage @10.0.42.3 # secondary got notified and transferred the zone
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;You can also check the behaviour when restarting either of the VMs, whenever the primary is available the zone is synchronised. If the primary is down, the secondary still serves the zone. When the secondary is started while the primary is down, it won't serve any data until the primary is online (the secondary polls periodically, the primary sends notifies on startup).&lt;/p&gt;
-&lt;h3&gt;Dynamic data updates via DNS, pushed to git&lt;/h3&gt;
+&lt;h3 id=&quot;dynamic-data-updates-via-dns-pushed-to-git&quot;&gt;Dynamic data updates via DNS, pushed to git&lt;/h3&gt;
 &lt;p&gt;DNS is a rich protocol, and it also has builtin &lt;a href=&quot;https://tools.ietf.org/html/rfc2136&quot;&gt;updates&lt;/a&gt; that are supported by OCaml DNS, again authenticated with hmac-sha256 and shared secrets. Bind provides the command-line utility &lt;code&gt;nsupdate&lt;/code&gt; to send these update packets, a simple &lt;code&gt;oupdate&lt;/code&gt; unix utility is available as well (i.e. for integration of dynamic DNS clients). You know the drill, add a shared secret to the primary, git push, notify the primary, and voila we can dynamically in-protocol update. An update received by the primary via this way will trigger a git push to the remote git repository, and notifications to the secondary servers as described above.&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-shell&quot;&gt;# being lazy, I reuse the key above
 $ oupdate 10.0.42.2 personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg= my-other.mirage 1.2.3.4
@ -963,7 +963,7 @@ $ dig my-other.mirage @10.0.42.2
 $ dig my-other.mirage @10.0.42.3
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;So we can deploy further &lt;code&gt;oupdate&lt;/code&gt; (or &lt;code&gt;nsupdate&lt;/code&gt;) clients, distribute hmac secrets, and have the DNS zone updated. The source of truth is still the git repository, where the primary-git pushes to. Merge conflicts and timing of pushes is not yet dealt with. They are unlikely to happen since the primary is notified on pushes and should have up-to-date data in storage. Sorry, I'm unsure about the error semantics, try it yourself.&lt;/p&gt;
-&lt;h3&gt;Let's encrypt!&lt;/h3&gt;
+&lt;h3 id=&quot;lets-encrypt&quot;&gt;Let's encrypt!&lt;/h3&gt;
 &lt;p&gt;&lt;a href=&quot;https://letsencrypt.org/&quot;&gt;Let's encrypt&lt;/a&gt; is a certificate authority (CA), which certificate is shipped as trust anchor in web browsers. They specified a protocol for &lt;a href=&quot;https://tools.ietf.org/html/draft-ietf-acme-acme-05&quot;&gt;automated certificate management environment (ACME)&lt;/a&gt;, used to get X509 certificates for your services. In the protocol, a certificate signing request (publickey and hostname) is sent to let's encrypt servers, which sends a challenge to proof the ownership of the hostnames. One widely-used way to solve this challenge is running a web server, another is to serve it as text record from the authoritative DNS server.&lt;/p&gt;
 &lt;p&gt;Since I avoid persistent storage when possible, and also don't want to integrate a HTTP client stack in the primary server, I developed a third unikernel that acts as (hidden) secondary server, performs the tedious HTTP communication with let's encrypt servers, and stores all data in the public DNS zone.&lt;/p&gt;
 &lt;p&gt;For encoding of certificates, the DANE working group specified &lt;a href=&quot;https://tools.ietf.org/html/rfc6698.html#section-7.1&quot;&gt;TLSA&lt;/a&gt; records in DNS. They are quadruples of usage, selector, matching type, and ASN.1 DER-encoded material. We set usage to 3 (domain-issued certificate), matching type to 0 (no hash), and selector to 0 (full certificate) or 255 (private usage) for certificate signing requests. The interaction is as follows:&lt;/p&gt;
@ -1008,27 +1008,27 @@ $ ocertify 10.0.42.2 foo.mirage
 &lt;/code&gt;&lt;/pre&gt;
 &lt;p&gt;For actual testing with let's encrypt servers you need to have the primary and secondary deployed on your remote hosts, and your domain needs to be delegated to these servers. Good luck. And ensure you have backup your git repository.&lt;/p&gt;
 &lt;p&gt;As fine print, while this tutorial was about the &lt;code&gt;mirage&lt;/code&gt; zone, you can stick any number of zones into the git repository. If you use a &lt;code&gt;_keys&lt;/code&gt; file (without any domain prefix), you can configure hmac secrets for all zones, i.e. something to use in your let's encrypt unikernel and secondary unikernel. Dynamic addition of zones is supported, just create a new zonefile and notify the primary, the secondary will be notified and pick it up. The primary responds to a signed SOA for the root zone (i.e. requested by the secondary) with the SOA response (not authoritative), and additionally notifications for all domains of the primary.&lt;/p&gt;
-&lt;h3&gt;Conclusion and thanks&lt;/h3&gt;
+&lt;h3 id=&quot;conclusion-and-thanks&quot;&gt;Conclusion and thanks&lt;/h3&gt;
 &lt;p&gt;This tutorial presented how to use the OCaml DNS based unikernels to run authoritative name servers for your domain, using a git repository as the source of truth, dynamic authenticated updates, and let's encrypt certificate issuing.&lt;/p&gt;
 &lt;p&gt;There are further steps to take, such as monitoring (&lt;code&gt;mirage configure --monitoring&lt;/code&gt;), which use a second network interface for reporting syslog and metrics to telegraf / influx / grafana. Some DNS features are still missing, most prominently DNSSec.&lt;/p&gt;
 &lt;p&gt;I'd like to thank all people involved in this software stack, without other key components, including &lt;a href=&quot;https://github.com/mirage/ocaml-git&quot;&gt;git&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage-crypto&quot;&gt;mirage-crypto&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/awa-ssh&quot;&gt;awa-ssh&lt;/a&gt;, &lt;a href=&quot;https://github.com/solo5/sol5&quot;&gt;solo5&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/mirage&quot;&gt;mirage&lt;/a&gt;, &lt;a href=&quot;https://github.com/mmaker/ocaml-letsencrypt&quot;&gt;ocaml-letsencrypt&lt;/a&gt;, and more.&lt;/p&gt;
 &lt;p&gt;If you want to support our work on MirageOS unikernels, please &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donate to robur&lt;/a&gt;. I'm interested in feedback, either via &lt;a href=&quot;https://twitter.com/h4nnes&quot;&gt;twitter&lt;/a&gt;, &lt;a href=&quot;https://mastodon.social/@hannesm&quot;&gt;hannesm@mastodon.social&lt;/a&gt; or via eMail.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/protocol" term="protocol"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:e3d4fd9e-e379-5c86-838e-46034ddd435d</id><title type="text">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</title><updated>2023-03-02T17:20:44-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;MirageOS unikernels are reproducible :)&lt;/p&gt;
-</summary><published>2019-12-16T18:29:30-00:00</published><link href="/Posts/ReproducibleOPAM" rel="alternate"/><content type="html">&lt;h2&gt;Reproducible builds summit&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/protocol" term="protocol"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:e3d4fd9e-e379-5c86-838e-46034ddd435d</id><title type="text">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</title><updated>2023-03-02T17:20:44-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;MirageOS unikernels are reproducible :)&lt;/p&gt;
+</summary><published>2019-12-16T18:29:30-00:00</published><link href="/Posts/ReproducibleOPAM" rel="alternate"/><content type="html">&lt;h2 id=&quot;reproducible-builds-summit&quot;&gt;Reproducible builds summit&lt;/h2&gt;
 &lt;p&gt;I'm just back from the &lt;a href=&quot;https://reproducible-builds.org/events/Marrakesh2019/&quot;&gt;Reproducible builds summit 2019&lt;/a&gt;. In 2018, several people developing &lt;a href=&quot;https://ocaml.org&quot;&gt;OCaml&lt;/a&gt; and &lt;a href=&quot;https://opam.ocaml.org&quot;&gt;opam&lt;/a&gt; and &lt;a href=&quot;https://mirage.io&quot;&gt;MirageOS&lt;/a&gt;, attended &lt;a href=&quot;https://reproducible-builds.org/events/paris2018/&quot;&gt;the Reproducible builds summit in Paris&lt;/a&gt;. The notes from last year on &lt;a href=&quot;https://reproducible-builds.org/events/paris2018/report/#Toc11410_331763073&quot;&gt;opam reproducibility&lt;/a&gt; and &lt;a href=&quot;https://reproducible-builds.org/events/paris2018/report/#Toc11681_331763073&quot;&gt;MirageOS reproducibility&lt;/a&gt; are online. After last years workshop, Raja started developing the opam reproducibilty builder &lt;a href=&quot;https://github.com/rjbou/orb&quot;&gt;orb&lt;/a&gt;, which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.&lt;/p&gt;
-&lt;h2&gt;Goal&lt;/h2&gt;
+&lt;h2 id=&quot;goal&quot;&gt;Goal&lt;/h2&gt;
 &lt;p&gt;Stepping back a bit, first look on the &lt;a href=&quot;https://reproducible-builds.org/&quot;&gt;goal of reproducible builds&lt;/a&gt;: when compiling source code multiple times, the produced binaries should be identical. It should be sufficient if the binaries are behaviourally equal, but this is pretty hard to check. It is much easier to check &lt;strong&gt;bit-wise identity of binaries&lt;/strong&gt;, and relaxes the burden on the checker -- checking for reproducibility is reduced to computing the hash of the binaries. Let's stick to the bit-wise identical binary definition, which also means software developers have to avoid non-determinism during compilation in their toolchains, dependent libraries, and developed code.&lt;/p&gt;
 &lt;p&gt;A &lt;a href=&quot;https://reproducible-builds.org/docs/test-bench/&quot;&gt;checklist&lt;/a&gt; of potential things leading to non-determinism has been written up by the reproducible builds project. Examples include recording the build timestamp into the binary, ordering of code and embedded data. The reproducible builds project also developed &lt;a href=&quot;https://packages.debian.org/sid/disorderfs&quot;&gt;disorderfs&lt;/a&gt; for testing reproducibility and &lt;a href=&quot;https://diffoscope.org/&quot;&gt;diffoscope&lt;/a&gt; for comparing binaries with file-dependent readers, falling back to &lt;code&gt;objdump&lt;/code&gt; and &lt;code&gt;hexdump&lt;/code&gt;. A giant &lt;a href=&quot;https://tests.reproducible-builds.org/&quot;&gt;test infrastructure&lt;/a&gt; with &lt;a href=&quot;https://tests.reproducible-builds.org/debian/index_variations.html&quot;&gt;lots of variations&lt;/a&gt; between the builds, mostly using Debian, has been setup over the years.&lt;/p&gt;
 &lt;p&gt;Reproducibility is a precondition for trustworthy binaries. See &lt;a href=&quot;https://reproducible-builds.org/#why-does-it-matter&quot;&gt;why does it matter&lt;/a&gt;. If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. &lt;strong&gt;I will only ever publish binaries if they are reproducible&lt;/strong&gt;.&lt;/p&gt;
 &lt;p&gt;My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of &lt;a href=&quot;https://mirage.io&quot;&gt;MirageOS unikernels&lt;/a&gt; -- a unikernel is a statically linked ELF binary to be run as Unix process or &lt;a href=&quot;https://github.com/solo5/solo5&quot;&gt;virtual machine&lt;/a&gt;. MirageOS heavily uses &lt;a href=&quot;https://ocaml.org&quot;&gt;OCaml&lt;/a&gt; and &lt;a href=&quot;https://opam.ocaml.org&quot;&gt;opam&lt;/a&gt;, the OCaml package manager, and is an opam package itself. Thus, &lt;em&gt;checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package&lt;/em&gt;.&lt;/p&gt;
-&lt;h2&gt;Reproducible builds with opam&lt;/h2&gt;
+&lt;h2 id=&quot;reproducible-builds-with-opam&quot;&gt;Reproducible builds with opam&lt;/h2&gt;
 &lt;p&gt;Testing for reproducibility is achieved by taking the sources and compile them twice independently. Afterwards the equality of the resulting binaries can be checked. In trivial projects, the sources is just a single file, or originate from a single tarball. In OCaml, opam uses &lt;a href=&quot;https://github.com/ocaml/opam-repository&quot;&gt;a community repository&lt;/a&gt; where OCaml developers publish their package releases to, but can also use custom repositores, and in addition pin packages to git remotes (url including branch or commit), or a directory on the local filesystem. Manually tracking and updating all dependent packages of a MirageOS unikernel is not feasible: our hello-world compiled for hvt (kvm/BHyve) already has 79 opam dependencies, including the OCaml compiler which is distribued as opam package. The unikernel serving this website depends on 175 opam packages.&lt;/p&gt;
 &lt;p&gt;Conceptually there should be two tools, the &lt;em&gt;initial builder&lt;/em&gt;, which takes the latest opam packages which do not conflict, and exports exact package versions used during the build, as well as hashes of binaries. The other tool is a &lt;em&gt;rebuilder&lt;/em&gt;, which imports the export, conducts a build, and outputs the hashes of the produced binaries.&lt;/p&gt;
 &lt;p&gt;Opam has the concept of a &lt;code&gt;switch&lt;/code&gt;, which is an environment where a package set is installed. Switches are independent of each other, and can already be exported and imported. Unfortunately the export is incomplete: if a package includes additional patches as part of the repository -- sometimes needed for fixing releases where the actual author or maintainer of a package responds slowly -- these package neither the patches end up in the export. Also, if a package is pinned to a git branch, the branch appears in the export, but this may change over time by pushing more commits or even force-pushing to that branch. In &lt;a href=&quot;https://github.com/ocaml/opam/pull/4040&quot;&gt;PR #4040&lt;/a&gt; (under discussion and review), also developed during the summit, I propose to embed the additional files as base64 encoded values in the opam file. To solve the latter issue, I modified the export mechanism to &lt;a href=&quot;https://github.com/ocaml/opam/pull/4055&quot;&gt;embed the git commit hash (PR #4055)&lt;/a&gt;, and avoid sources from a local directory and which do not have a checksum.&lt;/p&gt;
 &lt;p&gt;So the opam export contains the information required to gather the exact same sources and build instructions of the opam packages. If the opam repository would be self-contained (i.e. not depend on any other tools), this would be sufficient. But opam does not run in thin air, it requires some system utilities such as &lt;code&gt;/bin/sh&lt;/code&gt;, &lt;code&gt;sed&lt;/code&gt;, a GNU make, commonly &lt;code&gt;git&lt;/code&gt;, a C compiler, a linker, an assembler. Since opam is available on various operating systems, the plugin &lt;code&gt;depext&lt;/code&gt; handles host system dependencies, e.g. if your opam package requires &lt;code&gt;gmp&lt;/code&gt; to be installed, this requires slightly different names depending on host system or distribution, take a look at &lt;a href=&quot;https://github.com/ocaml/opam-repository/blob/master/packages/conf-gmp/conf-gmp.1/opam&quot;&gt;conf-gmp&lt;/a&gt;. This also means, opam has rather good information about both the opam dependencies and the host system dependencies for each package. Please note that the host system packages used during compilation are not yet recorded (i.e. which &lt;code&gt;gmp&lt;/code&gt; package was installed and used during the build, only that a &lt;code&gt;gmp&lt;/code&gt; package has to be installed). The base utilities mentioned above (C compiler, linker, shell) are also not recorded yet.&lt;/p&gt;
 &lt;p&gt;Operating system information available in opam (such as architecture, distribution, version), which in some cases maps to exact base utilities, is recorded in the build-environment, a separate artifact. The environment variable &lt;a href=&quot;https://reproducible-builds.org/specs/source-date-epoch/&quot;&gt;&lt;code&gt;SOURCE_DATE_EPOCH&lt;/code&gt;&lt;/a&gt;, used for communicating the same timestamp when software is required to record a timestamp into the resulting binary, is also captured in the build environment.&lt;/p&gt;
 &lt;p&gt;Additional environment variables may be captured or used by opam packages to produce different output. To avoid this, both the initial builder and the rebuilder are run with minimal environment variables: only &lt;code&gt;PATH&lt;/code&gt; (normalised to a whitelist of &lt;code&gt;/bin&lt;/code&gt;, &lt;code&gt;/usr/bin&lt;/code&gt;, &lt;code&gt;/usr/local/bin&lt;/code&gt; and &lt;code&gt;/opt/bin&lt;/code&gt;) and &lt;code&gt;HOME&lt;/code&gt; are defined. Missing information at the moment includes CPU features: some libraries (gmp?, nocrypto) emit different code depending on the CPU feature.&lt;/p&gt;
-&lt;h2&gt;Tooling&lt;/h2&gt;
+&lt;h2 id=&quot;tooling&quot;&gt;Tooling&lt;/h2&gt;
 &lt;p&gt;&lt;em&gt;TL;DR: A &lt;strong&gt;build&lt;/strong&gt; builds an opam package, and outputs &lt;code&gt;.opam-switch&lt;/code&gt;, &lt;code&gt;.build-hashes.N&lt;/code&gt;, and &lt;code&gt;.build-environment.N&lt;/code&gt;. A &lt;strong&gt;rebuild&lt;/strong&gt; uses these artifacts as input, builds the package and outputs another &lt;code&gt;.build-hashes.M&lt;/code&gt; and &lt;code&gt;.build-environment.M&lt;/code&gt;.&lt;/em&gt;&lt;/p&gt;
 &lt;p&gt;The command-line utility &lt;code&gt;orb&lt;/code&gt; can be installed and used:&lt;/p&gt;
 &lt;pre&gt;&lt;code class=&quot;language-sh&quot;&gt;$ opam pin add orb git+https://github.com/hannesm/orb.git#active
@ -1037,7 +1037,7 @@ $ orb build --twice --keep-build-dir --diffoscope &amp;lt;your-favourite-opam-pa
 &lt;p&gt;It provides two subcommands &lt;code&gt;build&lt;/code&gt; and &lt;code&gt;rebuild&lt;/code&gt;. The &lt;code&gt;build&lt;/code&gt; command takes a list of local opam &lt;code&gt;--repos&lt;/code&gt; where to take opam packages from (defaults to &lt;code&gt;default&lt;/code&gt;), a compiler (either a variant &lt;code&gt;--compiler=4.09.0+flambda&lt;/code&gt;, a version &lt;code&gt;--compiler=4.06.0&lt;/code&gt;, or a pin to a local development version &lt;code&gt;--compiler-pin=~/ocaml&lt;/code&gt;), and optionally an existing switch &lt;code&gt;--use-switch&lt;/code&gt;. It creates a switch, builds the packages, and emits the opam export, hashes of all files installed by these packages, and the build environment. The flags &lt;code&gt;--keep-build&lt;/code&gt; retains the build products, opam's &lt;code&gt;--keep-build-dir&lt;/code&gt; in addition temporary build products and generated source code. If &lt;code&gt;--twice&lt;/code&gt; is provided, a rebuild (described next) is executed after the initial build.&lt;/p&gt;
 &lt;p&gt;The &lt;code&gt;rebuild&lt;/code&gt; command takes a directory with the opam export and build environment to build the opam package. It first compares the build-environment with the host system, sets the &lt;code&gt;SOURCE_DATE_EPOCH&lt;/code&gt; and switch location accordingly and executes the import. Once the build is finished, it compares the hashes of the resulting files with the previous run. On divergence, if build directories were kept in the previous build, and if diffoscope is available and &lt;code&gt;--diffoscope&lt;/code&gt; was provided, diffoscope is run on the diverging files. If &lt;code&gt;--keep-build-dir&lt;/code&gt; was provided as well, &lt;code&gt;diff -ur&lt;/code&gt; can be used to compare the temporary build and sources, including build logs.&lt;/p&gt;
 &lt;p&gt;The builds are run in parallel, as opam does, this parallelism does not lead to different binaries in my experiments.&lt;/p&gt;
-&lt;h2&gt;Results and discussion&lt;/h2&gt;
+&lt;h2 id=&quot;results-and-discussion&quot;&gt;Results and discussion&lt;/h2&gt;
 &lt;p&gt;&lt;strong&gt;All MirageOS unikernels I have deployed are reproducible \o/&lt;/strong&gt;. Also, several binaries such as &lt;code&gt;orb&lt;/code&gt; itself, &lt;code&gt;opam&lt;/code&gt;, &lt;code&gt;solo5-hvt&lt;/code&gt;, and all &lt;code&gt;albatross&lt;/code&gt; utilities are reproducible.&lt;/p&gt;
 &lt;p&gt;The unikernel range from hello world, web servers (e.g. this blog, getting its data on startup via a git clone to memory), authoritative DNS servers, CalDAV server. They vary in size between 79 and 200 opam packages, resulting in 2MB - 16MB big ELF binaries (including debug symbols). The &lt;a href=&quot;https://github.com/roburio/reproducible-unikernel-repo&quot;&gt;unikernel opam repository&lt;/a&gt; contains some reproducible unikernels used for testing. Some work-in-progress enhancements are needed to achieve this:&lt;/p&gt;
 &lt;p&gt;At the moment, the opam package of a MirageOS unikernel is automatically generated by &lt;code&gt;mirage configure&lt;/code&gt;, but only used for tracking opam dependencies. I worked on &lt;a href=&quot;https://github.com/mirage/mirage/pull/1022&quot;&gt;mirage PR #1022&lt;/a&gt; to extend the generated opam package with build and install instructions.&lt;/p&gt;
@ -1046,7 +1046,7 @@ $ orb build --twice --keep-build-dir --diffoscope &amp;lt;your-favourite-opam-pa
 &lt;p&gt;In functoria, a tool used to configure MirageOS devices and their dependencies, can emit a list of opam packages which were required to build the unikernel. This uses &lt;code&gt;opam list --required-by --installed --rec &amp;lt;pkgs&amp;gt;&lt;/code&gt;, which uses the cudf graph (&lt;a href=&quot;https://github.com/mirage/functoria/pull/189#issuecomment-566696426&quot;&gt;thanks to Raja for explanation&lt;/a&gt;), that is during the rebuild dropping some packages. The &lt;a href=&quot;https://github.com/mirage/functoria/pull/189&quot;&gt;PR #189&lt;/a&gt; avoids by not using the &lt;code&gt;--rec&lt;/code&gt; argument, but manually computing the fixpoint.&lt;/p&gt;
 &lt;p&gt;Certainly, the choice of environment variables, and whether to vary them (as &lt;a href=&quot;https://tests.reproducible-builds.org/debian/index_variations.html&quot;&gt;debian does&lt;/a&gt;) or to not define them (or normalise) while building, is arguably. Since MirageOS does neither support time zone nor internationalisation, there is no need to prematurely solving this issue. On related note, even with different locale settings, MirageOS unikernels are reproducible apart from an &lt;a href=&quot;https://github.com/backtracking/ocamlgraph/pull/90&quot;&gt;issue in ocamlgraph #90&lt;/a&gt; embedding the output of &lt;a href=&quot;https://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html&quot;&gt;&lt;code&gt;date&lt;/code&gt;&lt;/a&gt;, which is different depending on &lt;code&gt;LANG&lt;/code&gt; and locale (&lt;code&gt;LC_*&lt;/code&gt;) settings.&lt;/p&gt;
 &lt;p&gt;Prior art in reproducible MirageOS unikernels is the &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall/&quot;&gt;mirage-qubes-firewall&lt;/a&gt;. Since &lt;a href=&quot;https://github.com/mirage/qubes-mirage-firewall/commit/07ff3d61477383860216c69869a1ffee59145e45&quot;&gt;early 2017&lt;/a&gt; it is reproducible. Their approach is different by building in a docker container with the opam repository pinned to an exact git commit.&lt;/p&gt;
-&lt;h2&gt;Further work&lt;/h2&gt;
+&lt;h2 id=&quot;further-work&quot;&gt;Further work&lt;/h2&gt;
 &lt;p&gt;I only tested a certain subset of opam packages and MirageOS unikernels, mainly on a single machine (my laptop) running FreeBSD, and am happy if others will test reproducibility of their OCaml programs with the tools provided. There could as well be CI machines rebuilding opam packages and reporting results to a central repository. I'm pretty sure there are more reproducibility issues in the opam ecosystem. I developed an &lt;a href=&quot;https://github.com/roburio/reproducible-testing-repo&quot;&gt;reproducible testing opam repository&lt;/a&gt; with opam packages that do not depend on OCaml, mainly for further tooling development. Some tests were also conducted on a Debian system with the same result. The variations, apart from build time, were using a different user, and different locale settings.&lt;/p&gt;
 &lt;p&gt;As mentioned above, more environment, such as the CPU features, and external system packages, should be captured in the build environment.&lt;/p&gt;
 &lt;p&gt;When comparing OCaml libraries, some output files (cmt / cmti / cma / cmxa) are not deterministic, but contain minimal diverge where I was not able to spot the root cause. It would be great to fix this, likely in the OCaml compiler distribution. Since the final result, the binary I'm interested in, is not affected by non-identical intermediate build products, I hope someone (you?) is interested in improving on this side. OCaml bytecode output also seems to be non-deterministic. There is &lt;a href=&quot;https://github.com/coq/coq/issues/11229&quot;&gt;a discussion on the coq issue tracker&lt;/a&gt; which may be related.&lt;/p&gt;
@ -1055,10 +1055,10 @@ $ orb build --twice --keep-build-dir --diffoscope &amp;lt;your-favourite-opam-pa
 &lt;p&gt;What was fun was to compare the unikernel when built on Linux with gcc against a built on FreeBSD with clang and lld - spoiler: they emit debug sections with different dwarf versions, it is pretty big. Other fun differences were between OCaml compiler versions: the difference between minor versions (4.08.0 vs 4.08.1) is pretty small (~100kB as human-readable output), while the difference between major version (4.08.1 vs 4.09.0) is rather big (~900kB as human-readable diff).&lt;/p&gt;
 &lt;p&gt;An item on my list for the future is to distribute the opam export, build hashes and build environment artifacts in a authenticated way. I want to integrate this as &lt;a href=&quot;https://in-toto.io/&quot;&gt;in-toto&lt;/a&gt; style into &lt;a href=&quot;https://github.com/hannesm/conex&quot;&gt;conex&lt;/a&gt;, my not-yet-deployed implementation of &lt;a href=&quot;https://theupdateframework.github.io/&quot;&gt;tuf&lt;/a&gt; for opam that needs further development and a test installation, hopefully in 2020.&lt;/p&gt;
 &lt;p&gt;If you want to support our work on MirageOS unikernels, please &lt;a href=&quot;https://robur.coop/Donate&quot;&gt;donate to robur&lt;/a&gt;. I'm interested in feedback, either via &lt;a href=&quot;https://twitter.com/h4nnes&quot;&gt;twitter&lt;/a&gt;, &lt;a href=&quot;https://mastodon.social/@hannesm&quot;&gt;hannesm@mastodon.social&lt;/a&gt; or via eMail.&lt;/p&gt;
-</content><category scheme="https://hannes.robur.coop/tags/package%20signing" term="package signing"/><category scheme="https://hannes.robur.coop/tags/security" term="security"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:09922d6b-56c8-595d-8086-5aef9656cbc4</id><title type="text">Reproducible MirageOS unikernel builds</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="text">&lt;p&gt;Five years since ocaml-x509 initial release, it has been reworked and used more widely&lt;/p&gt;
-</summary><published>2019-08-15T11:21:30-00:00</published><link href="/Posts/X50907" rel="alternate"/><content type="html">&lt;h2&gt;Cryptographic material&lt;/h2&gt;
+</content><category scheme="https://hannes.robur.coop/tags/package%20signing" term="package signing"/><category scheme="https://hannes.robur.coop/tags/security" term="security"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:09922d6b-56c8-595d-8086-5aef9656cbc4</id><title type="text">Reproducible MirageOS unikernel builds</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html">&lt;p&gt;Five years since ocaml-x509 initial release, it has been reworked and used more widely&lt;/p&gt;
+</summary><published>2019-08-15T11:21:30-00:00</published><link href="/Posts/X50907" rel="alternate"/><content type="html">&lt;h2 id=&quot;cryptographic-material&quot;&gt;Cryptographic material&lt;/h2&gt;
 &lt;p&gt;Once a private and public key pair is generated (doesn't matter whether it is plain RSA, DSA, ECC on any curve), this is fine from a scientific point of view, and can already be used for authenticating and encrypting. From a practical point of view, the public parts need to be exchanged and verified (usually a fingerprint or hash thereof). This leads to the struggle how to encode this cryptographic material, and how to embed an identity (or multiple), capabilities, and other information into it. &lt;a href=&quot;https://en.wikipedia.org/wiki/X.509&quot;&gt;X.509&lt;/a&gt; is a standard to solve this encoding and embedding, and provides more functionality, such as establishing chains of trust and revocation of invalidated or compromised material. X.509 uses certificates, which contain the public key, and additional information (in a extensible key-value store), and are signed by an issuer, either the private key corresponding to the public key - a so-called self-signed certificate - or by a different private key, an authority one step up the chain. A rather long, but very good introduction to certificates by Mike Malone is &lt;a href=&quot;https://smallstep.com/blog/everything-pki.html&quot;&gt;available here&lt;/a&gt;.&lt;/p&gt;
-&lt;h2&gt;OCaml ecosystem evolving&lt;/h2&gt;
+&lt;h2 id=&quot;ocaml-ecosystem-evolving&quot;&gt;OCaml ecosystem evolving&lt;/h2&gt;
 &lt;p&gt;More than 5 years ago David Kaloper and I &lt;a href=&quot;https://mirage.io/blog/introducing-x509&quot;&gt;released the initial ocaml-x509&lt;/a&gt; package as part of our &lt;a href=&quot;https://nqsb.io&quot;&gt;TLS stack&lt;/a&gt;, which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in &lt;a href=&quot;https://tools.ietf.org/html/rfc6125&quot;&gt;RFC 5280&lt;/a&gt;). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's &lt;a href=&quot;https://github.com/mirleft/ocaml-asn1-combinators&quot;&gt;asn1-combinators&lt;/a&gt; library changed much over time.&lt;/p&gt;
 &lt;p&gt;The OCaml ecosystem evolved over the years, which lead to some changes:&lt;/p&gt;
 &lt;ul&gt;
@ -1085,27 +1085,27 @@ $ orb build --twice --keep-build-dir --diffoscope &amp;lt;your-favourite-opam-pa
 &lt;li&gt;Usage of the &lt;a href=&quot;https://github.com/mirage/alcotest&quot;&gt;alcotest&lt;/a&gt; unit testing framework (instead of oUnit).
 &lt;/li&gt;
 &lt;/ul&gt;
-&lt;h2&gt;More use cases for X.509&lt;/h2&gt;
+&lt;h2 id=&quot;more-use-cases-for-x.509&quot;&gt;More use cases for X.509&lt;/h2&gt;
 &lt;p&gt;Initially, we designed and used ocaml-x509 for providing TLS server endpoints and validation in TLS clients - mostly on the public web, where each operating system ships a set of ~100 trust anchors to validate any web server certificate against. But once you have a X.509 implementation, every authentication problem can be solved by applying it.&lt;/p&gt;
-&lt;h3&gt;Authentication with path building&lt;/h3&gt;
+&lt;h3 id=&quot;authentication-with-path-building&quot;&gt;Authentication with path building&lt;/h3&gt;
 &lt;p&gt;It turns out that the trust anchor sets are not equal across operating systems and versions, thus some web servers serve sets, instead of chains, of certificates - as described in &lt;a href=&quot;https://tools.ietf.org/html/rfc4158&quot;&gt;RFC 4158&lt;/a&gt;, where the client implementation needs to build valid paths and accept a connection if any path can be validated. The path building was initially in 0.5.2 slightly wrong, but fixed quickly in &lt;a href=&quot;https://github.com/mirleft/ocaml-x509/commit/1a1476308d24bdcc49d45c4cd9ef539ca57461d2&quot;&gt;0.5.3&lt;/a&gt;.&lt;/p&gt;
-&lt;h3&gt;Fingerprint authentication&lt;/h3&gt;
+&lt;h3 id=&quot;fingerprint-authentication&quot;&gt;Fingerprint authentication&lt;/h3&gt;
 &lt;p&gt;The chain of trust validation is useful for the open web, where you as software developer don't know to which remote endpoint your software will ever connect to - as long as the remote has a certificate signed (via intermediates) by any of the trust anchors. In the early days, before &lt;a href=&quot;https://letsencrypt.org/&quot;&gt;let's encrypt&lt;/a&gt; was launched and embedded as trust anchors (or cross-signed by already deployed trust anchors), operators needed to pay for a certificate - a business model where some CAs did not bother to check the authenticity of a certificate signing request, and thus random people owning valid certificates for microsoft.com or google.com.&lt;/p&gt;
 &lt;p&gt;Instead of using the set of trust anchors, the fingerprint of the server certificate, or preferably the fingerprint of the public key of the certificate, can be used for authentication, as optionally done since some years in &lt;a href=&quot;https://github.com/hannesm/jackline/commit/a1e6f3159be1e45e6b690845e1b29366c41239a2&quot;&gt;jackline&lt;/a&gt;, an XMPP client. Support for this certificate / public key pinning was added in x509 0.2.1 / 0.5.0.&lt;/p&gt;
-&lt;h3&gt;Certificate signing requests&lt;/h3&gt;
+&lt;h3 id=&quot;certificate-signing-requests&quot;&gt;Certificate signing requests&lt;/h3&gt;
 &lt;p&gt;Until x509 0.4.0 there was no support for generating certificate signing requests (CSR), as defined in PKCS 10, which are self-signed blobs containing a public key, an identity, and possibly extensions. Such as CSR is sent to the certificate authority, and after validation of ownership of the identity and paying a fee, the certificate is issued. Let's encrypt specified the ACME protocol which automates the proof of ownership: they provide a HTTP API for requesting a challenge, providing the response (the proof of ownership) via HTTP or DNS, and then allow the submission of a CSR and downloading the signed certificate. The ocaml-x509 library provides operations for creating such a CSR, and also for signing a CSR to generate a certificate.&lt;/p&gt;
 &lt;p&gt;Mindy developed the command-line utility &lt;a href=&quot;https://github.com/yomimono/ocaml-certify/&quot;&gt;certify&lt;/a&gt; which uses these operations from the ocaml-x509 library and acts as a swiss-army knife purely in OCaml for these required operations.&lt;/p&gt;
 &lt;p&gt;Maker developed a &lt;a href=&quot;https://github.com/mmaker/ocaml-letsencrypt&quot;&gt;let's encrypt library&lt;/a&gt; which implements the above mentioned ACME protocol for provisioning CSR to certificates, also using our ocaml-x509 library.&lt;/p&gt;
 &lt;p&gt;To complete the required certificate authority functionality, in x509 0.6.0 certificate revocation lists, both validation and signing, was implemented.&lt;/p&gt;
-&lt;h3&gt;Deploying unikernels&lt;/h3&gt;
+&lt;h3 id=&quot;deploying-unikernels&quot;&gt;Deploying unikernels&lt;/h3&gt;
 &lt;p&gt;As &lt;a href=&quot;/Posts/VMM&quot;&gt;described in another post&lt;/a&gt;, I developed &lt;a href=&quot;https://github.com/hannesm/albatross&quot;&gt;albatross&lt;/a&gt;, an orchestration system for MirageOS unikernels. This uses ASN.1 for internal socket communication and allows remote management via a TLS connection which is mutually authenticated with a X.509 client certificate. To encrypt the X.509 client certificate, first a TLS handshake where the server authenticates itself to the client is established, and over that connection another TLS handshake is established where the client certificate is requested. Note that this mechanism can be dropped with TLS 1.3, since there the certificates are transmitted over an already encrypted channel.&lt;/p&gt;
 &lt;p&gt;The client certificate already contains the command to execute remotely - as a custom extension, being it &amp;quot;show me the console output&amp;quot;, or &amp;quot;destroy the unikernel with name = YYY&amp;quot;, or &amp;quot;deploy the included unikernel image&amp;quot;. The advantage is that the commands are already authenticated, and there is no need for developing an ad-hoc protocol on top of the TLS session. The resource limits, assigned by the authority, are also part of the certificate chain - i.e. the number of unikernels, access to network bridges, available accumulated memory, accumulated size for block devices, are constrained by the certificate chain presented to the server, and currently running unikernels. The names of the chain are used for access control - if Alice and Bob have intermediate certificates from the same CA, neither Alice may manage Bob's unikernels, nor Bob may manage Alice's unikernels. I'm using albatross since 2.5 years in production on two physical machines with ~20 unikernels total (multiple users, multiple administrative domains), and it works stable and is much nicer to deal with than &lt;code&gt;scp&lt;/code&gt; and custom hacked shell scripts.&lt;/p&gt;
-&lt;h2&gt;Why 0.7?&lt;/h2&gt;
+&lt;h2 id=&quot;why-0.7&quot;&gt;Why 0.7?&lt;/h2&gt;
 &lt;p&gt;There are still some missing pieces in our ocaml-x509 implementation, namely modern ECC certificates (depending on elliptic curve primitives not yet available in OCaml), RSA-PSS signing (should be straightforward), PKCS 12 (there is a &lt;a href=&quot;https://github.com/mirleft/ocaml-x509/pull/114&quot;&gt;pull request&lt;/a&gt;, but this should wait until asn1-combinators supports the &lt;code&gt;ANY defined BY&lt;/code&gt; construct to cleanup the code), ...
 Once these features are supported, the library should likely be named PKCS since it supports more than X.509, and released as 1.0.&lt;/p&gt;
 &lt;p&gt;The 0.7 release series moved a lot of modules and function names around, thus it is a major breaking release. By using a map instead of lists for extensions, GeneralName, ..., the API was further revised - invariants that each extension key (an ASN.1 object identifier) may occur at most once are now enforced. By not leaking exceptions through the public interface, the API is easier to use safely - see &lt;a href=&quot;https://github.com/mmaker/ocaml-letsencrypt/commit/dc53518f46310f384c9526b1d96a8e8f815a09c7&quot;&gt;let's encrypt&lt;/a&gt;, &lt;a href=&quot;https://git.robur.io/?p=openvpn.git;a=commitdiff;h=929c53116c1438ba1214f53df7506d32da566ccc&quot;&gt;openvpn&lt;/a&gt;, &lt;a href=&quot;https://github.com/yomimono/ocaml-certify/pull/17&quot;&gt;certify&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirleft/ocaml-tls/pull/394&quot;&gt;tls&lt;/a&gt;, &lt;a href=&quot;https://github.com/mirage/capnp-rpc/pull/158&quot;&gt;capnp&lt;/a&gt;, &lt;a href=&quot;https://github.com/hannesm/albatross/commit/50ed6a8d1ead169b3e322aaccb469e870ad72acc&quot;&gt;albatross&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;I intended in 0.7.0 to have much more precise types, esp. for the SubjectAlternativeName (SAN) extension that uses a GeneralName, but it turns out the GeneralName is as well used for NameConstraints (NC) in a different way -- IP in SAN is an IPv4 or IPv6 address, in CN it is the IP/netmask; DNS is a domain name in SAN, in CN it is a name starting with a leading dot (i.e. &amp;quot;.example.com&amp;quot;), which is not a valid domain name. In 0.7.1, based on a bug report, I had to revert these variants and use less precise types.&lt;/p&gt;
-&lt;h2&gt;Conclusion&lt;/h2&gt;
+&lt;h2 id=&quot;conclusion&quot;&gt;Conclusion&lt;/h2&gt;
 &lt;p&gt;The work on X.509 was sponsored by &lt;a href=&quot;http://ocamllabs.io/&quot;&gt;OCaml Labs&lt;/a&gt;. You can support our work at robur by a &lt;a href=&quot;https://robur.io/Donate&quot;&gt;donation&lt;/a&gt;, which we will use to work on our OCaml and MirageOS projects. You can also reach out to us to realize commercial products.&lt;/p&gt;
 &lt;p&gt;I'm interested in feedback, either via &lt;strike&gt;&lt;a href=&quot;https://twitter.com/h4nnes&quot;&gt;twitter&lt;/a&gt;&lt;/strike&gt; &lt;a href=&quot;https://mastodon.social/@hannesm&quot;&gt;hannesm@mastodon.social&lt;/a&gt; or via eMail.&lt;/p&gt;
 </content><category scheme="https://hannes.robur.coop/tags/tls" term="tls"/><category scheme="https://hannes.robur.coop/tags/security" term="security"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:f2cf2a6a-8eef-5c2c-be03-d81a1bf0f066</id><title type="text">X509 0.7</title><updated>2021-11-19T18:04:52-00:00</updated><author><name>hannes</name></author></entry></feed>