Compare commits
No commits in common. "main" and "pages" have entirely different histories.
58 changed files with 3575 additions and 3448 deletions
|
|
@ -1 +0,0 @@
|
||||||
full stack engineer
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
Posts
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
https://hannes.robur.coop
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
981361ca-e71d-4997-a52c-baeee78e4156
|
|
||||||
143
About
143
About
|
|
@ -1,113 +1,88 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: About
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>About</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="About" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>About</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/myself" class="tag">myself</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-01 (last updated: 2023-11-20)</span><article><h2 id="what-is-a-full-stack-engineer">What is a "full stack engineer"?</h2>
|
||||||
author: hannes
|
<p>Analysing the word literally, we should start with silicon and some electrons,
|
||||||
tags: overview, myself, background
|
|
||||||
abstract: introduction (myself, this site)
|
|
||||||
---
|
|
||||||
|
|
||||||
## What is a "full stack engineer"?
|
|
||||||
|
|
||||||
Analysing the word literally, we should start with silicon and some electrons,
|
|
||||||
maybe a soldering iron, and build everything all the way up to our favourite
|
maybe a soldering iron, and build everything all the way up to our favourite
|
||||||
communication system.
|
communication system.</p>
|
||||||
|
<p>While I know how to solder, I don't plan to write about hardware in here. I'll
|
||||||
While I know how to solder, I don't plan to write about hardware in here. I'll
|
|
||||||
assume that off-the-shelf hardware (arm/amd64) is available and trustworthy.
|
assume that off-the-shelf hardware (arm/amd64) is available and trustworthy.
|
||||||
Read the [Intel x86 considered
|
Read the <a href="http://blog.invisiblethings.org/papers/2015/x86_harmful.pdf">Intel x86 considered
|
||||||
harmful](http://blog.invisiblethings.org/papers/2015/x86_harmful.pdf) paper in
|
harmful</a> paper in
|
||||||
case you're interested in trustworthiness of hardware.
|
case you're interested in trustworthiness of hardware.</p>
|
||||||
|
<p>My current obsession is to enable people to take back control over their data:
|
||||||
My current obsession is to enable people to take back control over their data:
|
|
||||||
simple to setup, secure, decentralised infrastructure. We're not there yet,
|
simple to setup, secure, decentralised infrastructure. We're not there yet,
|
||||||
which also means I've plenty of projects :).
|
which also means I've plenty of projects :).</p>
|
||||||
|
<p>I will write about my projects, which cover topics on various software layers.</p>
|
||||||
I will write about my projects, which cover topics on various software layers.
|
<h3 id="myself">Myself</h3>
|
||||||
|
<p>I'm Hannes Mehnert, a <a href="http://www.catb.org/jargon/html/H/hacker.html">hacker</a>
|
||||||
### Myself
|
|
||||||
|
|
||||||
I'm Hannes Mehnert, a [hacker](http://www.catb.org/jargon/html/H/hacker.html)
|
|
||||||
(in the original sense of the word), 3X years old. In my spare time, I'm not
|
(in the original sense of the word), 3X years old. In my spare time, I'm not
|
||||||
only a hacker, but also a barista. I like to travel and repair my recumbent
|
only a hacker, but also a barista. I like to travel and repair my recumbent
|
||||||
bicycle.
|
bicycle.</p>
|
||||||
|
<p>Back in 199X, my family bought a PC. It came
|
||||||
Back in 199X, my family bought a PC. It came
|
|
||||||
with MS-DOS installed, I also remember Windows 3.1 (likely on a later computer).
|
with MS-DOS installed, I also remember Windows 3.1 (likely on a later computer).
|
||||||
This didn't really hook me into computers, but over the years I started with
|
This didn't really hook me into computers, but over the years I started with
|
||||||
friends to modify some computer games (e.g. modifying text of Civilization). I
|
friends to modify some computer games (e.g. modifying text of Civilization). I
|
||||||
first encountered programming in high school around 1995: Borland's Turbo Pascal
|
first encountered programming in high school around 1995: Borland's Turbo Pascal
|
||||||
(which chased me for several years).
|
(which chased me for several years).</p>
|
||||||
|
<p>Fast forwarding a bit, I learned about the operating system Linux (starting with
|
||||||
Fast forwarding a bit, I learned about the operating system Linux (starting with
|
|
||||||
SUSE 6.4) and got hooked (by providing basic network services (NFS/YP/Samba)) to
|
SUSE 6.4) and got hooked (by providing basic network services (NFS/YP/Samba)) to
|
||||||
UNIX. In 2000 I joined the [Chaos Computer Club](https://www.ccc.de).
|
UNIX. In 2000 I joined the <a href="https://www.ccc.de">Chaos Computer Club</a>.
|
||||||
Over the years I learned various things, from Linux kernel modifications,
|
Over the years I learned various things, from Linux kernel modifications,
|
||||||
Perl, PHP, basic network and security. I use [FreeBSD](https://www.FreeBSD.org) since 4.5, FreeBSD-CURRENT
|
Perl, PHP, basic network and security. I use <a href="https://www.FreeBSD.org">FreeBSD</a> since 4.5, FreeBSD-CURRENT
|
||||||
on my laptop. I helped to [reverse engineer and analyse the security of a voting
|
on my laptop. I helped to <a href="http://wijvertrouwenstemcomputersniet.nl">reverse engineer and analyse the security of a voting
|
||||||
computer](http://wijvertrouwenstemcomputersniet.nl) in the Netherlands, and some
|
computer</a> in the Netherlands, and some
|
||||||
[art installations](http://blinkenlights.net/) in Berlin and Paris. There were
|
<a href="http://blinkenlights.net/">art installations</a> in Berlin and Paris. There were
|
||||||
several annual Chaos Communication Congresses where I co-setup the network
|
several annual Chaos Communication Congresses where I co-setup the network
|
||||||
(backbone, access layer, wireless, network services such as DHCP/DNS), struggling with
|
(backbone, access layer, wireless, network services such as DHCP/DNS), struggling with
|
||||||
Cisco hardware from their demo pool, and also amongst others HP, Force10, Lucent, Juniper
|
Cisco hardware from their demo pool, and also amongst others HP, Force10, Lucent, Juniper
|
||||||
equipment.
|
equipment.</p>
|
||||||
|
<p>In the early 200X I started to program <a href="https://opendylan.org">Dylan</a>, a LISP
|
||||||
In the early 200X I started to program [Dylan](https://opendylan.org), a LISP
|
|
||||||
dialect (dynamic, multiple inheritance, object-oriented), which even resulted in
|
dialect (dynamic, multiple inheritance, object-oriented), which even resulted in
|
||||||
a [TCP/IP
|
a <a href="https://github.com/dylan-hackers/network-night-vision/">TCP/IP
|
||||||
implementation](https://github.com/dylan-hackers/network-night-vision/)
|
implementation</a>
|
||||||
including a wireshark-like GTK based user interface with a shell similar to IOS for configuring the stack.
|
including a wireshark-like GTK based user interface with a shell similar to IOS for configuring the stack.</p>
|
||||||
|
<p>I got excited about programming languages and type theory (thanks to
|
||||||
I got excited about programming languages and type theory (thanks to
|
<a href="https://www.cis.upenn.edu/~bcpierce/tapl/">types and programming languages</a>, an
|
||||||
[types and programming languages](https://www.cis.upenn.edu/~bcpierce/tapl/), an
|
excellent book); a key event for me was the <a href="http://cs.au.dk/~danvy/icfp05/">international conference on functional programming (ICFP)</a>. I wondered how a
|
||||||
excellent book); a key event for me was the [international conference on functional programming (ICFP)](http://cs.au.dk/~danvy/icfp05/). I wondered how a
|
<a href="http://homes.soic.indiana.edu/jsiek/what-is-gradual-typing/">gradually typed</a>
|
||||||
[gradually typed](http://homes.soic.indiana.edu/jsiek/what-is-gradual-typing/)
|
|
||||||
Dylan would look like, leading to my master thesis. Gradual typing is the idea to evolve untyped programs into typed ones, and runtime type errors must be in the dynamic part. To me, this sounded like a great idea, to start with some random code, and add types later.
|
Dylan would look like, leading to my master thesis. Gradual typing is the idea to evolve untyped programs into typed ones, and runtime type errors must be in the dynamic part. To me, this sounded like a great idea, to start with some random code, and add types later.
|
||||||
My result was not too convincing (too slow, unsound type system).
|
My result was not too convincing (too slow, unsound type system).
|
||||||
Another problem with Dylan is that the community is very small, without sufficient time and energy to maintain the
|
Another problem with Dylan is that the community is very small, without sufficient time and energy to maintain the
|
||||||
self-hosted compiler(s) and the graphical IDE.
|
self-hosted compiler(s) and the graphical IDE.</p>
|
||||||
|
<p>During my studies I met <a href="http://www.itu.dk/people/sestoft/">Peter Sestoft</a>.
|
||||||
During my studies I met [Peter Sestoft](http://www.itu.dk/people/sestoft/).
|
|
||||||
After half a year off in New Zealand (working on formalising some type systems),
|
After half a year off in New Zealand (working on formalising some type systems),
|
||||||
I did a PhD in the ambitious research project "[Tools and methods for
|
I did a PhD in the ambitious research project "<a href="https://itu.dk/research/tomeso/">Tools and methods for
|
||||||
scalable software verification](https://itu.dk/research/tomeso/)", where we mechanised proofs of the functional correctness
|
scalable software verification</a>", where we mechanised proofs of the functional correctness
|
||||||
of imperative code (PIs: Peter and [Lars Birkedal](http://cs.au.dk/~birke/)).
|
of imperative code (PIs: Peter and <a href="http://cs.au.dk/~birke/">Lars Birkedal</a>).
|
||||||
The idea was great, the project was fun, but we ended with 3000 lines of proof
|
The idea was great, the project was fun, but we ended with 3000 lines of proof
|
||||||
script for a 100 line Java program. The Java program was taken off-the-shelf,
|
script for a 100 line Java program. The Java program was taken off-the-shelf,
|
||||||
several times refactored, and most of its shared mutable state was removed. The
|
several times refactored, and most of its shared mutable state was removed. The
|
||||||
proof script was in [Coq](https://coq.inria.fr), using our higher-order separation logic.
|
proof script was in <a href="https://coq.inria.fr">Coq</a>, using our higher-order separation logic.</p>
|
||||||
|
<p>I concluded two things: formal verification is hard and usually not applicable
|
||||||
I concluded two things: formal verification is hard and usually not applicable
|
for off-the-shelf software. <em>Since we have to rewrite the software anyways, why
|
||||||
for off-the-shelf software. *Since we have to rewrite the software anyways, why
|
not do it in a declarative way?</em></p>
|
||||||
not do it in a declarative way?*
|
<p>Some artefacts from that time are still around: an <a href="https://coqoon.github.io/">eclipse plugin for
|
||||||
|
Coq</a>, I also started (with David) the <a href="https://github.com/idris-hackers/idris-mode">idris-mode for
|
||||||
Some artefacts from that time are still around: an [eclipse plugin for
|
emacs</a>. Idris is a dependently
|
||||||
Coq](https://coqoon.github.io/), I also started (with David) the [idris-mode for
|
|
||||||
emacs](https://github.com/idris-hackers/idris-mode). Idris is a dependently
|
|
||||||
typed programming language (you can express richer types), actively being
|
typed programming language (you can express richer types), actively being
|
||||||
researched (I would not consider it production ready yet, needs more work on a
|
researched (I would not consider it production ready yet, needs more work on a
|
||||||
faster runtime, and libraries).
|
faster runtime, and libraries).</p>
|
||||||
|
<p>After I finished my PhD, I decided to slack off for some time to make decent
|
||||||
After I finished my PhD, I decided to slack off for some time to make decent
|
|
||||||
espresso. I ended up spending the winter (beginning of 2014) in Mirleft,
|
espresso. I ended up spending the winter (beginning of 2014) in Mirleft,
|
||||||
Morocco. A good friend of mine pointed me to [MirageOS](https://mirage.io), a
|
Morocco. A good friend of mine pointed me to <a href="https://mirage.io">MirageOS</a>, a
|
||||||
clean-slate operating system written in the high-level language [OCaml](https://ocaml.org). I got
|
clean-slate operating system written in the high-level language <a href="https://ocaml.org">OCaml</a>. I got
|
||||||
hooked pretty fast, after some experience with LISP machines I imagined a modern
|
hooked pretty fast, after some experience with LISP machines I imagined a modern
|
||||||
OS written in a single functional programming language.
|
OS written in a single functional programming language.</p>
|
||||||
|
<p>From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the <a href="https://www.cl.cam.ac.uk/~pes20/rems">rigorous engineering of mainstream systems</a> project) with <a href="https://www.cl.cam.ac.uk/~pes20/">Peter Sewell</a>. I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on <a href="http://robur.coop">robur</a>.</p>
|
||||||
From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the [rigorous engineering of mainstream systems](https://www.cl.cam.ac.uk/~pes20/rems) project) with [Peter Sewell](https://www.cl.cam.ac.uk/~pes20/). I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on [robur](http://robur.coop).
|
<p>MirageOS had various bits and pieces into place, including infrastructure for
|
||||||
|
|
||||||
MirageOS had various bits and pieces into place, including infrastructure for
|
|
||||||
building and testing (and a neat self-hosted website). A big gap was security.
|
building and testing (and a neat self-hosted website). A big gap was security.
|
||||||
No access control, no secure sockets layer, nothing. This will be the topic of
|
No access control, no secure sockets layer, nothing. This will be the topic of
|
||||||
another post.
|
another post.</p>
|
||||||
|
<p>OCaml is <a href="http://compcert.inria.fr/">academically</a> and <a href="https://blogs.janestreet.com/">commercially</a> used, compiles to native code (arm/amd64/likely more), is
|
||||||
OCaml is [academically](http://compcert.inria.fr/) and [commercially](https://blogs.janestreet.com/) used, compiles to native code (arm/amd64/likely more), is
|
fast enough ("Reassuring, because our blanket performance statement 'OCaml
|
||||||
fast enough ("Reassuring, because our blanket performance statement 'OCaml
|
|
||||||
delivers at least 50% of the performance of a decent C compiler' is
|
delivers at least 50% of the performance of a decent C compiler' is
|
||||||
not invalidated :-)" [Xavier Leroy](https://lwn.net/Articles/19378/)), and the [community](https://opam.ocaml.org/packages/) is sufficiently large.
|
not invalidated :-)" <a href="https://lwn.net/Articles/19378/">Xavier Leroy</a>), and the <a href="https://opam.ocaml.org/packages/">community</a> is sufficiently large.</p>
|
||||||
|
<h3 id="me-on-the-intertubes">Me on the intertubes</h3>
|
||||||
### Me on the intertubes
|
<p>You can find me on <a href="https://twitter.com/h4nnes">twitter</a> and on
|
||||||
|
<a href="https://github.com/hannesm">GitHub</a>.</p>
|
||||||
You can find me on [twitter](https://twitter.com/h4nnes) and on
|
<p>The data of this blog is <a href="https://git.robur.coop/hannes/hannes.robur.coop">stored in a git repository</a>.</p>
|
||||||
[GitHub](https://github.com/hannesm).
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
The data of this blog is [stored in a git repository](https://git.robur.coop/hannes/hannes.robur.coop).
|
|
||||||
129
Posts/ARP
129
Posts/ARP
|
|
@ -1,77 +1,44 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Re-engineering ARP
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Re-engineering ARP</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Re-engineering ARP" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Re-engineering ARP</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2016-07-12 (last updated: 2021-11-19)</span><article><h2 id="what-is-arp">What is ARP?</h2>
|
||||||
author: hannes
|
<p>ARP is the <em>A</em>ddress <em>R</em>esolution <em>P</em>rotocol, widely used in legacy IP networks (which support only IPv4). It is responsible to translate an IPv4 address to an Ethernet address. It is strictly more general, abstracting over protocol and hardware addresses. It is basically DNS (the domain name system) on a different layer.</p>
|
||||||
tags: mirageos, protocol
|
<p>ARP is link-local: ARP frames are not routed into other networks, all stay in the same broadcast domain. Thus there is no need for a hop limit (time-to-live). A reverse lookup mechanism (hardware address to protocol) is also available, named reverse ARP ;).</p>
|
||||||
abstract: If you want it as you like, you've to do it yourself
|
<p>I will focus on ARP in this article, as used widely to translate IPv4 addresses into Ethernet addresses. There are two operations in ARP: request and response. A request is usually broadcasted to all hosts (by setting the destination to the broadcast Ethernet address, <code>ff:ff:ff:ff:ff:ff</code>), while a reply is send via unicast (to the host which requested that information).</p>
|
||||||
---
|
<p>The frame format is pretty straightforward: 2 bytes hardware address type, 2 bytes protocol type, 1 byte length for both types, 2 bytes operation, followed by source addresses (hardware and protocol), and target addresses. In total 28 bytes, considering 48 bit Ethernet addresses and 32 bit IPv4 addresses.</p>
|
||||||
|
<p>It was initially specified in <a href="https://tools.ietf.org/html/rfc826">RFC 826</a>, but reading through <a href="https://tools.ietf.org/html/rfc1122">RFC 1122</a> (requirements for Internet Hosts - Communication layer), and maybe the newer <a href="https://tools.ietf.org/html/rfc5227">RFC 5227</a> (IPv4 address conflict detection) does not hurt.</p>
|
||||||
## What is ARP?
|
<p>On UNIX systems, you can investigate your arp table, also called arp cache, using the <code>arp</code> command line utility.</p>
|
||||||
|
<h3 id="protocol-logic">Protocol logic</h3>
|
||||||
ARP is the *A*ddress *R*esolution *P*rotocol, widely used in legacy IP networks (which support only IPv4). It is responsible to translate an IPv4 address to an Ethernet address. It is strictly more general, abstracting over protocol and hardware addresses. It is basically DNS (the domain name system) on a different layer.
|
<p>Let us look what our ARP handler actually needs to do? Translating IPv4 addresses to Ethernet addresses, but where does it learn new information?</p>
|
||||||
|
<p>First of all, our ARP handler needs to know its own IPv4 address and its Ethernet address. It will even broadcast them on startup, so-called gratuitous ARP. The purpose of this is to inform all other hosts on the same network that we are here now. And if another host, let's name it barf, has the same IPv4 address, some sort of conflict resolution needs to happen (otherwise all hosts on the network are confused to whether to send us or barf packets).</p>
|
||||||
ARP is link-local: ARP frames are not routed into other networks, all stay in the same broadcast domain. Thus there is no need for a hop limit (time-to-live). A reverse lookup mechanism (hardware address to protocol) is also available, named reverse ARP ;).
|
<p>Once initialisation is over, our ARP handler needs to wait for ARP requests from other hosts on the network, and if addresses to our IPv4 address, issue a reply. The other event which might happen is that a user wants to send an IPv4 packet to another host on the network. In this case, we either already have the Ethernet address in our cache, or we need to send an ARP request to the network and wait for a reply. Since packets might get lost, we actually need to retry sending ARP requests until a limit is reached. To keep the cache in a reasonable size, old entries should be dropped if unused. Also, the Ethernet address of hosts may change, due to hardware replacement or failover.</p>
|
||||||
|
<p>That's it. Pretty straightforward.</p>
|
||||||
I will focus on ARP in this article, as used widely to translate IPv4 addresses into Ethernet addresses. There are two operations in ARP: request and response. A request is usually broadcasted to all hosts (by setting the destination to the broadcast Ethernet address, `ff:ff:ff:ff:ff:ff`), while a reply is send via unicast (to the host which requested that information).
|
<h2 id="design">Design</h2>
|
||||||
|
<p>Back in 2008, together with Andreas Bogk, we just used a hash table and installed expiration and retransmission timers when needed. Certainly timers sometimes needed to be cancelled, and testing the code was cumbersome. It were only <a href="https://github.com/dylan-hackers/network-night-vision/blob/master/network/ip-stack/layers/network/arp/arp.dylan">250 lines of Dylan code</a> plus some <a href="https://github.com/dylan-hackers/network-night-vision/blob/master/protocols/ipv4.dylan">wire format definition</a>.</p>
|
||||||
The frame format is pretty straightforward: 2 bytes hardware address type, 2 bytes protocol type, 1 byte length for both types, 2 bytes operation, followed by source addresses (hardware and protocol), and target addresses. In total 28 bytes, considering 48 bit Ethernet addresses and 32 bit IPv4 addresses.
|
<p>Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated. The code should not contain surprises, but straightforward to understand. The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should. Once this is achieved, testing is straightforward. If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused. It can also be repurposed to run as a test oracle. You can read more marketing of this style in our <a href="https://usenix15.nqsb.io">Usenix security paper</a>.</p>
|
||||||
|
<p>My proposed style and hash tables are not good friends, since hash tables in OCaml are imperative structures. Instead, a <em>Map</em> (<a href="http://caml.inria.fr/pub/docs/manual-ocaml/libref/Map.html">documentation</a>) is a functional data structure for associating keys with values. Its underlying data structure is a balanced binary tree.</p>
|
||||||
It was initially specified in [RFC 826](https://tools.ietf.org/html/rfc826), but reading through [RFC 1122](https://tools.ietf.org/html/rfc1122) (requirements for Internet Hosts - Communication layer), and maybe the newer [RFC 5227](https://tools.ietf.org/html/rfc5227) (IPv4 address conflict detection) does not hurt.
|
<p>Our ARP handler certainly has some <em>state</em>, at least its IPv4 address, its Ethernet address, and the map containing entries.</p>
|
||||||
|
<p>We have to deal with the various effects mentioned earlier:</p>
|
||||||
On UNIX systems, you can investigate your arp table, also called arp cache, using the `arp` command line utility.
|
<ul>
|
||||||
|
<li><em>Network</em> we provide a function taking a state and a packet, transforming to successor state, potentially output on the network, and potentially waking up tasks which are awaiting the mac address.
|
||||||
### Protocol logic
|
</li>
|
||||||
|
<li><em>Timer</em> we need to rely on an external periodic event calling our function <code>tick</code>, which transforms a state to a successor state, a list of ARP requests to be send out (retransmission), and a list of tasks to be informed that a timeout occurred.
|
||||||
Let us look what our ARP handler actually needs to do? Translating IPv4 addresses to Ethernet addresses, but where does it learn new information?
|
</li>
|
||||||
|
<li><em>Query</em> a query for an IPv4 address using some state leads to a successor state, and either an immediate answer with the Ethernet address, or an ARP request to be sent and waiting for an answer, or just waiting for an answer in the case another task has already requested that IPv4 address. Since we don't want to convolute the protocol core with tasks, we'll let the effectful layer decide how to achieve that by abstracting over some alpha to store, and requiring a <code>merge : alpha option -> alpha</code> function.
|
||||||
First of all, our ARP handler needs to know its own IPv4 address and its Ethernet address. It will even broadcast them on startup, so-called gratuitous ARP. The purpose of this is to inform all other hosts on the same network that we are here now. And if another host, let's name it barf, has the same IPv4 address, some sort of conflict resolution needs to happen (otherwise all hosts on the network are confused to whether to send us or barf packets).
|
</li>
|
||||||
|
</ul>
|
||||||
Once initialisation is over, our ARP handler needs to wait for ARP requests from other hosts on the network, and if addresses to our IPv4 address, issue a reply. The other event which might happen is that a user wants to send an IPv4 packet to another host on the network. In this case, we either already have the Ethernet address in our cache, or we need to send an ARP request to the network and wait for a reply. Since packets might get lost, we actually need to retry sending ARP requests until a limit is reached. To keep the cache in a reasonable size, old entries should be dropped if unused. Also, the Ethernet address of hosts may change, due to hardware replacement or failover.
|
<h3 id="excursion-security">Excursion: security</h3>
|
||||||
|
<p>ARP is a link-local protocol, thus attackers have to have access to the same link-layer: either a cable in the same switch or hub, or in the same wireless network (if you're into modern technology).</p>
|
||||||
That's it. Pretty straightforward.
|
<p>A very common attack vector for protocols is the so called person in the middle attack, where the attacker sits between you and the remote host. An attacker can achieve this using ARP spoofing: if they can convince your computer that the attacker is the gateway, your computer will send all packets to the attacker, who either forwards them to the remote host, or modifies them, or drops them.</p>
|
||||||
|
<p>ARP does not employ any security mechanism, it is more a question of receiving the first answer (depending on the implementation). A common countermeasure is to manually fill the cache with the gateway statically. This only needs updates if the gateway is replaced, or gets a new network card.</p>
|
||||||
## Design
|
<p>Denial of service attacks are also possible using ARP: if the implementation preserves all replies, the cache might expand immensely. This happens sometimes in switch hardware, which have a limited cache, and once it is full, they go into hub mode. This means all frames are broadcasted on all ports. This enables an attacker to passively sniff all traffic in the local network.</p>
|
||||||
|
<p>One denial of service attack vector is due to choosing a hash table as underlying store. Its hash function should be collision-resistant, one way, and its output should be fixed length. A good choice would be a cryptographic hash function (like SHA-256), but these are too expensive and thus rarely used for hash tables. <a href="https://www.usenix.org/conference/12th-usenix-security-symposium/denial-service-algorithmic-complexity-attacks">Denial of Service via Algorithmic Complexity Attacks</a> and <a href="https://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf">Efficient Denial of Service Attacks on Web Application Platforms</a> are worth studying. If you expose your hash function to user input (and don't use a private seed), you might accidentally open your attack surface.</p>
|
||||||
Back in 2008, together with Andreas Bogk, we just used a hash table and installed expiration and retransmission timers when needed. Certainly timers sometimes needed to be cancelled, and testing the code was cumbersome. It were only [250 lines of Dylan code](https://github.com/dylan-hackers/network-night-vision/blob/master/network/ip-stack/layers/network/arp/arp.dylan) plus some [wire format definition](https://github.com/dylan-hackers/network-night-vision/blob/master/protocols/ipv4.dylan).
|
<h3 id="back-to-our-design">Back to our design</h3>
|
||||||
|
<p>To mitigate person in the middle attacks, we provide an API to add static entries, which are never overwritten by network input. While our own IPv4 addresses are advertised if a matching ARP request was received, other static entries are not advertised (neither are dynamic entries). We do only insert entries to our cache if we have an outstanding request or already an entry. To provide low latency, just before a dynamic entry would timeout, we send another request for this IPv4 address to the network.</p>
|
||||||
Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated. The code should not contain surprises, but straightforward to understand. The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should. Once this is achieved, testing is straightforward. If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused. It can also be repurposed to run as a test oracle. You can read more marketing of this style in our [Usenix security paper](https://usenix15.nqsb.io).
|
<h3 id="implementation">Implementation</h3>
|
||||||
|
<p>I have the <a href="https://github.com/hannesm/arp">source</a>, its <a href="https://hannesm.github.io/arp">documentation</a>, a test suite and a <a href="https://hannesm.github.io/arp/coverage">coverage report</a> online.</p>
|
||||||
My proposed style and hash tables are not good friends, since hash tables in OCaml are imperative structures. Instead, a *Map* ([documentation](http://caml.inria.fr/pub/docs/manual-ocaml/libref/Map.html)) is a functional data structure for associating keys with values. Its underlying data structure is a balanced binary tree.
|
<p>The implementation of the core logic still fits in less than 250 lines of code. Below 100 more lines are needed for decoding and encoding byte buffers. And another 140 lines to implement the Mirage ARP interface. Tests are available which cover the protocol logic and decoding/encoding to 100%.</p>
|
||||||
|
<p>The effectful layer is underspecified (especially regarding conflicts: what happens if there is an outstanding request for an IPv4 address and I add a static entry for this?). There is an implementation based on hash tables, which I used to benchmark a bit.</p>
|
||||||
Our ARP handler certainly has some *state*, at least its IPv4 address, its Ethernet address, and the map containing entries.
|
<p>Correctness aside, the performance should be in the same ballpark. I am mainly interested in how much input can be processed, being it invalid input, random valid input, random requests, random replies, and a mix of all that above plus some valid requests which should be answered. I ran the tests in two modes, one with accelerated time (where a minute passed in a second) to increase the pressure on the cache (named fast), one in real time. The results are in the table below (bigger numbers are better). It shows that neither approach is slower by design (of course there is still room for improvement).</p>
|
||||||
|
<pre><code>| Test | Hashtable | fast | Map | fast |
|
||||||
We have to deal with the various effects mentioned earlier:
|
|
||||||
- *Network* we provide a function taking a state and a packet, transforming to successor state, potentially output on the network, and potentially waking up tasks which are awaiting the mac address.
|
|
||||||
- *Timer* we need to rely on an external periodic event calling our function `tick`, which transforms a state to a successor state, a list of ARP requests to be send out (retransmission), and a list of tasks to be informed that a timeout occurred.
|
|
||||||
- *Query* a query for an IPv4 address using some state leads to a successor state, and either an immediate answer with the Ethernet address, or an ARP request to be sent and waiting for an answer, or just waiting for an answer in the case another task has already requested that IPv4 address. Since we don't want to convolute the protocol core with tasks, we'll let the effectful layer decide how to achieve that by abstracting over some alpha to store, and requiring a `merge : alpha option -> alpha` function.
|
|
||||||
|
|
||||||
### Excursion: security
|
|
||||||
|
|
||||||
ARP is a link-local protocol, thus attackers have to have access to the same link-layer: either a cable in the same switch or hub, or in the same wireless network (if you're into modern technology).
|
|
||||||
|
|
||||||
A very common attack vector for protocols is the so called person in the middle attack, where the attacker sits between you and the remote host. An attacker can achieve this using ARP spoofing: if they can convince your computer that the attacker is the gateway, your computer will send all packets to the attacker, who either forwards them to the remote host, or modifies them, or drops them.
|
|
||||||
|
|
||||||
ARP does not employ any security mechanism, it is more a question of receiving the first answer (depending on the implementation). A common countermeasure is to manually fill the cache with the gateway statically. This only needs updates if the gateway is replaced, or gets a new network card.
|
|
||||||
|
|
||||||
Denial of service attacks are also possible using ARP: if the implementation preserves all replies, the cache might expand immensely. This happens sometimes in switch hardware, which have a limited cache, and once it is full, they go into hub mode. This means all frames are broadcasted on all ports. This enables an attacker to passively sniff all traffic in the local network.
|
|
||||||
|
|
||||||
One denial of service attack vector is due to choosing a hash table as underlying store. Its hash function should be collision-resistant, one way, and its output should be fixed length. A good choice would be a cryptographic hash function (like SHA-256), but these are too expensive and thus rarely used for hash tables. [Denial of Service via Algorithmic Complexity Attacks](https://www.usenix.org/conference/12th-usenix-security-symposium/denial-service-algorithmic-complexity-attacks) and [Efficient Denial of Service Attacks on Web Application Platforms](https://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf) are worth studying. If you expose your hash function to user input (and don't use a private seed), you might accidentally open your attack surface.
|
|
||||||
|
|
||||||
### Back to our design
|
|
||||||
|
|
||||||
To mitigate person in the middle attacks, we provide an API to add static entries, which are never overwritten by network input. While our own IPv4 addresses are advertised if a matching ARP request was received, other static entries are not advertised (neither are dynamic entries). We do only insert entries to our cache if we have an outstanding request or already an entry. To provide low latency, just before a dynamic entry would timeout, we send another request for this IPv4 address to the network.
|
|
||||||
|
|
||||||
### Implementation
|
|
||||||
|
|
||||||
I have the [source](https://github.com/hannesm/arp), its [documentation](https://hannesm.github.io/arp), a test suite and a [coverage report](https://hannesm.github.io/arp/coverage) online.
|
|
||||||
|
|
||||||
The implementation of the core logic still fits in less than 250 lines of code. Below 100 more lines are needed for decoding and encoding byte buffers. And another 140 lines to implement the Mirage ARP interface. Tests are available which cover the protocol logic and decoding/encoding to 100%.
|
|
||||||
|
|
||||||
The effectful layer is underspecified (especially regarding conflicts: what happens if there is an outstanding request for an IPv4 address and I add a static entry for this?). There is an implementation based on hash tables, which I used to benchmark a bit.
|
|
||||||
|
|
||||||
Correctness aside, the performance should be in the same ballpark. I am mainly interested in how much input can be processed, being it invalid input, random valid input, random requests, random replies, and a mix of all that above plus some valid requests which should be answered. I ran the tests in two modes, one with accelerated time (where a minute passed in a second) to increase the pressure on the cache (named fast), one in real time. The results are in the table below (bigger numbers are better). It shows that neither approach is slower by design (of course there is still room for improvement).
|
|
||||||
|
|
||||||
```
|
|
||||||
| Test | Hashtable | fast | Map | fast |
|
|
||||||
| ------------- | --------- | ------- | ------- | ------- |
|
| ------------- | --------- | ------- | ------- | ------- |
|
||||||
| invalid | 2813076 | 2810684 | 2806899 | 2835905 |
|
| invalid | 2813076 | 2810684 | 2806899 | 2835905 |
|
||||||
| valid | 1126805 | 1320737 | 1770123 | 1785630 |
|
| valid | 1126805 | 1320737 | 1770123 | 1785630 |
|
||||||
|
|
@ -79,11 +46,9 @@ Correctness aside, the performance should be in the same ballpark. I am mainly
|
||||||
| replies | 1293293 | 1313405 | 1432225 | 1449860 |
|
| replies | 1293293 | 1313405 | 1432225 | 1449860 |
|
||||||
| mixed | 2158481 | 2191617 | 2196092 | 2213530 |
|
| mixed | 2158481 | 2191617 | 2196092 | 2213530 |
|
||||||
| queries | 42058 | 45258 | 44803 | 44379 |
|
| queries | 42058 | 45258 | 44803 | 44379 |
|
||||||
```
|
</code></pre>
|
||||||
|
<p>I ran each benchmark 3 times on a single core (used <code>cpuset -l 3</code> to pin it to one specific core) and picked the best set of results. The measure is number of packets processed over 5 seconds, using the Mirage ARP API. The full source code is in the <code>bench</code> subdirectory. As always, take benchmarks with a grain of salt: everybody will always find the right parameters for their microbenchmarks.</p>
|
||||||
I ran each benchmark 3 times on a single core (used `cpuset -l 3` to pin it to one specific core) and picked the best set of results. The measure is number of packets processed over 5 seconds, using the Mirage ARP API. The full source code is in the `bench` subdirectory. As always, take benchmarks with a grain of salt: everybody will always find the right parameters for their microbenchmarks.
|
<p>There was even a bug in the MirageOS ARP code: <a href="https://github.com/mirage/mirage-tcpip/issues/225">its definition of gratuitous ARP is wrong</a>.</p>
|
||||||
|
<p>I'm interested in feedback, either via
|
||||||
There was even a bug in the MirageOS ARP code: [its definition of gratuitous ARP is wrong](https://github.com/mirage/mirage-tcpip/issues/225).
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
I'm interested in feedback, either via
|
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
redirect: /About
|
|
||||||
213
Posts/Albatross
213
Posts/Albatross
|
|
@ -1,142 +1,83 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Deploying reproducible unikernels with albatross
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying reproducible unikernels with albatross</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying reproducible unikernels with albatross" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying reproducible unikernels with albatross</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2022-11-17 (last updated: 2023-05-16)</span><article><p>EDIT (2023-05-16): Updated with albatross release version 2.0.0.</p>
|
||||||
author: hannes
|
<h2 id="deploying-mirageos-unikernels">Deploying MirageOS unikernels</h2>
|
||||||
tags: mirageos, deployment
|
<p>More than five years ago, I posted <a href="/Posts/VMM">how to deploy MirageOS unikernels</a>. My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described <a href="/Posts/Deploy">how to deploy reproducible unikernels</a>.</p>
|
||||||
abstract: fleet management for MirageOS unikernels using a mutually authenticated TLS handshake
|
<h2 id="albatross">Albatross</h2>
|
||||||
---
|
<p>In recent months we worked hard on the underlying infrastructure: <a href="https://github.com/roburio/albatross">albatross</a>. Albatross is the orchestration system for MirageOS unikernels that use solo5 with <a href="https://github.com/Solo5/solo5/blob/master/docs/architecture.md">hvt or spt tender</a>. It deals with three tasks:</p>
|
||||||
|
<ul>
|
||||||
EDIT (2023-05-16): Updated with albatross release version 2.0.0.
|
<li>unikernel creation (destroyal, restart)
|
||||||
|
</li>
|
||||||
## Deploying MirageOS unikernels
|
<li>capturing console output
|
||||||
|
</li>
|
||||||
More than five years ago, I posted [how to deploy MirageOS unikernels](/Posts/VMM). My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described [how to deploy reproducible unikernels](/Posts/Deploy).
|
<li>collecting metrics in the host system about unikernels
|
||||||
|
</li>
|
||||||
## Albatross
|
</ul>
|
||||||
|
<p>An addition to the above is dealing with multiple tenants on the same machine: remote management of your unikernel fleet via TLS, and resource policies.</p>
|
||||||
In recent months we worked hard on the underlying infrastructure: [albatross](https://github.com/roburio/albatross). Albatross is the orchestration system for MirageOS unikernels that use solo5 with [hvt or spt tender](https://github.com/Solo5/solo5/blob/master/docs/architecture.md). It deals with three tasks:
|
<h2 id="history">History</h2>
|
||||||
- unikernel creation (destroyal, restart)
|
<p>The initial commit of albatross was in May 2017. Back then it replaced the shell scripts and manual <code>scp</code> of unikernel images to the server. Over time it evolved and adapted to new environments. Initially a solo5 unikernel would only know of a single network interface, these days there can be multiple distinguished by name. Initially there was no support for block devices. Only FreeBSD was supported in the early days. Nowadays we built daily packages for Debian, Ubuntu, FreeBSD, and have support for NixOS, and the client side is supported on macOS as well.</p>
|
||||||
- capturing console output
|
<h3 id="asn.1">ASN.1</h3>
|
||||||
- collecting metrics in the host system about unikernels
|
<p>The communication format between the albatross daemons and clients was changed multiple times. I'm glad that albatross uses ASN.1 as communication format, which makes extension with optional fields easy, and also allows "choice" (the sum type) to be not tagged (the binary is the same as no choice type), thus adding choice to an existing grammar, and preserving the old in the default (untagged) case is a decent solution.</p>
|
||||||
|
<p>So, if you care about backward and forward compatibility, as we do, since we may be in control of which albatross servers are deployed on our machine, but not what albatross versions the clients are using -- it may be wise to look into ASN.1. Recent efforts (json with schema, ...) may solve similar issues, but ASN.1 is as well very tiny in size.</p>
|
||||||
An addition to the above is dealing with multiple tenants on the same machine: remote management of your unikernel fleet via TLS, and resource policies.
|
<h2 id="what-resources-does-a-unikernel-need">What resources does a unikernel need?</h2>
|
||||||
|
<p>A unikernel is just an operating system for a single service, there can't be much it can need.</p>
|
||||||
## History
|
<h3 id="name">Name</h3>
|
||||||
|
<p>So, first of all a unikernel has a name, or a handle. This is useful for reporting statistics, but also to specify which console output you're interested in. The name is a string with printable ASCII characters (and dash '-' and dot '.'), with a length up to 64 characters - so yes, you can use an UUID if you like.</p>
|
||||||
The initial commit of albatross was in May 2017. Back then it replaced the shell scripts and manual `scp` of unikernel images to the server. Over time it evolved and adapted to new environments. Initially a solo5 unikernel would only know of a single network interface, these days there can be multiple distinguished by name. Initially there was no support for block devices. Only FreeBSD was supported in the early days. Nowadays we built daily packages for Debian, Ubuntu, FreeBSD, and have support for NixOS, and the client side is supported on macOS as well.
|
<h3 id="memory">Memory</h3>
|
||||||
|
<p>Another resource is the amount of memory assigned to the unikernel. This is specified in megabyte (as solo5 does), with the range being 10 (below not even a hello world wants to start) to 1024.</p>
|
||||||
### ASN.1
|
<h3 id="arguments">Arguments</h3>
|
||||||
The communication format between the albatross daemons and clients was changed multiple times. I'm glad that albatross uses ASN.1 as communication format, which makes extension with optional fields easy, and also allows "choice" (the sum type) to be not tagged (the binary is the same as no choice type), thus adding choice to an existing grammar, and preserving the old in the default (untagged) case is a decent solution.
|
<p>Of course, you can pass via albatross boot parameters to the unikernel. Albatross doesn't impose any restrictions here, but the lower levels may.</p>
|
||||||
|
<h3 id="cpu">CPU</h3>
|
||||||
So, if you care about backward and forward compatibility, as we do, since we may be in control of which albatross servers are deployed on our machine, but not what albatross versions the clients are using -- it may be wise to look into ASN.1. Recent efforts (json with schema, ...) may solve similar issues, but ASN.1 is as well very tiny in size.
|
<p>Due to multiple tenants, and side channel attacks, it looked right at the beginning like a good idea to restrict each unikernel to a specific CPU. This way, one tenant may use CPU 5, and another CPU 9 - and they'll not starve each other (best to make sure that these CPUs are in different packages). So, albatross takes a number as the CPU, and executes the solo5 tender within <code>taskset</code>/<code>cpuset</code>.</p>
|
||||||
|
<h3 id="fail-behaviour">Fail behaviour</h3>
|
||||||
## What resources does a unikernel need?
|
<p>In normal operations, exceptional behaviour may occur. I have to admit that I've seen MirageOS unikernels that suffer from not freeing all the memory they have allocated. To avoid having to get up at 4 AM just to start the unikernel that went out of memory, there's the possibility to restart the unikernel when it exited. You can even specify on which exit codes it should be restarted (the exit code is the only piece of information we have from the outside what caused the exit). This feature was implemented in October 2019, and has been very precious since then. :)</p>
|
||||||
|
<h3 id="network">Network</h3>
|
||||||
A unikernel is just an operating system for a single service, there can't be much it can need.
|
<p>This becomes a bit more complex: a MirageOS unikernel can have network interfaces, and solo5 specifies a so-called manifest with a list of these (name and type, and type is so far always basic). Then, on the actual server there are bridges (virtual switches) configured. Now, these may have the same name, or may need to be mapped. And of course, the unikernel expects a tap interface that is connected to such a bridge, not the bridge itself. Thus, albatross creates tap devices, attaches these to the respective bridges, and takes care about cleaning them up on teardown. The albatross client verifies that for each network interface in the manifest, there is a command-line argument specified (<code>--net service:my_bridge</code> or just <code>--net service</code> if the bridge is named service). The tap interface name is not really of interest to the user, and will not be exposed.</p>
|
||||||
|
<h3 id="block-devices">Block devices</h3>
|
||||||
### Name
|
<p>On the host system, it's just a file, and passed to the unikernel. There's the need to be able to create one, dump it, and ensure that each file is only used by one unikernel. That's all that is there.</p>
|
||||||
|
<h2 id="metrics">Metrics</h2>
|
||||||
So, first of all a unikernel has a name, or a handle. This is useful for reporting statistics, but also to specify which console output you're interested in. The name is a string with printable ASCII characters (and dash '-' and dot '.'), with a length up to 64 characters - so yes, you can use an UUID if you like.
|
<p>Everyone likes graphs, over time, showing how much traffic or CPU or memory or whatever has been used by your service. Some of these statistics are only available in the host system, and it is also crucial for development purposes to compare whether the bytes sent in the unikernel sum up to the same on the host system's tap interface.</p>
|
||||||
|
<p>The albatross-stats daemon collects metrics from three sources: network interfaces, getrusage (of a child process), VMM debug counters (to count VM exits etc.). Since the recent 1.5.3, albatross-stats now connects at startup to the albatross-daemon and then retrieves the information which unikernels are up and running, and starts periodically collecting data in memory.</p>
|
||||||
### Memory
|
<p>Other clients, being it a dump on your console window, a write into an rrd file (good old MRTG times), or a push to influx, can use the stats data to correlate and better analyse what is happening on the grand scale of things. This helped a lot by running several unikernels with different opam package sets to figure out which opam packages leave their hands on memory over time.</p>
|
||||||
|
<p>As a side note, if you make the unikernel name also available in the unikernel, it can tag its own metrics with the same identifier, and you can correlate high-level events (such as amount of HTTP requests) with low-level things "allocated more memory" or "consumed a lot of CPU".</p>
|
||||||
Another resource is the amount of memory assigned to the unikernel. This is specified in megabyte (as solo5 does), with the range being 10 (below not even a hello world wants to start) to 1024.
|
<h2 id="console">Console</h2>
|
||||||
|
<p>There's not much to say about the console, just that the albatross-console daemon is running with low privileges, and reading from a FIFO that the unikernel writes to. It never writes anything to disk, but keeps the last 1000 lines in memory, available from a client asking for it.</p>
|
||||||
### Arguments
|
<h2 id="the-daemons">The daemons</h2>
|
||||||
|
<p>So, the main albatross-daemon runs with superuser privileges to create virtual machines, and opens a unix domain socket where the clients and other daemons are connecting to. The other daemons are executed with normal user privileges, and never write anything to disk.</p>
|
||||||
Of course, you can pass via albatross boot parameters to the unikernel. Albatross doesn't impose any restrictions here, but the lower levels may.
|
<p>The albatross-daemon keeps state about the running unikernels, and if it is restarted, the unikernels are started again. Maybe worth to mention that this lead sometimes to headaches (due to data being dumped to disk, and the old format should always be supported), but was also a huge relief to not have to care about creating all the unikernels just because albatross-daemon was killed.</p>
|
||||||
|
<h2 id="remote-management">Remote management</h2>
|
||||||
### CPU
|
<p>There's one more daemon program: albatross-tls-endpoint. It accepts clients via a remote TCP connection, and establish a mutual-authenticated TLS handshake. When done, the command is forwarded to the respective Unix domain socket, and the reply is sent back to the client.</p>
|
||||||
|
<p>The daemon itself has a X.509 certificate to authenticate, but the client is requested to show its certificate chain as well. This by now requires TLS 1.3, so the client certificates are sent over the encrypted channel.</p>
|
||||||
Due to multiple tenants, and side channel attacks, it looked right at the beginning like a good idea to restrict each unikernel to a specific CPU. This way, one tenant may use CPU 5, and another CPU 9 - and they'll not starve each other (best to make sure that these CPUs are in different packages). So, albatross takes a number as the CPU, and executes the solo5 tender within `taskset`/`cpuset`.
|
<p>A step back, X.509 certificate contains a public key and a signature from one level up. When the server knows about the root (or certificate authority (CA)) certificate, and following the chain can verify that the leaf certificate is valid. Additionally, a X.509 certificate is a ASN.1 structure with some fixed fields, but also contains extensions, a key-value store where the keys are object identifiers, and the values are key-dependent data. Also note that this key-value store is cryptographically signed.</p>
|
||||||
|
<p>Albatross uses the object identifier, assigned to Camelus Dromedarius (MirageOS - 1.3.6.1.4.1.49836.42) to encode the command to be executed. This means that once the TLS handshake is established, the command to be executed is already transferred.</p>
|
||||||
### Fail behaviour
|
<p>In the leaf certificate, there may be the "create unikernel" command with the unikernel image, it's boot parameters, and other resources. Or a "read the console of my unikernel". In the intermediate certificates (from root to leaf), resource policies are encoded (this path may only have X unikernels running with a total of Y MB memory, and Z MB of block storage, using CPUs A and B, accessing bridges C and D). From the root downwards these policies may only decrease. When a unikernel should be created (or other commands are executed), the policies are verified to hold. If they do not, an error is reported.</p>
|
||||||
|
<h2 id="fleet-management">Fleet management</h2>
|
||||||
In normal operations, exceptional behaviour may occur. I have to admit that I've seen MirageOS unikernels that suffer from not freeing all the memory they have allocated. To avoid having to get up at 4 AM just to start the unikernel that went out of memory, there's the possibility to restart the unikernel when it exited. You can even specify on which exit codes it should be restarted (the exit code is the only piece of information we have from the outside what caused the exit). This feature was implemented in October 2019, and has been very precious since then. :)
|
<p>Of course it is very fine to create your locally compiled unikernel to your albatross server, go for it. But in terms of "what is actually running here?" and "does this unikernel need to be updated because some opam package had a security issues?", this is not optimal.</p>
|
||||||
|
<p>Since we provide <a href="https://builds.robur.coop">daily reproducible builds</a> with the current HEAD of the main opam-repository, and these unikernels have no configuration embedded (but take everything as boot parameters), we just deploy them. They come with the information what opam packages contributed to the binary, which environment variables were set, and which system packages were installed with which versions.</p>
|
||||||
### Network
|
<p>The whole result of reproducible builds for us means: we have a hash of a unikernel image that we can lookup in our build infrastructure, and take a look whether there is a newer image for the same job. And if there is, we provide a diff between the packages contributed to the currently running unikernel and the new image. That is what the albatross-client update command is all about.</p>
|
||||||
|
<p>Of course, your mileage may vary and you want automated deployments where each git commit triggers recompilation and redeployment. The downside would be that sometimes only dependencies are updated and you've to cope with that.</p>
|
||||||
This becomes a bit more complex: a MirageOS unikernel can have network interfaces, and solo5 specifies a so-called manifest with a list of these (name and type, and type is so far always basic). Then, on the actual server there are bridges (virtual switches) configured. Now, these may have the same name, or may need to be mapped. And of course, the unikernel expects a tap interface that is connected to such a bridge, not the bridge itself. Thus, albatross creates tap devices, attaches these to the respective bridges, and takes care about cleaning them up on teardown. The albatross client verifies that for each network interface in the manifest, there is a command-line argument specified (`--net service:my_bridge` or just `--net service` if the bridge is named service). The tap interface name is not really of interest to the user, and will not be exposed.
|
<p>There is a client <code>albatross-client</code>, depending on arguments either connects to a local Unix domain socket, or to a remote albatross instance via TCP and TLS, or outputs a certificate signing request for later usage. Data, such as the unikernel ELF image, is compressed in certificates.</p>
|
||||||
|
<h2 id="installation">Installation</h2>
|
||||||
### Block devices
|
<p>For Debian and Ubuntu systems, we provide package repositories. Browse the dists folder for one matching your distribution, and add it to <code>/etc/apt/sources.list</code>:</p>
|
||||||
|
<pre><code>$ wget -q -O /etc/apt/trusted.gpg.d/apt.robur.coop.gpg https://apt.robur.coop/gpg.pub
|
||||||
On the host system, it's just a file, and passed to the unikernel. There's the need to be able to create one, dump it, and ensure that each file is only used by one unikernel. That's all that is there.
|
$ echo "deb https://apt.robur.coop ubuntu-20.04 main" >> /etc/apt/sources.list # replace ubuntu-20.04 with e.g. debian-11 on a debian buster machine
|
||||||
|
|
||||||
## Metrics
|
|
||||||
|
|
||||||
Everyone likes graphs, over time, showing how much traffic or CPU or memory or whatever has been used by your service. Some of these statistics are only available in the host system, and it is also crucial for development purposes to compare whether the bytes sent in the unikernel sum up to the same on the host system's tap interface.
|
|
||||||
|
|
||||||
The albatross-stats daemon collects metrics from three sources: network interfaces, getrusage (of a child process), VMM debug counters (to count VM exits etc.). Since the recent 1.5.3, albatross-stats now connects at startup to the albatross-daemon and then retrieves the information which unikernels are up and running, and starts periodically collecting data in memory.
|
|
||||||
|
|
||||||
Other clients, being it a dump on your console window, a write into an rrd file (good old MRTG times), or a push to influx, can use the stats data to correlate and better analyse what is happening on the grand scale of things. This helped a lot by running several unikernels with different opam package sets to figure out which opam packages leave their hands on memory over time.
|
|
||||||
|
|
||||||
As a side note, if you make the unikernel name also available in the unikernel, it can tag its own metrics with the same identifier, and you can correlate high-level events (such as amount of HTTP requests) with low-level things "allocated more memory" or "consumed a lot of CPU".
|
|
||||||
|
|
||||||
## Console
|
|
||||||
|
|
||||||
There's not much to say about the console, just that the albatross-console daemon is running with low privileges, and reading from a FIFO that the unikernel writes to. It never writes anything to disk, but keeps the last 1000 lines in memory, available from a client asking for it.
|
|
||||||
|
|
||||||
## The daemons
|
|
||||||
|
|
||||||
So, the main albatross-daemon runs with superuser privileges to create virtual machines, and opens a unix domain socket where the clients and other daemons are connecting to. The other daemons are executed with normal user privileges, and never write anything to disk.
|
|
||||||
|
|
||||||
The albatross-daemon keeps state about the running unikernels, and if it is restarted, the unikernels are started again. Maybe worth to mention that this lead sometimes to headaches (due to data being dumped to disk, and the old format should always be supported), but was also a huge relief to not have to care about creating all the unikernels just because albatross-daemon was killed.
|
|
||||||
|
|
||||||
## Remote management
|
|
||||||
|
|
||||||
There's one more daemon program: albatross-tls-endpoint. It accepts clients via a remote TCP connection, and establish a mutual-authenticated TLS handshake. When done, the command is forwarded to the respective Unix domain socket, and the reply is sent back to the client.
|
|
||||||
|
|
||||||
The daemon itself has a X.509 certificate to authenticate, but the client is requested to show its certificate chain as well. This by now requires TLS 1.3, so the client certificates are sent over the encrypted channel.
|
|
||||||
|
|
||||||
A step back, X.509 certificate contains a public key and a signature from one level up. When the server knows about the root (or certificate authority (CA)) certificate, and following the chain can verify that the leaf certificate is valid. Additionally, a X.509 certificate is a ASN.1 structure with some fixed fields, but also contains extensions, a key-value store where the keys are object identifiers, and the values are key-dependent data. Also note that this key-value store is cryptographically signed.
|
|
||||||
|
|
||||||
Albatross uses the object identifier, assigned to Camelus Dromedarius (MirageOS - 1.3.6.1.4.1.49836.42) to encode the command to be executed. This means that once the TLS handshake is established, the command to be executed is already transferred.
|
|
||||||
|
|
||||||
In the leaf certificate, there may be the "create unikernel" command with the unikernel image, it's boot parameters, and other resources. Or a "read the console of my unikernel". In the intermediate certificates (from root to leaf), resource policies are encoded (this path may only have X unikernels running with a total of Y MB memory, and Z MB of block storage, using CPUs A and B, accessing bridges C and D). From the root downwards these policies may only decrease. When a unikernel should be created (or other commands are executed), the policies are verified to hold. If they do not, an error is reported.
|
|
||||||
|
|
||||||
## Fleet management
|
|
||||||
|
|
||||||
Of course it is very fine to create your locally compiled unikernel to your albatross server, go for it. But in terms of "what is actually running here?" and "does this unikernel need to be updated because some opam package had a security issues?", this is not optimal.
|
|
||||||
|
|
||||||
Since we provide [daily reproducible builds](https://builds.robur.coop) with the current HEAD of the main opam-repository, and these unikernels have no configuration embedded (but take everything as boot parameters), we just deploy them. They come with the information what opam packages contributed to the binary, which environment variables were set, and which system packages were installed with which versions.
|
|
||||||
|
|
||||||
The whole result of reproducible builds for us means: we have a hash of a unikernel image that we can lookup in our build infrastructure, and take a look whether there is a newer image for the same job. And if there is, we provide a diff between the packages contributed to the currently running unikernel and the new image. That is what the albatross-client update command is all about.
|
|
||||||
|
|
||||||
Of course, your mileage may vary and you want automated deployments where each git commit triggers recompilation and redeployment. The downside would be that sometimes only dependencies are updated and you've to cope with that.
|
|
||||||
|
|
||||||
There is a client `albatross-client`, depending on arguments either connects to a local Unix domain socket, or to a remote albatross instance via TCP and TLS, or outputs a certificate signing request for later usage. Data, such as the unikernel ELF image, is compressed in certificates.
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
For Debian and Ubuntu systems, we provide package repositories. Browse the dists folder for one matching your distribution, and add it to `/etc/apt/sources.list`:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ wget -q -O /etc/apt/trusted.gpg.d/apt.robur.coop.gpg https://apt.robur.coop/gpg.pub
|
|
||||||
$ echo "deb https://apt.robur.coop ubuntu-20.04 main" >> /etc/apt/sources.list # replace ubuntu-20.04 with e.g. debian-11 on a debian buster machine
|
|
||||||
$ apt update
|
$ apt update
|
||||||
$ apt install solo5 albatross
|
$ apt install solo5 albatross
|
||||||
```
|
</code></pre>
|
||||||
|
<p>On FreeBSD:</p>
|
||||||
On FreeBSD:
|
<pre><code>$ fetch -o /usr/local/etc/pkg/robur.pub https://pkg.robur.coop/repo.pub # download RSA public key
|
||||||
|
|
||||||
```
|
|
||||||
$ fetch -o /usr/local/etc/pkg/robur.pub https://pkg.robur.coop/repo.pub # download RSA public key
|
|
||||||
$ echo 'robur: {
|
$ echo 'robur: {
|
||||||
url: "https://pkg.robur.coop/${ABI}",
|
url: "https://pkg.robur.coop/${ABI}",
|
||||||
mirror_type: "srv",
|
mirror_type: "srv",
|
||||||
signature_type: "pubkey",
|
signature_type: "pubkey",
|
||||||
pubkey: "/usr/local/etc/pkg/robur.pub",
|
pubkey: "/usr/local/etc/pkg/robur.pub",
|
||||||
enabled: yes
|
enabled: yes
|
||||||
}' > /usr/local/etc/pkg/repos/robur.conf # Check https://pkg.robur.coop which ABI are available
|
}' > /usr/local/etc/pkg/repos/robur.conf # Check https://pkg.robur.coop which ABI are available
|
||||||
$ pkg update
|
$ pkg update
|
||||||
$ pkg install solo5 albatross
|
$ pkg install solo5 albatross
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Please ensure to have at least version 2.0.0 of albatross installed.</p>
|
||||||
Please ensure to have at least version 2.0.0 of albatross installed.
|
<p>For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (<code>opam install solo5 albatross</code>). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.</p>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (`opam install solo5 albatross`). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.
|
<p>After five years of development and operating albatross, feel free to get it and try it out. Or read the code, discuss issues and shortcomings with us - either at the issue tracker or via eMail.</p>
|
||||||
|
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
|
||||||
## Conclusion
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
After five years of development and operating albatross, feel free to get it and try it out. Or read the code, discuss issues and shortcomings with us - either at the issue tracker or via eMail.
|
|
||||||
|
|
||||||
Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on [donations](https://robur.coop/Donate) for doing our work - everyone can contribute.
|
|
||||||
|
|
@ -1,36 +1,17 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Catch the bug, walking through the stack
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Catch the bug, walking through the stack</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Catch the bug, walking through the stack" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a></div><span class="date">Published: 2016-05-03 (last updated: 2021-11-19)</span><article><h2 id="bad-record-mac">BAD RECORD MAC</h2>
|
||||||
author: hannes
|
<p>Roughly 2 weeks ago, <a href="https://github.com/Engil">Engil</a> informed me that a TLS alert pops up in his browser sometimes when he reads this website. His browser reported that the <a href="https://en.wikipedia.org/wiki/Message_authentication_code">message authentication code</a> was wrong. From <a href="https://tools.ietf.org/html/rfc5246">RFC 5246</a>: This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).</p>
|
||||||
tags: mirageos, security
|
<p>I tried hard, but could not reproduce, but was very worried and was eager to find the root cause (some little fear remained that it was in our TLS stack). I setup this website with some TLS-level tracing (extending the code from our <a href="https://tls.openmirage.org">TLS handshake server</a>). We tried to reproduce the issue with traces and packet captures (both on client and server side) in place from our computer labs office with no success. Later, Engil tried from his home and after 45MB of wire data, ran into this issue. Finally, evidence! Isolating the TCP flow with the alert resulted in just about 200KB of packet capture data (TLS ASCII trace around 650KB).</p>
|
||||||
abstract: 10BTC could've been yours
|
<p><img src="/static/img/encrypted-alert.png" alt="encrypted alert" /></p>
|
||||||
---
|
<p>What is happening on the wire? After some data is successfully transferred, at some point the client sends an encrypted alert (see above). The TLS session used a RSA key exchange and I could decrypt the TLS stream with Wireshark, which revealed that the alert was indeed a bad record MAC. Wireshark's "follow SSL stream" showed all client requests, but not all server responses. The TLS level trace from the server showed properly encrypted data. I tried to spot the TCP payload which caused the bad record MAC, starting from the alert in the client capture (the offending TCP frame should be closely before the alert).</p>
|
||||||
|
<p><img src="/static/img/tcp-frame-client.png" alt="client TCP frame" /></p>
|
||||||
## BAD RECORD MAC
|
<p>There is plaintext data which looks like a HTTP request in the TCP frame sent by the server to the client? WTF? This should never happen! The same TCP frame on the server side looked even more strange: it had an invalid checksum.</p>
|
||||||
|
<p><img src="/static/img/tcp-frame-server.png" alt="server TCP frame" /></p>
|
||||||
Roughly 2 weeks ago, [Engil](https://github.com/Engil) informed me that a TLS alert pops up in his browser sometimes when he reads this website. His browser reported that the [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code) was wrong. From [RFC 5246](https://tools.ietf.org/html/rfc5246): This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
|
<p>What do we have so far? We spotted some plaintext data in a TCP frame which is part of a TLS session. The TCP checksum is invalid.</p>
|
||||||
|
<p>This at least explains why we were not able to reproduce from our office: usually, TCP frames with invalid checksums are dropped by the receiving TCP stack, and the sender will retransmit TCP frames which have not been acknowledged by the recipient. However, this mechanism only works if the checksums haven't been changed by a well-meaning middleman to be correct! Our traces are from a client behind a router doing <a href="https://en.wikipedia.org/wiki/Network_address_translation">network address translation</a>, which has to recompute the <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Checksum_computation">TCP checksum</a> because it modifies destination IP address and port. It seems like this specific router does not validate the TCP checksum before recomputing it, so it replaced the invalid TCP checksum with a valid one.</p>
|
||||||
I tried hard, but could not reproduce, but was very worried and was eager to find the root cause (some little fear remained that it was in our TLS stack). I setup this website with some TLS-level tracing (extending the code from our [TLS handshake server](https://tls.openmirage.org)). We tried to reproduce the issue with traces and packet captures (both on client and server side) in place from our computer labs office with no success. Later, Engil tried from his home and after 45MB of wire data, ran into this issue. Finally, evidence! Isolating the TCP flow with the alert resulted in just about 200KB of packet capture data (TLS ASCII trace around 650KB).
|
<p>Next steps are: what did the TLS layer intended to send? Why is there a TCP frame with an invalid checksum emitted?</p>
|
||||||
|
<p>Looking into the TLS trace, the TCP payload in question should have started with the following data:</p>
|
||||||
|
<pre><code>0000 0B C9 E5 F3 C5 32 43 6F 53 68 ED 42 F8 67 DA 8B .....2Co Sh.B.g..
|
||||||
|
|
||||||
What is happening on the wire? After some data is successfully transferred, at some point the client sends an encrypted alert (see above). The TLS session used a RSA key exchange and I could decrypt the TLS stream with Wireshark, which revealed that the alert was indeed a bad record MAC. Wireshark's "follow SSL stream" showed all client requests, but not all server responses. The TLS level trace from the server showed properly encrypted data. I tried to spot the TCP payload which caused the bad record MAC, starting from the alert in the client capture (the offending TCP frame should be closely before the alert).
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
There is plaintext data which looks like a HTTP request in the TCP frame sent by the server to the client? WTF? This should never happen! The same TCP frame on the server side looked even more strange: it had an invalid checksum.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
What do we have so far? We spotted some plaintext data in a TCP frame which is part of a TLS session. The TCP checksum is invalid.
|
|
||||||
|
|
||||||
This at least explains why we were not able to reproduce from our office: usually, TCP frames with invalid checksums are dropped by the receiving TCP stack, and the sender will retransmit TCP frames which have not been acknowledged by the recipient. However, this mechanism only works if the checksums haven't been changed by a well-meaning middleman to be correct! Our traces are from a client behind a router doing [network address translation](https://en.wikipedia.org/wiki/Network_address_translation), which has to recompute the [TCP checksum](https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Checksum_computation) because it modifies destination IP address and port. It seems like this specific router does not validate the TCP checksum before recomputing it, so it replaced the invalid TCP checksum with a valid one.
|
|
||||||
|
|
||||||
Next steps are: what did the TLS layer intended to send? Why is there a TCP frame with an invalid checksum emitted?
|
|
||||||
|
|
||||||
Looking into the TLS trace, the TCP payload in question should have started with the following data:
|
|
||||||
|
|
||||||
```
|
|
||||||
0000 0B C9 E5 F3 C5 32 43 6F 53 68 ED 42 F8 67 DA 8B .....2Co Sh.B.g..
|
|
||||||
0010 17 87 AB EA 3F EC 99 D4 F3 38 88 E6 E3 07 D5 6E ....?... .8.....n
|
0010 17 87 AB EA 3F EC 99 D4 F3 38 88 E6 E3 07 D5 6E ....?... .8.....n
|
||||||
0020 94 9A 81 AF DD 76 E2 7C 6F 2A C6 98 BA 70 1A AD .....v.| o*...p..
|
0020 94 9A 81 AF DD 76 E2 7C 6F 2A C6 98 BA 70 1A AD .....v.| o*...p..
|
||||||
0030 95 5E 13 B0 F7 A3 8C 25 6B 3D 59 CE 30 EC 56 B8 .^.....% k=Y.0.V.
|
0030 95 5E 13 B0 F7 A3 8C 25 6B 3D 59 CE 30 EC 56 B8 .^.....% k=Y.0.V.
|
||||||
|
|
@ -40,40 +21,39 @@ Looking into the TLS trace, the TCP payload in question should have started with
|
||||||
0070 45 6D 7F A6 1D B7 0F 43 C4 D0 8C CF 52 77 9F 06 Em.....C ....Rw..
|
0070 45 6D 7F A6 1D B7 0F 43 C4 D0 8C CF 52 77 9F 06 Em.....C ....Rw..
|
||||||
0080 59 31 E0 9D B2 B5 34 BD A4 4B 3F 02 2E 56 B9 A9 Y1....4. .K?..V..
|
0080 59 31 E0 9D B2 B5 34 BD A4 4B 3F 02 2E 56 B9 A9 Y1....4. .K?..V..
|
||||||
0090 95 38 FD AD 4A D6 35 E4 66 86 6E 03 AF 2C C9 00 .8..J.5. f.n..,..
|
0090 95 38 FD AD 4A D6 35 E4 66 86 6E 03 AF 2C C9 00 .8..J.5. f.n..,..
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The ethernet, IP, and TCP headers are in total 54 bytes, thus we have to compare starting at 0x0036 in the screenshot above. The first 74 bytes (till 0x007F in the screenshot, 0x0049 in the text dump) are very much the same, but then they diverge (for another 700 bytes).</p>
|
||||||
The ethernet, IP, and TCP headers are in total 54 bytes, thus we have to compare starting at 0x0036 in the screenshot above. The first 74 bytes (till 0x007F in the screenshot, 0x0049 in the text dump) are very much the same, but then they diverge (for another 700 bytes).
|
<p>I manually computed the TCP checksum using the TCP/IP payload from the TLS trace, and it matches the one reported as invalid. Thus, a big relief: both the TLS and the TCP/IP stack have used the correct data. Our memory disclosure issue must be after the <a href="https://github.com/mirage/mirage-tcpip/blob/1617953b4674c9c832786c1ab3236b91d00f5c25/tcp/wire.ml#L78">TCP checksum is computed</a>. After this:</p>
|
||||||
|
<ul>
|
||||||
I manually computed the TCP checksum using the TCP/IP payload from the TLS trace, and it matches the one reported as invalid. Thus, a big relief: both the TLS and the TCP/IP stack have used the correct data. Our memory disclosure issue must be after the [TCP checksum is computed](https://github.com/mirage/mirage-tcpip/blob/1617953b4674c9c832786c1ab3236b91d00f5c25/tcp/wire.ml#L78). After this:
|
<li>the <a href="https://github.com/mirage/mirage-tcpip/blob/1617953b4674c9c832786c1ab3236b91d00f5c25/lib/ipv4.ml#L98">IP frame header is filled</a>
|
||||||
* the [IP frame header is filled](https://github.com/mirage/mirage-tcpip/blob/1617953b4674c9c832786c1ab3236b91d00f5c25/lib/ipv4.ml#L98)
|
</li>
|
||||||
* the [mac addresses are put](https://github.com/mirage/mirage-tcpip/blob/1617953b4674c9c832786c1ab3236b91d00f5c25/lib/ipv4.ml#L126) into the ethernet frame
|
<li>the <a href="https://github.com/mirage/mirage-tcpip/blob/1617953b4674c9c832786c1ab3236b91d00f5c25/lib/ipv4.ml#L126">mac addresses are put</a> into the ethernet frame
|
||||||
* the frame is then passed to [mirage-net-xen for sending](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L538) via the Xen hypervisor.
|
</li>
|
||||||
|
<li>the frame is then passed to <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L538">mirage-net-xen for sending</a> via the Xen hypervisor.
|
||||||
As mentioned [earlier](/Posts/OCaml) I'm still using mirage-net-xen release 1.4.1.
|
</li>
|
||||||
|
</ul>
|
||||||
Communication with the Xen hypervisor is done via shared memory. The memory is allocated by mirage-net-xen, which then grants access to the hypervisor using [Xen grant tables](http://wiki.xen.org/wiki/Grant_Table). The TX protocol is implemented [here in mirage-net-xen](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L84-L132), which includes allocation of a [ring buffer](https://github.com/mirage/shared-memory-ring/blob/2955bf502c79bc963a02d090481b0e8958cc0c49/lwt/lwt_ring.mli). The TX protocol also has implementations for writing requests and waiting for responses, both of which are identified using a 16bit integer. When a response has arrived from the hypervisor, the respective page is returned into the pool of [shared pages](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L134-L213), to be reused by the next packet to be transmitted.
|
<p>As mentioned <a href="/Posts/OCaml">earlier</a> I'm still using mirage-net-xen release 1.4.1.</p>
|
||||||
|
<p>Communication with the Xen hypervisor is done via shared memory. The memory is allocated by mirage-net-xen, which then grants access to the hypervisor using <a href="http://wiki.xen.org/wiki/Grant_Table">Xen grant tables</a>. The TX protocol is implemented <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L84-L132">here in mirage-net-xen</a>, which includes allocation of a <a href="https://github.com/mirage/shared-memory-ring/blob/2955bf502c79bc963a02d090481b0e8958cc0c49/lwt/lwt_ring.mli">ring buffer</a>. The TX protocol also has implementations for writing requests and waiting for responses, both of which are identified using a 16bit integer. When a response has arrived from the hypervisor, the respective page is returned into the pool of <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L134-L213">shared pages</a>, to be reused by the next packet to be transmitted.</p>
|
||||||
Instead of a whole page (4096 byte) per request/response, each page is [split into two blocks](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L194-L198) (since the most common [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit) for ethernet is 1500 bytes). The [identifier in use](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L489-L490) is the grant reference, which might be unique per page, but not per block.
|
<p>Instead of a whole page (4096 byte) per request/response, each page is <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L194-L198">split into two blocks</a> (since the most common <a href="https://en.wikipedia.org/wiki/Maximum_transmission_unit">MTU</a> for ethernet is 1500 bytes). The <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L489-L490">identifier in use</a> is the grant reference, which might be unique per page, but not per block.</p>
|
||||||
|
<p>Thus, when two blocks are requested to be sent, the first <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L398">polled response</a> will immediately <a href="https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L182-L185">release</a> both into the list of free blocks. When another packet is sent, the block still waiting to be sent in the ringbuffer can be reused. This leads to corrupt data being sent.</p>
|
||||||
Thus, when two blocks are requested to be sent, the first [polled response](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L398) will immediately [release](https://github.com/mirage/mirage-net-xen/blob/541e86f53cb8cf426aabdd7f090779fc5ea9fe93/lib/netif.ml#L182-L185) both into the list of free blocks. When another packet is sent, the block still waiting to be sent in the ringbuffer can be reused. This leads to corrupt data being sent.
|
<p>The fix was already done <a href="https://github.com/mirage/mirage-net-xen/commit/47de2edfad9c56110d98d0312c1a7e0b9dcc8fbf">back in December</a> to the master branch of mirage-net-xen, and has now been <a href="https://github.com/mirage/mirage-net-xen/pull/40/commits/ec9b1046b75cba5ae3473b2d3b223c3d1284489d">backported to the 1.4 branch</a>. In addition, a patch to <a href="https://github.com/mirage/mirage-net-xen/commit/0b1e53c0875062a50e2d5823b7da0d8e0a64dc37">avoid collisions on the receiving side</a> has been applied to both branches (and released in versions 1.4.2 resp. 1.6.1).</p>
|
||||||
|
<p>What can we learn from this? Read the interface documentation (if there is any), and make sure unique identifiers are really unique. Think about the lifecycle of pieces of memory. Investigation of high level bugs pays off, you might find some subtle error on a different layer. There is no perfect security, and code only gets better if more people read and understand it.</p>
|
||||||
The fix was already done [back in December](https://github.com/mirage/mirage-net-xen/commit/47de2edfad9c56110d98d0312c1a7e0b9dcc8fbf) to the master branch of mirage-net-xen, and has now been [backported to the 1.4 branch](https://github.com/mirage/mirage-net-xen/pull/40/commits/ec9b1046b75cba5ae3473b2d3b223c3d1284489d). In addition, a patch to [avoid collisions on the receiving side](https://github.com/mirage/mirage-net-xen/commit/0b1e53c0875062a50e2d5823b7da0d8e0a64dc37) has been applied to both branches (and released in versions 1.4.2 resp. 1.6.1).
|
<p>The issue was in mirage-net-xen since its initial release, but only occured under load, and thanks to reliable protocols, was silently discarded (an invalid TCP checksum leads to a dropped frame and retransmission of its payload).</p>
|
||||||
|
<p>We have seen plain data in a TLS encrypted stream. The plain data was intended to be sent to the dom0 for logging access to the webserver. The <a href="https://github.com/mirleft/btc-pinata/blob/master/logger.ml">same code</a> is used in our <a href="http://ownme.ipredator.se">Piñata</a>, thus it could have been yours (although I tried hard and couldn't get the Piñata to leak data).</p>
|
||||||
What can we learn from this? Read the interface documentation (if there is any), and make sure unique identifiers are really unique. Think about the lifecycle of pieces of memory. Investigation of high level bugs pays off, you might find some subtle error on a different layer. There is no perfect security, and code only gets better if more people read and understand it.
|
<p>Certainly, interfacing the outside world is complex. The <a href="https://github.com/mirage/mirage-block-xen">mirage-block-xen</a> library uses a similar protocol to access block devices. From a brief look, that library seems to be safe (using 64bit identifiers).</p>
|
||||||
|
<p>I'm interested in feedback, either via
|
||||||
The issue was in mirage-net-xen since its initial release, but only occured under load, and thanks to reliable protocols, was silently discarded (an invalid TCP checksum leads to a dropped frame and retransmission of its payload).
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
|
<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
|
||||||
We have seen plain data in a TLS encrypted stream. The plain data was intended to be sent to the dom0 for logging access to the webserver. The [same code](https://github.com/mirleft/btc-pinata/blob/master/logger.ml) is used in our [Piñata](http://ownme.ipredator.se), thus it could have been yours (although I tried hard and couldn't get the Piñata to leak data).
|
<ul>
|
||||||
|
<li>Canopy uses a <a href="https://github.com/Engil/Canopy/issues/30#issuecomment-215010365">map instead of a hashtable</a>, <a href="https://hannes.nqsb.io/tags">tags</a> now contains a list of tags (<a href="https://github.com/Engil/Canopy/pull/39">PR here</a>), both thanks to voila! I also use the <a href="https://github.com/Engil/Canopy/pull/38">new CSS</a> from Engil
|
||||||
Certainly, interfacing the outside world is complex. The [mirage-block-xen](https://github.com/mirage/mirage-block-xen) library uses a similar protocol to access block devices. From a brief look, that library seems to be safe (using 64bit identifiers).
|
</li>
|
||||||
|
<li>There is a <a href="http://www.openwall.com/lists/oss-security/2016/04/29/1">CVE for OCaml <=4.03</a>
|
||||||
I'm interested in feedback, either via
|
</li>
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
<li><a href="https://github.com/mirage/mirage/pull/534">Mirage 2.9.0</a> was released, which integrates support of the logs library (now already used in <a href="https://github.com/mirage/mirage-net-xen/pull/43">mirage-net-xen</a> and <a href="https://github.com/mirage/mirage-tcpip/pull/199">mirage-tcpip</a>)
|
||||||
|
</li>
|
||||||
## Other updates in the MirageOS ecosystem
|
<li>This blog post has an accompanied <a href="https://mirage.io/blog/MSA00">MirageOS security advisory</a>
|
||||||
|
</li>
|
||||||
- Canopy uses a [map instead of a hashtable](https://github.com/Engil/Canopy/issues/30#issuecomment-215010365), [tags](https://hannes.nqsb.io/tags) now contains a list of tags ([PR here](https://github.com/Engil/Canopy/pull/39)), both thanks to voila! I also use the [new CSS](https://github.com/Engil/Canopy/pull/38) from Engil
|
<li>cfcs documented some <a href="https://github.com/cfcs/mirage-examples">basic unikernels</a>
|
||||||
- There is a [CVE for OCaml <=4.03](http://www.openwall.com/lists/oss-security/2016/04/29/1)
|
</li>
|
||||||
- [Mirage 2.9.0](https://github.com/mirage/mirage/pull/534) was released, which integrates support of the logs library (now already used in [mirage-net-xen](https://github.com/mirage/mirage-net-xen/pull/43) and [mirage-tcpip](https://github.com/mirage/mirage-tcpip/pull/199))
|
</ul>
|
||||||
- This blog post has an accompanied [MirageOS security advisory](https://mirage.io/blog/MSA00)
|
</article></div></div></main></body></html>
|
||||||
- cfcs documented some [basic unikernels](https://github.com/cfcs/mirage-examples)
|
|
||||||
145
Posts/BottomUp
145
Posts/BottomUp
|
|
@ -1,108 +1,61 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Counting Bytes
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Counting Bytes</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Counting Bytes" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Counting Bytes</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-06-11 (last updated: 2021-11-19)</span><article><p>I was busy writing code, text, talks, and also spend a week without Internet, where I ground and brewed 15kg espresso.</p>
|
||||||
author: hannes
|
<h2 id="size-of-a-mirageos-unikernel">Size of a MirageOS unikernel</h2>
|
||||||
tags: mirageos, background
|
<p>There have been lots of claims and myths around the concrete size of MirageOS unikernels. In this article I'll apply some measurements which overapproximate the binary sizes. The tools used for the visualisations are available online, and soon hopefully upstreamed into the mirage tool. This article uses mirage-2.9.0 (which might be outdated at the time of reading).</p>
|
||||||
abstract: looking into dependencies and their sizes
|
<p>Let us start with a very minimal unikernel, consisting of a <code>unikernel.ml</code>:</p>
|
||||||
---
|
<pre><code class="language-OCaml">module Main (C: V1_LWT.CONSOLE) = struct
|
||||||
|
let start c = C.log_s c "hello world"
|
||||||
I was busy writing code, text, talks, and also spend a week without Internet, where I ground and brewed 15kg espresso.
|
|
||||||
|
|
||||||
## Size of a MirageOS unikernel
|
|
||||||
|
|
||||||
There have been lots of claims and myths around the concrete size of MirageOS unikernels. In this article I'll apply some measurements which overapproximate the binary sizes. The tools used for the visualisations are available online, and soon hopefully upstreamed into the mirage tool. This article uses mirage-2.9.0 (which might be outdated at the time of reading).
|
|
||||||
|
|
||||||
Let us start with a very minimal unikernel, consisting of a `unikernel.ml`:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
module Main (C: V1_LWT.CONSOLE) = struct
|
|
||||||
let start c = C.log_s c "hello world"
|
|
||||||
end
|
end
|
||||||
```
|
</code></pre>
|
||||||
|
<p>and the following <code>config.ml</code>:</p>
|
||||||
and the following `config.ml`:
|
<pre><code class="language-OCaml">open Mirage
|
||||||
|
|
||||||
```OCaml
|
|
||||||
open Mirage
|
|
||||||
|
|
||||||
let () =
|
let () =
|
||||||
register "console" [
|
register "console" [
|
||||||
foreign "Unikernel.Main" (console @-> job) $ default_console
|
foreign "Unikernel.Main" (console @-> job) $ default_console
|
||||||
]
|
]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>If we <code>mirage configure --unix</code> and <code>mirage build</code>, we end up (at least on a 64bit FreeBSD-11 system with OCaml 4.02.3) with a 2.8MB <code>main.native</code>, dynamically linked against <code>libthr</code>, <code>libm</code> and <code>libc</code> (<code>ldd</code> ftw), or a 4.5MB Xen virtual image (built on a 64bit Linux computer).</p>
|
||||||
If we `mirage configure --unix` and `mirage build`, we end up (at least on a 64bit FreeBSD-11 system with OCaml 4.02.3) with a 2.8MB `main.native`, dynamically linked against `libthr`, `libm` and `libc` (`ldd` ftw), or a 4.5MB Xen virtual image (built on a 64bit Linux computer).
|
<p>In the <code>_build</code> directory, we can find some object files and their byte sizes:</p>
|
||||||
|
<pre><code class="language-bash"> 7144 key_gen.o
|
||||||
In the `_build` directory, we can find some object files and their byte sizes:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
7144 key_gen.o
|
|
||||||
14568 main.o
|
14568 main.o
|
||||||
3552 unikernel.o
|
3552 unikernel.o
|
||||||
```
|
</code></pre>
|
||||||
|
<p>These do not sum up to 2.8MB ;)</p>
|
||||||
These do not sum up to 2.8MB ;)
|
<p>We did not specify any dependencies ourselves, thus all bits have been injected automatically by the <code>mirage</code> tool. Let us dig a bit deeper what we actually used. <code>mirage configure</code> generates a <code>Makefile</code> which includes the dependent OCaml libraries, and the packages which are used:</p>
|
||||||
|
<pre><code class="language-Makefile">LIBS = -pkgs functoria.runtime, mirage-clock-unix, mirage-console.unix, mirage-logs, mirage-types.lwt, mirage-unix, mirage.runtime
|
||||||
We did not specify any dependencies ourselves, thus all bits have been injected automatically by the `mirage` tool. Let us dig a bit deeper what we actually used. `mirage configure` generates a `Makefile` which includes the dependent OCaml libraries, and the packages which are used:
|
|
||||||
|
|
||||||
```Makefile
|
|
||||||
LIBS = -pkgs functoria.runtime, mirage-clock-unix, mirage-console.unix, mirage-logs, mirage-types.lwt, mirage-unix, mirage.runtime
|
|
||||||
PKGS = functoria lwt mirage-clock-unix mirage-console mirage-logs mirage-types mirage-types-lwt mirage-unix
|
PKGS = functoria lwt mirage-clock-unix mirage-console mirage-logs mirage-types mirage-types-lwt mirage-unix
|
||||||
```
|
</code></pre>
|
||||||
|
<p>I explained bits of our configuration DSL <a href="/Posts/Functoria">Functoria</a> earlier. The <a href="https://github.com/mirage/mirage-clock">mirage-clock</a> device is automatically injected by mirage, providing an implementation of the <code>CLOCK</code> device. We use a <a href="https://github.com/mirage/mirage-console">mirage-console</a> device, where we print the <code>hello world</code>. Since <code>mirage-2.9.0</code> the logging library (and its reporter, <a href="https://github.com/mirage/mirage-logs">mirage-logs</a>) is automatically injected as well, which actually uses the clock. Also, the <a href="https://github.com/mirage/mirage/tree/master/types">mirage type signatures</a> are required. The <a href="https://github.com/mirage/mirage-platform/tree/master/unix">mirage-unix</a> contains a <code>sleep</code>, a <code>main</code>, and provides the argument vector <code>argv</code> (all symbols in the <code>OS</code> module).</p>
|
||||||
I explained bits of our configuration DSL [Functoria](/Posts/Functoria) earlier. The [mirage-clock](https://github.com/mirage/mirage-clock) device is automatically injected by mirage, providing an implementation of the `CLOCK` device. We use a [mirage-console](https://github.com/mirage/mirage-console) device, where we print the `hello world`. Since `mirage-2.9.0` the logging library (and its reporter, [mirage-logs](https://github.com/mirage/mirage-logs)) is automatically injected as well, which actually uses the clock. Also, the [mirage type signatures](https://github.com/mirage/mirage/tree/master/types) are required. The [mirage-unix](https://github.com/mirage/mirage-platform/tree/master/unix) contains a `sleep`, a `main`, and provides the argument vector `argv` (all symbols in the `OS` module).
|
<p>Looking into the archive files of those libraries, we end up with ~92KB (NB <code>mirage-types</code> only contains types, and thus no runtime data):</p>
|
||||||
|
<pre><code class="language-bash">15268 functoria/functoria-runtime.a
|
||||||
Looking into the archive files of those libraries, we end up with ~92KB (NB `mirage-types` only contains types, and thus no runtime data):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
15268 functoria/functoria-runtime.a
|
|
||||||
3194 mirage-clock-unix/mirage-clock.a
|
3194 mirage-clock-unix/mirage-clock.a
|
||||||
12514 mirage-console/mirage_console_unix.a
|
12514 mirage-console/mirage_console_unix.a
|
||||||
24532 mirage-logs/mirage_logs.a
|
24532 mirage-logs/mirage_logs.a
|
||||||
14244 mirage-unix/OS.a
|
14244 mirage-unix/OS.a
|
||||||
21964 mirage/mirage-runtime.a
|
21964 mirage/mirage-runtime.a
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This still does not sum up to 2.8MB since we're missing the transitive dependencies.</p>
|
||||||
This still does not sum up to 2.8MB since we're missing the transitive dependencies.
|
<h3 id="visualising-recursive-dependencies">Visualising recursive dependencies</h3>
|
||||||
|
<p>Let's use a different approach: first recursively find all dependencies. We do this by using <code>ocamlfind</code> to read <code>META</code> files which contain a list of dependent libraries in their <code>requires</code> line. As input we use <code>LIBS</code> from the Makefile snippet above. The code (OCaml script) is <a href="https://gist.github.com/hannesm/bcbe54c5759ed5854f05c8f8eaee4c79">available here</a>. The colour scheme is red for pieces of the OCaml distribution, yellow for input packages, and orange for the dependencies.</p>
|
||||||
### Visualising recursive dependencies
|
<p><a href="/static/img/mirage-console.svg"><img src="/static/img/mirage-console.svg" title="UNIX dependencies of hello world" width="700" /></a></p>
|
||||||
|
<p>This is the UNIX version only, the Xen version looks similar (but worth mentioning).</p>
|
||||||
Let's use a different approach: first recursively find all dependencies. We do this by using `ocamlfind` to read `META` files which contain a list of dependent libraries in their `requires` line. As input we use `LIBS` from the Makefile snippet above. The code (OCaml script) is [available here](https://gist.github.com/hannesm/bcbe54c5759ed5854f05c8f8eaee4c79). The colour scheme is red for pieces of the OCaml distribution, yellow for input packages, and orange for the dependencies.
|
<p><a href="/static/img/mirage-console-xen.svg"><img src="/static/img/mirage-console-xen.svg" title="Xen dependencies of hello world" width="700" /></a></p>
|
||||||
|
<p>You can spot at the right that <code>mirage-bootvar</code> uses <code>re</code>, which provoked me to <a href="https://github.com/mirage/mirage-bootvar-xen/pull/19">open a PR</a>, but Jon Ludlam <a href="https://github.com/mirage/mirage-bootvar-xen/pull/18">already had a nicer PR</a> which is now merged (and a <a href="https://github.com/mirage/mirage-bootvar-xen/pull/20">new release is in preparation</a>).</p>
|
||||||
[<img src="/static/img/mirage-console.svg" title="UNIX dependencies of hello world" width="700" />](/static/img/mirage-console.svg)
|
<h3 id="counting-bytes">Counting bytes</h3>
|
||||||
|
<p>While a dependency graphs gives a big picture of what the composed libraries of a MirageOS unikernel, we also want to know how many bytes they contribute to the unikernel. The dependency graph only contains the OCaml-level dependencies, but MirageOS has in addition to that a <code>pkg-config</code> universe of the libraries written in C (such as mini-os, openlibm, ...).</p>
|
||||||
This is the UNIX version only, the Xen version looks similar (but worth mentioning).
|
<p>We overapproximate the sizes here by assuming that a linker simply concatenates all required object files. This is not true, since the sum of all objects is empirically factor two of the actual size of the unikernel.</p>
|
||||||
|
<p>I developed a pie chart visualisation, but a friend of mine reminded me that such a chart is pretty useless for comparing slices for the human brain. I spent some more time to develop a treemap visualisation to satisfy the brain. The implemented algorithm is based on <a href="http://www.win.tue.nl/~vanwijk/stm.pdf">squarified treemaps</a>, but does not use implicit mutable state. In addition, the <a href="https://gist.github.com/hannesm/c8a9b2e75bb4f98b5100a838ea125f3b">provided script</a> parses common linker flags (<code>-o -L -l</code>) and collects arguments to be linked in. It can be passed to <code>ocamlopt</code> as the C linker, more instructions at the end of <code>treemap.ml</code> (which should be cleaned up and integrated into the mirage tool, as mentioned earlier).</p>
|
||||||
[<img src="/static/img/mirage-console-xen.svg" title="Xen dependencies of hello world" width="700" />](/static/img/mirage-console-xen.svg)
|
<p><a href="/static/img/mirage-console-bytes.svg"><img src="/static/img/mirage-console-bytes.svg" title="byte sizes of hello-world (UNIX)" width="700" /></a></p>
|
||||||
|
<p><a href="/static/img/mirage-console-xen-bytes-full.svg"><img src="/static/img/mirage-console-xen-bytes-full.svg" title="byte sizes of hello-world (Xen)" width="700" /></a></p>
|
||||||
You can spot at the right that `mirage-bootvar` uses `re`, which provoked me to [open a PR](https://github.com/mirage/mirage-bootvar-xen/pull/19), but Jon Ludlam [already had a nicer PR](https://github.com/mirage/mirage-bootvar-xen/pull/18) which is now merged (and a [new release is in preparation](https://github.com/mirage/mirage-bootvar-xen/pull/20)).
|
<p>As mentioned above, this is an overapproximation. The <code>libgcc.a</code> is only needed on Xen (see <a href="https://github.com/mirage/mirage/commit/c17f2f60a6309322ba45cecb00a808f62f05cf82#commitcomment-17573123">this comment</a>), I have not yet tracked down why there is a <code>libasmrun.a</code> and a <code>libxenasmrun.a</code>.</p>
|
||||||
|
<h3 id="more-complex-examples">More complex examples</h3>
|
||||||
### Counting bytes
|
<p>Besides the hello world, I used the same tools on our <a href="http://ownme.ipredator.se">BTC Piñata</a>.</p>
|
||||||
|
<p><a href="/static/img/pinata-deps.svg"><img src="/static/img/pinata-deps.svg" title="Piñata dependencies" width="700" /></a></p>
|
||||||
While a dependency graphs gives a big picture of what the composed libraries of a MirageOS unikernel, we also want to know how many bytes they contribute to the unikernel. The dependency graph only contains the OCaml-level dependencies, but MirageOS has in addition to that a `pkg-config` universe of the libraries written in C (such as mini-os, openlibm, ...).
|
<p><a href="/static/img/pinata-bytes.svg"><img src="/static/img/pinata-bytes.svg" title="Piñata byte sizes" width="700" /></a></p>
|
||||||
|
<h3 id="conclusion">Conclusion</h3>
|
||||||
We overapproximate the sizes here by assuming that a linker simply concatenates all required object files. This is not true, since the sum of all objects is empirically factor two of the actual size of the unikernel.
|
<p>OCaml does not yet do dead code elimination, but there <a href="https://github.com/ocaml/ocaml/pull/608">is a PR</a> based on the flambda middle-end which does so. I haven't yet investigated numbers using that branch.</p>
|
||||||
|
<p>Those counting statistics could go into more detail (e.g. using <code>nm</code> to count the sizes of concrete symbols - which opens the possibility to see which symbols are present in the objects, but not in the final binary). Also, collecting the numbers for each module in a library would be great to have. In the end, it would be great to easily spot the source fragments which are responsible for a huge binary size (and getting rid of them).</p>
|
||||||
I developed a pie chart visualisation, but a friend of mine reminded me that such a chart is pretty useless for comparing slices for the human brain. I spent some more time to develop a treemap visualisation to satisfy the brain. The implemented algorithm is based on [squarified treemaps](http://www.win.tue.nl/~vanwijk/stm.pdf), but does not use implicit mutable state. In addition, the [provided script](https://gist.github.com/hannesm/c8a9b2e75bb4f98b5100a838ea125f3b) parses common linker flags (`-o -L -l`) and collects arguments to be linked in. It can be passed to `ocamlopt` as the C linker, more instructions at the end of `treemap.ml` (which should be cleaned up and integrated into the mirage tool, as mentioned earlier).
|
<p>I'm interested in feedback, either via
|
||||||
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
[<img src="/static/img/mirage-console-bytes.svg" title="byte sizes of hello-world (UNIX)" width="700" />](/static/img/mirage-console-bytes.svg)
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
[<img src="/static/img/mirage-console-xen-bytes-full.svg" title="byte sizes of hello-world (Xen)" width="700" />](/static/img/mirage-console-xen-bytes-full.svg)
|
|
||||||
|
|
||||||
As mentioned above, this is an overapproximation. The `libgcc.a` is only needed on Xen (see [this comment](https://github.com/mirage/mirage/commit/c17f2f60a6309322ba45cecb00a808f62f05cf82#commitcomment-17573123)), I have not yet tracked down why there is a `libasmrun.a` and a `libxenasmrun.a`.
|
|
||||||
|
|
||||||
### More complex examples
|
|
||||||
|
|
||||||
Besides the hello world, I used the same tools on our [BTC Piñata](http://ownme.ipredator.se).
|
|
||||||
|
|
||||||
[<img src="/static/img/pinata-deps.svg" title="Piñata dependencies" width="700" />](/static/img/pinata-deps.svg)
|
|
||||||
|
|
||||||
[<img src="/static/img/pinata-bytes.svg" title="Piñata byte sizes" width="700" />](/static/img/pinata-bytes.svg)
|
|
||||||
|
|
||||||
### Conclusion
|
|
||||||
|
|
||||||
OCaml does not yet do dead code elimination, but there [is a PR](https://github.com/ocaml/ocaml/pull/608) based on the flambda middle-end which does so. I haven't yet investigated numbers using that branch.
|
|
||||||
|
|
||||||
Those counting statistics could go into more detail (e.g. using `nm` to count the sizes of concrete symbols - which opens the possibility to see which symbols are present in the objects, but not in the final binary). Also, collecting the numbers for each module in a library would be great to have. In the end, it would be great to easily spot the source fragments which are responsible for a huge binary size (and getting rid of them).
|
|
||||||
|
|
||||||
I'm interested in feedback, either via
|
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
|
||||||
471
Posts/Conex
471
Posts/Conex
|
|
@ -1,34 +1,23 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Conex, establish trust in community repositories
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Conex, establish trust in community repositories</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Conex, establish trust in community repositories" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/package signing" class="tag">package signing</a><a href="/tags/security" class="tag">security</a><a href="/tags/overview" class="tag">overview</a></div><span class="date">Published: 2017-02-16 (last updated: 2023-11-20)</span><article><p>Less than two years after the initial proposal, we're happy to present conex
|
||||||
author: hannes
|
|
||||||
tags: package signing, security, overview
|
|
||||||
abstract: Conex is a library to verify and attest package release integrity and authenticity through the use of cryptographic signatures.
|
|
||||||
---
|
|
||||||
|
|
||||||
Less than two years after the initial proposal, we're happy to present conex
|
|
||||||
0.9.2. Pleas note that this is still work in progress, to be deployed with opam
|
0.9.2. Pleas note that this is still work in progress, to be deployed with opam
|
||||||
2.0 and the [opam repository](https://github.com/ocaml/opam-repository).
|
2.0 and the <a href="https://github.com/ocaml/opam-repository">opam repository</a>.</p>
|
||||||
|
<p><img src="/static/img/conex.png" alt="screenshot" /></p>
|
||||||
|
<p><a href="https://github.com/hannesm/conex">Conex</a> is a library to verify and attest release integrity and
|
||||||
|
authenticity of a community repository through the use of cryptographic signatures.</p>
|
||||||
[Conex](https://github.com/hannesm/conex) is a library to verify and attest release integrity and
|
<p>Packages are collected in a community repository to provide an index and
|
||||||
authenticity of a community repository through the use of cryptographic signatures.
|
|
||||||
|
|
||||||
Packages are collected in a community repository to provide an index and
|
|
||||||
allowing cross-references. Authors submit their packages to the repository. which
|
allowing cross-references. Authors submit their packages to the repository. which
|
||||||
is curated by a team of janitors. Information
|
is curated by a team of janitors. Information
|
||||||
about a package stored in a repository includes: license, author, releases,
|
about a package stored in a repository includes: license, author, releases,
|
||||||
their dependencies, build instructions, url, tarball checksum. When someone
|
their dependencies, build instructions, url, tarball checksum. When someone
|
||||||
publishes a new package, the janitors integrate it into the repository, if it
|
publishes a new package, the janitors integrate it into the repository, if it
|
||||||
compiles and passes some validity checks. For example, its name must not be misleading,
|
compiles and passes some validity checks. For example, its name must not be misleading,
|
||||||
nor may it be too general.
|
nor may it be too general.</p>
|
||||||
|
<p>Janitors keep an eye on the repository and fix emergent failures. A new
|
||||||
Janitors keep an eye on the repository and fix emergent failures. A new
|
|
||||||
compiler release, or a release of a package on which other packages depend, might break the compilation of
|
compiler release, or a release of a package on which other packages depend, might break the compilation of
|
||||||
a package. Janitors usually fix these problems by adding a patch to the build script, or introducing
|
a package. Janitors usually fix these problems by adding a patch to the build script, or introducing
|
||||||
a version constraint in the repository.
|
a version constraint in the repository.</p>
|
||||||
|
<p><em>Conex</em> ensures that every release of each package has been approved by its author or a quorum of janitors.
|
||||||
*Conex* ensures that every release of each package has been approved by its author or a quorum of janitors.
|
|
||||||
A conex-aware client initially verifies the repository using janitor key fingerprints as anchor.
|
A conex-aware client initially verifies the repository using janitor key fingerprints as anchor.
|
||||||
Afterwards, the on-disk repository is trusted, and every update is verified (as a patch) individually.
|
Afterwards, the on-disk repository is trusted, and every update is verified (as a patch) individually.
|
||||||
This incremental verification is accomplished by ensuring all resources
|
This incremental verification is accomplished by ensuring all resources
|
||||||
|
|
@ -37,240 +26,204 @@ sufficient approvals. Additionally, monotonicity is preserved by
|
||||||
embedding counters in each resource, and enforcing a counter
|
embedding counters in each resource, and enforcing a counter
|
||||||
increment after modification.
|
increment after modification.
|
||||||
This mechanism avoids rollback attacks, when an
|
This mechanism avoids rollback attacks, when an
|
||||||
attacker presents you an old version of the repository.
|
attacker presents you an old version of the repository.</p>
|
||||||
|
<p>A timestamping service (NYI) will periodically approve a global view of the
|
||||||
A timestamping service (NYI) will periodically approve a global view of the
|
|
||||||
verified repository, together with a timestamp. This is then used by the client
|
verified repository, together with a timestamp. This is then used by the client
|
||||||
to prevent mix-and-match attacks, where an attacker mixes some old packages and
|
to prevent mix-and-match attacks, where an attacker mixes some old packages and
|
||||||
some new ones. Also, the client is able to detect freeze attacks, since at
|
some new ones. Also, the client is able to detect freeze attacks, since at
|
||||||
least every day there should be a new signature done by the timestamping service.
|
least every day there should be a new signature done by the timestamping service.</p>
|
||||||
|
<p>The trust is rooted in digital signatures by package authors. The server which
|
||||||
The trust is rooted in digital signatures by package authors. The server which
|
|
||||||
hosts the repository does not need to be trusted. Neither does the host serving
|
hosts the repository does not need to be trusted. Neither does the host serving
|
||||||
release tarballs.
|
release tarballs.</p>
|
||||||
|
<p>If a single janitor would be powerful enough to approve a key for any author,
|
||||||
If a single janitor would be powerful enough to approve a key for any author,
|
|
||||||
compromising one janitor would be sufficient to enroll any new identities,
|
compromising one janitor would be sufficient to enroll any new identities,
|
||||||
modify dependencies, build scripts, etc. In conex, a quorum of janitors (let's
|
modify dependencies, build scripts, etc. In conex, a quorum of janitors (let's
|
||||||
say 3) have to approve such changes. This is different from current workflows,
|
say 3) have to approve such changes. This is different from current workflows,
|
||||||
where a single janitor with access to the repository can merge fixes.
|
where a single janitor with access to the repository can merge fixes.</p>
|
||||||
|
<p>Conex adds metadata, in form of resources, to the repository to ensure integrity and
|
||||||
Conex adds metadata, in form of resources, to the repository to ensure integrity and
|
authenticity. There are different kinds of resources:</p>
|
||||||
authenticity. There are different kinds of resources:
|
<ul>
|
||||||
|
<li><em>Authors</em>, consisting of a unique identifier, public key(s), accounts.
|
||||||
- *Authors*, consisting of a unique identifier, public key(s), accounts.
|
</li>
|
||||||
- *Teams*, sharing the same namespace as authors, containing a set of members.
|
<li><em>Teams</em>, sharing the same namespace as authors, containing a set of members.
|
||||||
- *Authorisation*, one for each package, describing which identities are authorised for the package.
|
</li>
|
||||||
- *Package index*, for each package, listing all releases.
|
<li><em>Authorisation</em>, one for each package, describing which identities are authorised for the package.
|
||||||
- *Release*, for each release, listing checksums of all data files.
|
</li>
|
||||||
|
<li><em>Package index</em>, for each package, listing all releases.
|
||||||
Modifications to identities and authorisations need to be approved by a quorum
|
</li>
|
||||||
|
<li><em>Release</em>, for each release, listing checksums of all data files.
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
<p>Modifications to identities and authorisations need to be approved by a quorum
|
||||||
of janitors, package index and release files can be modified either by an authorised
|
of janitors, package index and release files can be modified either by an authorised
|
||||||
id or by a quorum of janitors.
|
id or by a quorum of janitors.</p>
|
||||||
|
<h2 id="documentation">Documentation</h2>
|
||||||
## Documentation
|
<p><a href="https://hannesm.github.io/conex/doc/">API documentation</a> is
|
||||||
|
available online, also a <a href="https://hannesm.github.io/conex/coverage/">coverage
|
||||||
[API documentation](https://hannesm.github.io/conex/doc/) is
|
report</a>.</p>
|
||||||
available online, also a [coverage
|
<p>We presented an <a href="https://github.com/hannesm/conex-paper/raw/master/paper.pdf">abstract at OCaml
|
||||||
report](https://hannesm.github.io/conex/coverage/).
|
2016</a> about an
|
||||||
|
earlier design.</p>
|
||||||
We presented an [abstract at OCaml
|
<p>Another article on an <a href="http://opam.ocaml.org/blog/Signing-the-opam-repository/">earlier design (from
|
||||||
2016](https://github.com/hannesm/conex-paper/raw/master/paper.pdf) about an
|
2015)</a> is also
|
||||||
earlier design.
|
available.</p>
|
||||||
|
<p>Conex is inspired by <a href="https://theupdateframework.github.io/">the update
|
||||||
Another article on an [earlier design (from
|
framework</a>, especially on their <a href="https://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf">CCS 2010
|
||||||
2015)](http://opam.ocaml.org/blog/Signing-the-opam-repository/) is also
|
paper</a>, and
|
||||||
available.
|
adapted to the opam repository.</p>
|
||||||
|
<p>The <a href="https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt">TUF
|
||||||
Conex is inspired by [the update
|
spec</a>
|
||||||
framework](https://theupdateframework.github.io/), especially on their [CCS 2010
|
has a good overview of attacks and threat model, both of which are shared by conex.</p>
|
||||||
paper](https://isis.poly.edu/~jcappos/papers/samuel_tuf_ccs_2010.pdf), and
|
<h2 id="whats-missing">What's missing</h2>
|
||||||
adapted to the opam repository.
|
<ul>
|
||||||
|
<li>See <a href="https://github.com/hannesm/conex/issues/7">issue 7</a> for a laundry list
|
||||||
The [TUF
|
</li>
|
||||||
spec](https://github.com/theupdateframework/tuf/blob/develop/docs/tuf-spec.txt)
|
<li>Timestamping service
|
||||||
has a good overview of attacks and threat model, both of which are shared by conex.
|
</li>
|
||||||
|
<li>Key revocation and rollover
|
||||||
## What's missing
|
</li>
|
||||||
|
<li>Tool to approve a PR (for janitors)
|
||||||
- See [issue 7](https://github.com/hannesm/conex/issues/7) for a laundry list
|
</li>
|
||||||
- Timestamping service
|
<li>Camelus like opam-repository check bot
|
||||||
- Key revocation and rollover
|
</li>
|
||||||
- Tool to approve a PR (for janitors)
|
<li>Integration into release management systems
|
||||||
- Camelus like opam-repository check bot
|
</li>
|
||||||
- Integration into release management systems
|
</ul>
|
||||||
|
<h2 id="getting-started">Getting started</h2>
|
||||||
## Getting started
|
<p>At the moment, our <a href="https://github.com/ocaml/opam-repository">opam repository</a>
|
||||||
|
|
||||||
At the moment, our [opam repository](https://github.com/ocaml/opam-repository)
|
|
||||||
does not include any metadata needed for signing. We're in a bootstrap phase:
|
does not include any metadata needed for signing. We're in a bootstrap phase:
|
||||||
we need you to generate a keypair, claim your packages, and approve your releases.
|
we need you to generate a keypair, claim your packages, and approve your releases.</p>
|
||||||
|
<p>We cannot verify the main opam repository yet, but opam2 has support for a
|
||||||
We cannot verify the main opam repository yet, but opam2 has support for a
|
<a href="http://opam.ocaml.org/doc/2.0/Manual.html#configfield-repository-validation-command"><code>repository validation command</code></a>,
|
||||||
[`repository validation command`](http://opam.ocaml.org/doc/2.0/Manual.html#configfield-repository-validation-command),
|
builtin, which should then call out to <code>conex_verify</code> (there is a <code>--nostrict</code>
|
||||||
builtin, which should then call out to `conex_verify` (there is a `--nostrict`
|
flag for the impatient). There is also an <a href="https://github.com/hannesm/testrepo">example repository</a> which uses the opam validation command.</p>
|
||||||
flag for the impatient). There is also an [example repository](https://github.com/hannesm/testrepo) which uses the opam validation command.
|
<p>To reduce the manual work, we analysed 7000 PRs of the opam repository within
|
||||||
|
the last 4.5 years (more details <a href="/Posts/Maintainers">here</a>.
|
||||||
To reduce the manual work, we analysed 7000 PRs of the opam repository within
|
|
||||||
the last 4.5 years (more details [here](/Posts/Maintainers).
|
|
||||||
This resulted in an educated guess who are the people
|
This resulted in an educated guess who are the people
|
||||||
modifying each package, which we use as a basis whom to authorise for
|
modifying each package, which we use as a basis whom to authorise for
|
||||||
which packages. Please check with `conex_author status` below whether your team
|
which packages. Please check with <code>conex_author status</code> below whether your team
|
||||||
membership and authorised packages were inferred correctly.
|
membership and authorised packages were inferred correctly.</p>
|
||||||
|
<p>Each individual author - you - need to generate their private key, submit
|
||||||
Each individual author - you - need to generate their private key, submit
|
|
||||||
their public key and starts approving releases (and old ones after careful
|
their public key and starts approving releases (and old ones after careful
|
||||||
checking that the build script, patches, and tarball checksum are valid).
|
checking that the build script, patches, and tarball checksum are valid).
|
||||||
Each resource can be approved in multiple versions at the same time.
|
Each resource can be approved in multiple versions at the same time.</p>
|
||||||
|
<h3 id="installation">Installation</h3>
|
||||||
### Installation
|
<p>TODO: remove clone once <a href="https://github.com/ocaml/opam-repository/pull/8494">PR 8494</a> is merged.</p>
|
||||||
|
<pre><code class="language-bash">$ git clone -b auth https://github.com/hannesm/opam-repository.git repo
|
||||||
TODO: remove clone once [PR 8494](https://github.com/ocaml/opam-repository/pull/8494) is merged.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ git clone -b auth https://github.com/hannesm/opam-repository.git repo
|
|
||||||
$ opam install conex
|
$ opam install conex
|
||||||
$ cd repo
|
$ cd repo
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This will install conex, namely command line utilities, <code>conex_author</code> and
|
||||||
This will install conex, namely command line utilities, `conex_author` and
|
<code>conex_verify_nocrypto</code>/<code>conex_verify_openssl</code>. All files read and written by conex are in the usual
|
||||||
`conex_verify_nocrypto`/`conex_verify_openssl`. All files read and written by conex are in the usual
|
|
||||||
opam file format. This means can always manually modify them (but be careful,
|
opam file format. This means can always manually modify them (but be careful,
|
||||||
modifications need to increment counters, add checksums, and be signed). Conex
|
modifications need to increment counters, add checksums, and be signed). Conex
|
||||||
does not deal with git, you have to manually `git add` files and open pull
|
does not deal with git, you have to manually <code>git add</code> files and open pull
|
||||||
requests.
|
requests.</p>
|
||||||
|
<h3 id="author-enrollment">Author enrollment</h3>
|
||||||
### Author enrollment
|
<p>For the opam repository, we will use GitHub ids as conex ids. Thus, your conex
|
||||||
|
id and your GitHub id should match up.</p>
|
||||||
For the opam repository, we will use GitHub ids as conex ids. Thus, your conex
|
<pre><code class="language-bash">repo$ conex_author init --repo ~/repo --id hannesm
|
||||||
id and your GitHub id should match up.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ conex_author init --repo ~/repo --id hannesm
|
|
||||||
Created keypair hannesm. Join teams, claim your packages, sign your approved resources and open a PR :)
|
Created keypair hannesm. Join teams, claim your packages, sign your approved resources and open a PR :)
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This attempts to parse <code>~/repo/id/hannesm</code>, errors if it is a team or an author
|
||||||
This attempts to parse `~/repo/id/hannesm`, errors if it is a team or an author
|
|
||||||
with a publickey. Otherwise it generates a keypair, writes the private part as
|
with a publickey. Otherwise it generates a keypair, writes the private part as
|
||||||
`home.hannes.repo.hannesm.private` (the absolute path separated by dots,
|
<code>home.hannes.repo.hannesm.private</code> (the absolute path separated by dots,
|
||||||
followed by your id, and `private` - if you move your repository, rename your
|
followed by your id, and <code>private</code> - if you move your repository, rename your
|
||||||
private key) into `~/.conex/`, the checksums of the public part and your
|
private key) into <code>~/.conex/</code>, the checksums of the public part and your
|
||||||
accounts into `~/repo/id/hannesm`. See `conex_author help init` for more
|
accounts into <code>~/repo/id/hannesm</code>. See <code>conex_author help init</code> for more
|
||||||
options (esp. additional verbosity `-v` can be helpful).
|
options (esp. additional verbosity <code>-v</code> can be helpful).</p>
|
||||||
|
<pre><code class="language-bash">repo$ git status -s
|
||||||
```bash
|
|
||||||
repo$ git status -s
|
|
||||||
M id/hannesm
|
M id/hannesm
|
||||||
|
|
||||||
repo$ git diff //abbreviated output
|
repo$ git diff //abbreviated output
|
||||||
- ["counter" 0x0]
|
- ["counter" 0x0]
|
||||||
+ ["counter" 0x1]
|
+ ["counter" 0x1]
|
||||||
|
|
||||||
- ["resources" []]
|
- ["resources" []]
|
||||||
+ [
|
+ [
|
||||||
+ "resources"
|
+ "resources"
|
||||||
+ [
|
+ [
|
||||||
+ [
|
+ [
|
||||||
+ ["typ" "key"]
|
+ ["typ" "key"]
|
||||||
+ ["name" "hannesm"]
|
+ ["name" "hannesm"]
|
||||||
+ ["index" 0x1]
|
+ ["index" 0x1]
|
||||||
+ ["digest" ["SHA256" "ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY="]]
|
+ ["digest" ["SHA256" "ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY="]]
|
||||||
+ ]
|
+ ]
|
||||||
+ [
|
+ [
|
||||||
+ ["typ" "account"]
|
+ ["typ" "account"]
|
||||||
+ ["name" "hannesm"]
|
+ ["name" "hannesm"]
|
||||||
+ ["index" 0x2]
|
+ ["index" 0x2]
|
||||||
+ ["digest" ["SHA256" "aCsktJ5M9PI6T+m1NIQtuIFYILFkqoHKwBxwvuzpuzg="]]
|
+ ["digest" ["SHA256" "aCsktJ5M9PI6T+m1NIQtuIFYILFkqoHKwBxwvuzpuzg="]]
|
||||||
+ ]
|
+ ]
|
||||||
+
|
+
|
||||||
+keys: [
|
+keys: [
|
||||||
+ [
|
+ [
|
||||||
+ [
|
+ [
|
||||||
+ "RSA"
|
+ "RSA"
|
||||||
+ """
|
+ """
|
||||||
+-----BEGIN PUBLIC KEY-----
|
+-----BEGIN PUBLIC KEY-----
|
||||||
+MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAyUhArwt4XcxLanARyH9S
|
+MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAyUhArwt4XcxLanARyH9S
|
||||||
...
|
...
|
||||||
+9KQdg6QnLsQh/j74QKLOZacCAwEAAQ==
|
+9KQdg6QnLsQh/j74QKLOZacCAwEAAQ==
|
||||||
+-----END PUBLIC KEY-----"""
|
+-----END PUBLIC KEY-----"""
|
||||||
+ 0x58A3419F
|
+ 0x58A3419F
|
||||||
+ ]
|
+ ]
|
||||||
+ [
|
+ [
|
||||||
+ 0x58A79A1D
|
+ 0x58A79A1D
|
||||||
+ "RSA-PSS-SHA256"
|
+ "RSA-PSS-SHA256"
|
||||||
+ "HqqicsDx4hG9pFM5E7"
|
+ "HqqicsDx4hG9pFM5E7"
|
||||||
+ ]
|
+ ]
|
||||||
+ ]
|
+ ]
|
||||||
+]
|
+]
|
||||||
```
|
</code></pre>
|
||||||
|
<h3 id="status">Status</h3>
|
||||||
### Status
|
<p>If you have a single identity and contribute to a single signed opam repository,
|
||||||
|
you don't need to specify <code>--id</code> or <code>--repo</code> from now on.</p>
|
||||||
If you have a single identity and contribute to a single signed opam repository,
|
<p>The <code>status</code> subcommand presents an author-specific view on the repository. It
|
||||||
you don't need to specify `--id` or `--repo` from now on.
|
|
||||||
|
|
||||||
The `status` subcommand presents an author-specific view on the repository. It
|
|
||||||
lists the own public keys, team membership, queued resources, and authorised
|
lists the own public keys, team membership, queued resources, and authorised
|
||||||
packages.
|
packages.</p>
|
||||||
|
<p>The opam repository is in a transitionary state, we explicitly pass <code>--quorum 0</code>, which means that every checksum is valid (approved by a quorum of 0
|
||||||
The opam repository is in a transitionary state, we explicitly pass `--quorum
|
janitors).</p>
|
||||||
0`, which means that every checksum is valid (approved by a quorum of 0
|
<pre><code class="language-bash">repo$ conex_author status --quorum 0 arp
|
||||||
janitors).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ conex_author status --quorum 0 arp
|
|
||||||
author hannesm #1 (created 0) verified 3 resources, 0 queued
|
author hannesm #1 (created 0) verified 3 resources, 0 queued
|
||||||
4096 bit RSA key created 1487094175 approved, SHA256: ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY=
|
4096 bit RSA key created 1487094175 approved, SHA256: ht9ztjjDwWwD/id6LSVi7nKqVyCHQuQu9ORpr8Zo2aY=
|
||||||
account GitHub hannesm approved
|
account GitHub hannesm approved
|
||||||
account email hannes@mehnert.org approved
|
account email hannes@mehnert.org approved
|
||||||
package arp authorisation approved
|
package arp authorisation approved
|
||||||
conex_author: [ERROR] package index arp was not found in repository
|
conex_author: [ERROR] package index arp was not found in repository
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This shows your key material and accounts, team membership and packages you are
|
||||||
This shows your key material and accounts, team membership and packages you are
|
|
||||||
authorised to modify (inferred as described
|
authorised to modify (inferred as described
|
||||||
[here](/Posts/Maintainers).
|
<a href="/Posts/Maintainers">here</a>.</p>
|
||||||
|
<p>The <code>--noteam</code> argument limits the package list to only these you are personally
|
||||||
The `--noteam` argument limits the package list to only these you are personally
|
authorised for. The <code>--id</code> argument presents you with a view of another author,
|
||||||
authorised for. The `--id` argument presents you with a view of another author,
|
|
||||||
or from a team perspective. The positional argument is a prefix matching on
|
or from a team perspective. The positional argument is a prefix matching on
|
||||||
package names (leave empty for all).
|
package names (leave empty for all).</p>
|
||||||
|
<h3 id="resource-approval">Resource approval</h3>
|
||||||
### Resource approval
|
<p>Each resource needs to be approved individually. Each author has a local queue
|
||||||
|
for to-be-signed resources, which is extended with <code>authorisation</code>, <code>init</code>,
|
||||||
Each resource needs to be approved individually. Each author has a local queue
|
<code>key</code>, <code>release</code>, and <code>team</code> (all have a <code>--dry-run</code> flag). The queue can be
|
||||||
for to-be-signed resources, which is extended with `authorisation`, `init`,
|
dropped using <code>conex_author reset</code>. Below shown is <code>conex_author sign</code>, which
|
||||||
`key`, `release`, and `team` (all have a `--dry-run` flag). The queue can be
|
|
||||||
dropped using `conex_author reset`. Below shown is `conex_author sign`, which
|
|
||||||
let's you interactively approve queued resources and cryptopgraphically signs
|
let's you interactively approve queued resources and cryptopgraphically signs
|
||||||
your approved resources afterwards.
|
your approved resources afterwards.</p>
|
||||||
|
<p>The output of <code>conex_author status</code> listed an authorisation for <code>conf-gsl</code>,
|
||||||
The output of `conex_author status` listed an authorisation for `conf-gsl`,
|
which I don't feel responsible for. Let's drop my privileges:</p>
|
||||||
which I don't feel responsible for. Let's drop my privileges:
|
<pre><code class="language-bash">repo$ conex_author authorisation conf-gsl --remove -m hannesm
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ conex_author authorisation conf-gsl --remove -m hannesm
|
|
||||||
modified authorisation and added resource to your queue.
|
modified authorisation and added resource to your queue.
|
||||||
```
|
</code></pre>
|
||||||
|
<p>I checked my arp release careful (checksums of tarballs are correct, opam files
|
||||||
I checked my arp release careful (checksums of tarballs are correct, opam files
|
|
||||||
do not execute arbitrary shell code, etc.), and approve this package and its
|
do not execute arbitrary shell code, etc.), and approve this package and its
|
||||||
single release:
|
single release:</p>
|
||||||
|
<pre><code class="language-bash">repo$ conex_author release arp
|
||||||
```bash
|
|
||||||
repo$ conex_author release arp
|
|
||||||
conex_author.native: [WARNING] package index arp was not found in repository
|
conex_author.native: [WARNING] package index arp was not found in repository
|
||||||
conex_author.native: [WARNING] release arp.0.1.1 was not found in repository
|
conex_author.native: [WARNING] release arp.0.1.1 was not found in repository
|
||||||
wrote release and added resources to your queue.
|
wrote release and added resources to your queue.
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Once finished with joining and leaving teams (using the <code>team</code> subcommand),
|
||||||
Once finished with joining and leaving teams (using the `team` subcommand),
|
claiming packages (using the <code>authorisation</code> subcommand), and approve releases
|
||||||
claiming packages (using the `authorisation` subcommand), and approve releases
|
(using the <code>release</code> subcommand), you have to cryprographically sign your queued
|
||||||
(using the `release` subcommand), you have to cryprographically sign your queued
|
resource modifications:</p>
|
||||||
resource modifications:
|
<pre><code class="language-bash">repo$ conex_author sign
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ conex_author sign
|
|
||||||
release arp.0.1.1 #1 (created 1487269425)
|
release arp.0.1.1 #1 (created 1487269425)
|
||||||
[descr: SHA256: aCsNvcj3cBKO0GESWG4r3AzoUEnI0pHGSyEDYNPouoE=;
|
[descr: SHA256: aCsNvcj3cBKO0GESWG4r3AzoUEnI0pHGSyEDYNPouoE=;
|
||||||
opam: SHA256: nqy6lD1UP+kXj3+oPXLt2VMUIENEuHMVlVaG2V4z3p0=;
|
opam: SHA256: nqy6lD1UP+kXj3+oPXLt2VMUIENEuHMVlVaG2V4z3p0=;
|
||||||
|
|
@ -289,35 +242,26 @@ account GitHub hannesm approved
|
||||||
account email hannes@mehnert.org approved
|
account email hannes@mehnert.org approved
|
||||||
package arp authorisation approved package index approved
|
package arp authorisation approved package index approved
|
||||||
release arp.0.1.1: approved
|
release arp.0.1.1: approved
|
||||||
```
|
</code></pre>
|
||||||
|
<p>If you now modify anything in <code>packages/arp</code> (add subdirectories, modify opam,
|
||||||
If you now modify anything in `packages/arp` (add subdirectories, modify opam,
|
etc.), this will not be automatically approved (see below for how to do this).</p>
|
||||||
etc.), this will not be automatically approved (see below for how to do this).
|
<p>You manually need to <code>git add</code> some created files.</p>
|
||||||
|
<pre><code class="language-bash">repo$ git status -s
|
||||||
You manually need to `git add` some created files.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ git status -s
|
|
||||||
M id/hannesm
|
M id/hannesm
|
||||||
M packages/conf-gsl/authorisation
|
M packages/conf-gsl/authorisation
|
||||||
?? packages/arp/arp.0.1.1/release
|
?? packages/arp/arp.0.1.1/release
|
||||||
?? packages/arp/package
|
?? packages/arp/package
|
||||||
|
|
||||||
repo$ git add packages/arp/arp.0.1.1/release packages/arp/package
|
repo$ git add packages/arp/arp.0.1.1/release packages/arp/package
|
||||||
repo$ git commit -m "hannesm key enrollment and some fixes" id packages
|
repo$ git commit -m "hannesm key enrollment and some fixes" id packages
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Now push this to your fork, and open a PR on opam-repository!</p>
|
||||||
Now push this to your fork, and open a PR on opam-repository!
|
<h3 id="editing-a-package">Editing a package</h3>
|
||||||
|
<p>If you need to modify a released package, you modify the opam file (as before,
|
||||||
### Editing a package
|
|
||||||
|
|
||||||
If you need to modify a released package, you modify the opam file (as before,
|
|
||||||
e.g. introducing a conflict with a dependency), and then approve the
|
e.g. introducing a conflict with a dependency), and then approve the
|
||||||
modifications. After your local modifications, `conex_author status` will
|
modifications. After your local modifications, <code>conex_author status</code> will
|
||||||
complain:
|
complain:</p>
|
||||||
|
<pre><code class="language-bash">repo$ conex_author status arp --quorum 0
|
||||||
```bash
|
|
||||||
repo$ conex_author status arp --quorum 0
|
|
||||||
package arp authorisation approved package index approved
|
package arp authorisation approved package index approved
|
||||||
release arp.0.1.1: checksums for arp.0.1.1 differ, missing on disk: empty, missing in checksums file: empty, checksums differ: [have opam: SHA256: QSGUU9HdPOrwoRs6XJka4cZpd8h+8NN1Auu5IMN8ew4= want opam: SHA256: nqy6lD1UP+kXj3+oPXLt2VMUIENEuHMVlVaG2V4z3p0=]
|
release arp.0.1.1: checksums for arp.0.1.1 differ, missing on disk: empty, missing in checksums file: empty, checksums differ: [have opam: SHA256: QSGUU9HdPOrwoRs6XJka4cZpd8h+8NN1Auu5IMN8ew4= want opam: SHA256: nqy6lD1UP+kXj3+oPXLt2VMUIENEuHMVlVaG2V4z3p0=]
|
||||||
|
|
||||||
|
|
@ -331,62 +275,43 @@ opam: SHA256: QSGUU9HdPOrwoRs6XJka4cZpd8h+8NN1Auu5IMN8ew4=;
|
||||||
url: SHA256: FaUPievda6cEMjNkWdi0kGVK7t6EpWGfQ4q2NTSTcy0=]
|
url: SHA256: FaUPievda6cEMjNkWdi0kGVK7t6EpWGfQ4q2NTSTcy0=]
|
||||||
approved (yes/No)? y
|
approved (yes/No)? y
|
||||||
wrote hannesm to disk
|
wrote hannesm to disk
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The <code>release</code> subcommand recomputed the checksums, incremented the counter, and
|
||||||
The `release` subcommand recomputed the checksums, incremented the counter, and
|
added it to your queue. The <code>sign</code> command signed the approved resource.</p>
|
||||||
added it to your queue. The `sign` command signed the approved resource.
|
<pre><code class="language-bash">repo$ git status -s
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ git status -s
|
|
||||||
M id/hannesm
|
M id/hannesm
|
||||||
M packages/arp/arp.0.1.1/opam
|
M packages/arp/arp.0.1.1/opam
|
||||||
M packages/arp/arp.0.1.1/package
|
M packages/arp/arp.0.1.1/package
|
||||||
|
|
||||||
repo$ git commit -m "fixed broken arp package" id packages
|
repo$ git commit -m "fixed broken arp package" id packages
|
||||||
```
|
</code></pre>
|
||||||
|
<h3 id="janitor-tools">Janitor tools</h3>
|
||||||
### Janitor tools
|
<p>Janitors need to approve teams, keys, accounts, and authorisations.</p>
|
||||||
|
<p>To approve resources which are already in the repository on disk,
|
||||||
Janitors need to approve teams, keys, accounts, and authorisations.
|
the <code>key</code> subcommand queues approval of keys and accounts of the provided author:</p>
|
||||||
|
<pre><code class="language-bash">repo$ conex_author key avsm
|
||||||
To approve resources which are already in the repository on disk,
|
|
||||||
the `key` subcommand queues approval of keys and accounts of the provided author:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
repo$ conex_author key avsm
|
|
||||||
added keys and accounts to your resource list.
|
added keys and accounts to your resource list.
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The <code>authorisation</code> subcommand, and <code>team</code> subcommand behave similarly for
|
||||||
The `authorisation` subcommand, and `team` subcommand behave similarly for
|
authorisations and teams.</p>
|
||||||
authorisations and teams.
|
<p>Bulk operations are supported as well:</p>
|
||||||
|
<pre><code class="language-bash">conex_author authorisation all
|
||||||
Bulk operations are supported as well:
|
</code></pre>
|
||||||
|
<p>This will approve all authorisations of the repository which are not yet
|
||||||
```bash
|
approved by you. Similar for the <code>key</code> and <code>team</code> subcommands, which also
|
||||||
conex_author authorisation all
|
accept <code>all</code>.</p>
|
||||||
```
|
<p>Don't forget to <code>conex_author sign</code> afterwards (or <code>yes | conex_author sign</code>).</p>
|
||||||
|
<h3 id="verification">Verification</h3>
|
||||||
This will approve all authorisations of the repository which are not yet
|
<p>The two command line utlities, <code>conex_verify_openssl</code> and
|
||||||
approved by you. Similar for the `key` and `team` subcommands, which also
|
<code>conex_verify_nocrypto</code> contain the same logic and same command line arguments.</p>
|
||||||
accept `all`.
|
<p>For bootstrapping purposes (<code>nocrypto</code> is an opam package with dependencies),
|
||||||
|
<code>conex_verify_openssl</code> relies on the openssl command line tool (version 1.0.0
|
||||||
Don't forget to `conex_author sign` afterwards (or `yes | conex_author sign`).
|
and above) for digest computation and verification of the RSA-PSS signature.</p>
|
||||||
|
<p>The goal is to use the opam2 provided hooks, but before we have signatures we
|
||||||
### Verification
|
cannot enable them.</p>
|
||||||
|
<p>See the <a href="https://github.com/hannesm/testrepo">example repository</a> for initial
|
||||||
The two command line utlities, `conex_verify_openssl` and
|
verification experiments, and opam2 integration.</p>
|
||||||
`conex_verify_nocrypto` contain the same logic and same command line arguments.
|
<p>I'm interested in feedback, please open an issue on the <a href="https://github.com/hannesm/conex">conex
|
||||||
|
repository</a>. This article itself is stored as
|
||||||
For bootstrapping purposes (`nocrypto` is an opam package with dependencies),
|
Markdown <a href="https://git.robur.coop/hannes/hannes.robur.coop">in a different repository</a>.</p>
|
||||||
`conex_verify_openssl` relies on the openssl command line tool (version 1.0.0
|
</article></div></div></main></body></html>
|
||||||
and above) for digest computation and verification of the RSA-PSS signature.
|
|
||||||
|
|
||||||
The goal is to use the opam2 provided hooks, but before we have signatures we
|
|
||||||
cannot enable them.
|
|
||||||
|
|
||||||
See the [example repository](https://github.com/hannesm/testrepo) for initial
|
|
||||||
verification experiments, and opam2 integration.
|
|
||||||
|
|
||||||
I'm interested in feedback, please open an issue on the [conex
|
|
||||||
repository](https://github.com/hannesm/conex). This article itself is stored as
|
|
||||||
Markdown [in a different repository](https://git.robur.coop/hannes/hannes.robur.coop).
|
|
||||||
312
Posts/DNS
312
Posts/DNS
|
|
@ -1,40 +1,31 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: My 2018 contains robur and starts with re-engineering DNS
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>My 2018 contains robur and starts with re-engineering DNS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="My 2018 contains robur and starts with re-engineering DNS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2018-01-11 (last updated: 2023-11-20)</span><article><h2 id="section">2018</h2>
|
||||||
author: hannes
|
<p>At the end of 2017, I resigned from my PostDoc position at University of
|
||||||
tags: mirageos, protocol
|
Cambridge (in the <a href="https://www.cl.cam.ac.uk/~pes20/rems/">rems</a> project). Early
|
||||||
abstract: New year brings new possibilities and a new environment. I've been working on the most Widely deployed key-value store, the domain name system. Primary and secondary name services are available, including dynamic updates, notify, and tsig authentication.
|
December 2017 I organised the <a href="https://mirage.io/blog/2017-winter-hackathon-roundup">4th MirageOS hack
|
||||||
---
|
retreat</a>, with which I'm
|
||||||
|
very satisfied. In March 2018 the <a href="http://retreat.mirage.io">5th retreat</a> will
|
||||||
## 2018
|
happen (please sign up!).</p>
|
||||||
|
<p>In 2018 I moved to Berlin and started to work for the (non-profit) <a href="https://techcultivation.org">Center for
|
||||||
At the end of 2017, I resigned from my PostDoc position at University of
|
the cultivation of technology</a> with our
|
||||||
Cambridge (in the [rems](https://www.cl.cam.ac.uk/~pes20/rems/) project). Early
|
<a href="http://robur.coop">robur.coop</a> project "At robur, we build performant bespoke
|
||||||
December 2017 I organised the [4th MirageOS hack
|
minimal operating systems for high-assurance services". robur is only possible
|
||||||
retreat](https://mirage.io/blog/2017-winter-hackathon-roundup), with which I'm
|
|
||||||
very satisfied. In March 2018 the [5th retreat](http://retreat.mirage.io) will
|
|
||||||
happen (please sign up!).
|
|
||||||
|
|
||||||
In 2018 I moved to Berlin and started to work for the (non-profit) [Center for
|
|
||||||
the cultivation of technology](https://techcultivation.org) with our
|
|
||||||
[robur.coop](http://robur.coop) project "At robur, we build performant bespoke
|
|
||||||
minimal operating systems for high-assurance services". robur is only possible
|
|
||||||
by generous donations in autumn 2017, enthusiastic collaborateurs, supportive
|
by generous donations in autumn 2017, enthusiastic collaborateurs, supportive
|
||||||
friends, and a motivated community, thanks to all. We will receive funding from
|
friends, and a motivated community, thanks to all. We will receive funding from
|
||||||
the [prototypefund](https://prototypefund.de/project/robur-io/) to work on a
|
the <a href="https://prototypefund.de/project/robur-io/">prototypefund</a> to work on a
|
||||||
[CalDAV server](https://robur.coop/Our%20Work/Projects#CalDAV-Server) implementation in OCaml
|
<a href="https://robur.coop/Our%20Work/Projects#CalDAV-Server">CalDAV server</a> implementation in OCaml
|
||||||
targeting MirageOS. We're still looking for donations and further funding,
|
targeting MirageOS. We're still looking for donations and further funding,
|
||||||
please get in touch. Apart from CalDAV, I want to start the year by finishing
|
please get in touch. Apart from CalDAV, I want to start the year by finishing
|
||||||
several projects which I discovered on my hard drive. This includes DNS, [opam
|
several projects which I discovered on my hard drive. This includes DNS, <a href="/Posts/Conex">opam
|
||||||
signing](/Posts/Conex), TCP, ... . My personal goal for 2018 is to develop a
|
signing</a>, TCP, ... . My personal goal for 2018 is to develop a
|
||||||
flexible `mirage deploy`, because after configuring and building a unikernel, I
|
flexible <code>mirage deploy</code>, because after configuring and building a unikernel, I
|
||||||
want to get it smoothly up and running (spoiler: I already use
|
want to get it smoothly up and running (spoiler: I already use
|
||||||
[albatross](/Posts/VMM) in production).
|
<a href="/Posts/VMM">albatross</a> in production).</p>
|
||||||
|
<p>To kick off (3% of 2018 is already used) this year, I'll talk in more detail
|
||||||
To kick off (3% of 2018 is already used) this year, I'll talk in more detail
|
about <a href="https://github.com/roburio/udns">µDNS</a>, an opinionated from-scratch
|
||||||
about [µDNS](https://github.com/roburio/udns), an opinionated from-scratch
|
|
||||||
re-engineered DNS library, which I've been using since Christmas 2017 in production for
|
re-engineered DNS library, which I've been using since Christmas 2017 in production for
|
||||||
[ns.nqsb.io](https://github.com/hannesm/ns.nqsb.io) and
|
<a href="https://github.com/hannesm/ns.nqsb.io">ns.nqsb.io</a> and
|
||||||
[ns.robur.io](https://git.robur.io/?p=ns.robur.io.git;a=summary). The
|
<a href="https://git.robur.io/?p=ns.robur.io.git;a=summary">ns.robur.io</a>. The
|
||||||
development started in March 2017, and continued over several evenings and long
|
development started in March 2017, and continued over several evenings and long
|
||||||
weekends. My initial motivation was to implement a recursive resolver to run on
|
weekends. My initial motivation was to implement a recursive resolver to run on
|
||||||
my laptop. I had a working prototype in use on my laptop over 4 months in the
|
my laptop. I had a working prototype in use on my laptop over 4 months in the
|
||||||
|
|
@ -44,72 +35,61 @@ resolver needs a server, as local overlay, usually anyways. Furthermore,
|
||||||
dynamic updates are standardised and thus a configuration interface exists
|
dynamic updates are standardised and thus a configuration interface exists
|
||||||
inside the protocol, even with hmac-signatures for authentication!
|
inside the protocol, even with hmac-signatures for authentication!
|
||||||
Coincidentally, I started to solve another issue, namely automated management of let's
|
Coincidentally, I started to solve another issue, namely automated management of let's
|
||||||
encrypt certificates (see [this
|
encrypt certificates (see <a href="https://github.com/hannesm/ocaml-letsencrypt/tree/nsupdate">this
|
||||||
branch](https://github.com/hannesm/ocaml-letsencrypt/tree/nsupdate) for an
|
branch</a> for an
|
||||||
initial hack). On my journey, I also reported a cache poisoning vulnerability,
|
initial hack). On my journey, I also reported a cache poisoning vulnerability,
|
||||||
which was fixed in [Docker for
|
which was fixed in <a href="https://docs.docker.com/docker-for-windows/release-notes/#docker-community-edition-17090-ce-win32-2017-10-02-stable">Docker for
|
||||||
Windows](https://docs.docker.com/docker-for-windows/release-notes/#docker-community-edition-17090-ce-win32-2017-10-02-stable).
|
Windows</a>.</p>
|
||||||
|
<p>But let's get started with some content. Please keep in mind that while the
|
||||||
But let's get started with some content. Please keep in mind that while the
|
|
||||||
code is publicly available, it is not yet released (mainly since the test
|
code is publicly available, it is not yet released (mainly since the test
|
||||||
coverage is not high enough, and the lack of documentation). I appreciate early
|
coverage is not high enough, and the lack of documentation). I appreciate early
|
||||||
adopters, please let me know if you find any issues or find a use case which is
|
adopters, please let me know if you find any issues or find a use case which is
|
||||||
not straightforward to solve. This won't be the last article about DNS this
|
not straightforward to solve. This won't be the last article about DNS this
|
||||||
year - persistent storage, resolver, let's encrypt support are still missing.
|
year - persistent storage, resolver, let's encrypt support are still missing.</p>
|
||||||
|
<h2 id="what-is-dns">What is DNS?</h2>
|
||||||
## What is DNS?
|
<p>The <a href="https://en.wikipedia.org/wiki/DNS">domain name system</a> is a core Internet
|
||||||
|
|
||||||
The [domain name system](https://en.wikipedia.org/wiki/DNS) is a core Internet
|
|
||||||
protocol, which translates domain names to IP addresses. A domain name is
|
protocol, which translates domain names to IP addresses. A domain name is
|
||||||
easier to memorise for human beings than an IP address. DNS is hierarchical and
|
easier to memorise for human beings than an IP address. DNS is hierarchical and
|
||||||
decentralised. It was initially "specified" in Nov 1987 in [RFC
|
decentralised. It was initially "specified" in Nov 1987 in <a href="https://tools.ietf.org/html/rfc1034">RFC
|
||||||
1034](https://tools.ietf.org/html/rfc1034) and [RFC
|
1034</a> and <a href="https://tools.ietf.org/html/rfc1035">RFC
|
||||||
1035](https://tools.ietf.org/html/rfc1035). Nowadays it spans over more than 20
|
1035</a>. Nowadays it spans over more than 20
|
||||||
technical RFCs, 10 security related, 5 best current practises and another 10
|
technical RFCs, 10 security related, 5 best current practises and another 10
|
||||||
informational. The basic encoding and mechanisms did not change.
|
informational. The basic encoding and mechanisms did not change.</p>
|
||||||
|
<p>On the Internet, there is a set of root servers (administrated by IANA) which
|
||||||
On the Internet, there is a set of root servers (administrated by IANA) which
|
|
||||||
provide the information about which name servers are authoritative for which top level
|
provide the information about which name servers are authoritative for which top level
|
||||||
domain (such as ".com"). They provide the information about which name servers are
|
domain (such as ".com"). They provide the information about which name servers are
|
||||||
responsible for which second level domain name (such as "example.com"), and so
|
responsible for which second level domain name (such as "example.com"), and so
|
||||||
on. There are at least two name servers for each domain name in separate
|
on. There are at least two name servers for each domain name in separate
|
||||||
networks - in case one is unavailable the other can be reached.
|
networks - in case one is unavailable the other can be reached.</p>
|
||||||
|
<p>The building blocks for DNS are: the resolver, a stub (<code>gethostbyname</code> provided
|
||||||
The building blocks for DNS are: the resolver, a stub (`gethostbyname` provided
|
|
||||||
by your C library) or caching forwarding resolver (at your ISP), which send DNS
|
by your C library) or caching forwarding resolver (at your ISP), which send DNS
|
||||||
packets to another resolver, or a recursive resolver which, once seeded with the
|
packets to another resolver, or a recursive resolver which, once seeded with the
|
||||||
root servers, finds out the IP address of a requested domain name. The other
|
root servers, finds out the IP address of a requested domain name. The other
|
||||||
part are authoritative servers, which reply to requests for their configured
|
part are authoritative servers, which reply to requests for their configured
|
||||||
domain.
|
domain.</p>
|
||||||
|
<p>To get some terminology, a DNS client sends a query, consisting of a domain
|
||||||
To get some terminology, a DNS client sends a query, consisting of a domain
|
|
||||||
name and a query type, and expects a set of answers, which are called resource
|
name and a query type, and expects a set of answers, which are called resource
|
||||||
records, and contain: name, time to live, type, and data. The resolver
|
records, and contain: name, time to live, type, and data. The resolver
|
||||||
iteratively requests resource records from authoritative servers, until the requested
|
iteratively requests resource records from authoritative servers, until the requested
|
||||||
domain name is resolved or fails (name does not exist, server
|
domain name is resolved or fails (name does not exist, server
|
||||||
failure, server offline).
|
failure, server offline).</p>
|
||||||
|
<p>DNS usually uses UDP as transport which is not reliable and limited to 512 byte
|
||||||
DNS usually uses UDP as transport which is not reliable and limited to 512 byte
|
|
||||||
payload on the Internet (due to various middleboxes). DNS can also be
|
payload on the Internet (due to various middleboxes). DNS can also be
|
||||||
transported via TCP, and even via TLS over UDP or TCP. If a DNS packet
|
transported via TCP, and even via TLS over UDP or TCP. If a DNS packet
|
||||||
transferred via UDP is larger than 512 bytes, it is cut at the 512 byte mark,
|
transferred via UDP is larger than 512 bytes, it is cut at the 512 byte mark,
|
||||||
and a bit in its header is set. The receiver can decide whether to use the 512
|
and a bit in its header is set. The receiver can decide whether to use the 512
|
||||||
bytes of information, or to throw it away and attempt a TCP connection.
|
bytes of information, or to throw it away and attempt a TCP connection.</p>
|
||||||
|
<h3 id="dns-packet">DNS packet</h3>
|
||||||
### DNS packet
|
<p>The packet encoding starts with a 16bit identifier followed by a 16bit header
|
||||||
|
|
||||||
The packet encoding starts with a 16bit identifier followed by a 16bit header
|
|
||||||
(containing operation, flags, status code), and four counters, each 16bit,
|
(containing operation, flags, status code), and four counters, each 16bit,
|
||||||
specifying the amount of resource records in the body: questions, answers,
|
specifying the amount of resource records in the body: questions, answers,
|
||||||
authority records, and additional records. The header starts with one bit
|
authority records, and additional records. The header starts with one bit
|
||||||
operation (query or response), four bits opcode, various flags (recursion,
|
operation (query or response), four bits opcode, various flags (recursion,
|
||||||
authoritative, truncation, ...), and the last four bit encode the response code.
|
authoritative, truncation, ...), and the last four bit encode the response code.</p>
|
||||||
|
<p>A question consists of a domain name, a query type, and a query class. A
|
||||||
A question consists of a domain name, a query type, and a query class. A
|
|
||||||
resource record additionally contains a 32bit time to live, a length, and the
|
resource record additionally contains a 32bit time to live, a length, and the
|
||||||
data.
|
data.</p>
|
||||||
|
<p>Each domain name is a case sensitive string of up to 255 bytes, separated by <code>.</code>
|
||||||
Each domain name is a case sensitive string of up to 255 bytes, separated by `.`
|
|
||||||
into labels of up to 63 bytes each. A label is either encoded by its length
|
into labels of up to 63 bytes each. A label is either encoded by its length
|
||||||
followed by the content, or by an offset to the start of a label in the current
|
followed by the content, or by an offset to the start of a label in the current
|
||||||
DNS frame (poor mans compression). Care must be taken during decoding to avoid
|
DNS frame (poor mans compression). Care must be taken during decoding to avoid
|
||||||
|
|
@ -117,15 +97,13 @@ cycles in offsets. Common operations on domain names are comparison: equality,
|
||||||
ordering, and also whether some domain name is a subdomain of another domain
|
ordering, and also whether some domain name is a subdomain of another domain
|
||||||
name, should be efficient. My initial representation naïvely was a list of
|
name, should be efficient. My initial representation naïvely was a list of
|
||||||
strings, now it is an array of strings in reverse order. This speeds up common
|
strings, now it is an array of strings in reverse order. This speeds up common
|
||||||
operations by a factor of 5 (see test/bench.ml).
|
operations by a factor of 5 (see test/bench.ml).</p>
|
||||||
|
<p>The only really used class is <code>IN</code> (for Internet), as mentioned in <a href="https://tools.ietf.org/html/rfc6895">RFC
|
||||||
The only really used class is `IN` (for Internet), as mentioned in [RFC
|
6895</a>. Various query types (<code>MD</code>, <code>MF</code>,
|
||||||
6895](https://tools.ietf.org/html/rfc6895). Various query types (`MD`, `MF`,
|
<code>MB</code>, <code>MG</code>, <code>MR</code>, <code>NULL</code>, <code>AFSDB</code>, ...) are barely or never used. There is no
|
||||||
`MB`, `MG`, `MR`, `NULL`, `AFSDB`, ...) are barely or never used. There is no
|
|
||||||
need to convolute the implementation and its API with these legacy options (if
|
need to convolute the implementation and its API with these legacy options (if
|
||||||
you have a use case and see those in the wild, please tell me).
|
you have a use case and see those in the wild, please tell me).</p>
|
||||||
|
<p>My implemented packet decoding does decompression, only allows valid internet
|
||||||
My implemented packet decoding does decompression, only allows valid internet
|
|
||||||
domain names, and may return a partial parse - to use as many resource records
|
domain names, and may return a partial parse - to use as many resource records
|
||||||
in truncated packets as possible. There are no exceptions raised, the parsing
|
in truncated packets as possible. There are no exceptions raised, the parsing
|
||||||
uses a monadic style error handling. Since label decompression requires the
|
uses a monadic style error handling. Since label decompression requires the
|
||||||
|
|
@ -134,9 +112,8 @@ passed around at all times, instead of using smaller views on the buffer. The
|
||||||
decoder does not allow for gaps, when the outer resource data length specifies a
|
decoder does not allow for gaps, when the outer resource data length specifies a
|
||||||
byte length which is not completely consumed by the specific resource data
|
byte length which is not completely consumed by the specific resource data
|
||||||
subparser (an A record must always consume four bytes). Failing to check this can
|
subparser (an A record must always consume four bytes). Failing to check this can
|
||||||
lead to a way to exfiltrate data without getting noticed.
|
lead to a way to exfiltrate data without getting noticed.</p>
|
||||||
|
<p>Each zone (a served domain name) contains a SOA "start of authority" entry,
|
||||||
Each zone (a served domain name) contains a SOA "start of authority" entry,
|
|
||||||
which includes the primary nameserver name, the hostmaster's email address (both
|
which includes the primary nameserver name, the hostmaster's email address (both
|
||||||
encoded as domain name), a serial number of the zone, a refresh, retry, expiry,
|
encoded as domain name), a serial number of the zone, a refresh, retry, expiry,
|
||||||
and minimum interval (all encoded as 32bit unsigned number in seconds). Common
|
and minimum interval (all encoded as 32bit unsigned number in seconds). Common
|
||||||
|
|
@ -145,68 +122,56 @@ resource records include A, which payload is 32bit IPv4 address. A nameserver
|
||||||
payload is a 16bit priority and a domain name. A CNAME record is an alias to
|
payload is a 16bit priority and a domain name. A CNAME record is an alias to
|
||||||
another domain name. These days, there are even records to specify the
|
another domain name. These days, there are even records to specify the
|
||||||
certificate authority authorisation (CAA) records containing a flag (critical),
|
certificate authority authorisation (CAA) records containing a flag (critical),
|
||||||
a tag ("issue") and a value ("letsencrypt.org").
|
a tag ("issue") and a value ("letsencrypt.org").</p>
|
||||||
|
<h2 id="server">Server</h2>
|
||||||
## Server
|
<p>The operation of a DNS server is to listen for a request and serve a reply.
|
||||||
|
|
||||||
The operation of a DNS server is to listen for a request and serve a reply.
|
|
||||||
Data to be served can be canonically encoded (the RFC describes the format) in a
|
Data to be served can be canonically encoded (the RFC describes the format) in a
|
||||||
zone file. Apart from insecurity in DNS server implementations, another attack
|
zone file. Apart from insecurity in DNS server implementations, another attack
|
||||||
vector are amplification attacks where an attacker crafts a small UDP frame
|
vector are amplification attacks where an attacker crafts a small UDP frame
|
||||||
with a fake source IP address, and the server answers with a large response to
|
with a fake source IP address, and the server answers with a large response to
|
||||||
that address which may lead to a DoS attack. Various mitigations exist
|
that address which may lead to a DoS attack. Various mitigations exist
|
||||||
including rate limiting, serving large replies only via TCP, ...
|
including rate limiting, serving large replies only via TCP, ...</p>
|
||||||
|
<p>Internally, the zone file data is stored in a tree (module
|
||||||
Internally, the zone file data is stored in a tree (module
|
<a href="https://github.com/roburio/udns/blob/master/server/dns_trie.mli">Dns_trie</a>
|
||||||
[Dns_trie](https://github.com/roburio/udns/blob/master/server/dns_trie.mli)
|
<a href="https://github.com/roburio/udns/blob/master/server/dns_trie.ml">implementation</a>),
|
||||||
[implementation](https://github.com/roburio/udns/blob/master/server/dns_trie.ml)),
|
where each node contains two maps: <code>sub</code>, which key is a label and value is a
|
||||||
where each node contains two maps: `sub`, which key is a label and value is a
|
subtree and <code>dns_map</code> (module Dns_map), which key is a resource record type and
|
||||||
subtree and `dns_map` (module Dns_map), which key is a resource record type and
|
|
||||||
value is the resource record. Both use the OCaml
|
value is the resource record. Both use the OCaml
|
||||||
[Map](http://caml.inria.fr/pub/docs/manual-ocaml/libref/Map.html) ("also known
|
<a href="http://caml.inria.fr/pub/docs/manual-ocaml/libref/Map.html">Map</a> ("also known
|
||||||
as finite maps or dictionaries, given a total ordering function over the
|
as finite maps or dictionaries, given a total ordering function over the
|
||||||
keys. All operations over maps are purely applicative (no side-effects). The
|
keys. All operations over maps are purely applicative (no side-effects). The
|
||||||
implementation uses balanced binary trees, and therefore searching and insertion
|
implementation uses balanced binary trees, and therefore searching and insertion
|
||||||
take time logarithmic in the size of the map").
|
take time logarithmic in the size of the map").</p>
|
||||||
|
<p>The server looks up the queried name, and in the returned Dns_map the queried
|
||||||
The server looks up the queried name, and in the returned Dns_map the queried
|
|
||||||
type. The found resource records are sent as answer, which also includes the
|
type. The found resource records are sent as answer, which also includes the
|
||||||
question and authority information (NS records of the zone) and additional glue
|
question and authority information (NS records of the zone) and additional glue
|
||||||
records (IP addresses of names mentioned earlier in the same zone).
|
records (IP addresses of names mentioned earlier in the same zone).</p>
|
||||||
|
<h3 id="dns_map">Dns_map</h3>
|
||||||
### Dns_map
|
<p>The data structure which contains resource record types as key, and a collection
|
||||||
|
|
||||||
The data structure which contains resource record types as key, and a collection
|
|
||||||
of matching resource records as values. In OCaml the value type must be
|
of matching resource records as values. In OCaml the value type must be
|
||||||
homogenous - using a normal sum type leads to an unneccessary unpacking step
|
homogenous - using a normal sum type leads to an unneccessary unpacking step
|
||||||
(or lacking type information):
|
(or lacking type information):</p>
|
||||||
|
<pre><code class="language-OCaml">let lookup_ns t =
|
||||||
```OCaml
|
|
||||||
let lookup_ns t =
|
|
||||||
match Map.find NS t with
|
match Map.find NS t with
|
||||||
| None -> Error `NotFound
|
| None -> Error `NotFound
|
||||||
| Some (NS nameservers) -> Ok nameservers
|
| Some (NS nameservers) -> Ok nameservers
|
||||||
| Some _ -> Error `NotFound
|
| Some _ -> Error `NotFound
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Instead, I use in my current rewrite <a href="https://en.wikipedia.org/wiki/Generalized_algebraic_data_type">generalized algebraic data
|
||||||
Instead, I use in my current rewrite [generalized algebraic data
|
types</a> (read
|
||||||
types](https://en.wikipedia.org/wiki/Generalized_algebraic_data_type) (read
|
<a href="http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#sec251">OCaml manual</a> and
|
||||||
[OCaml manual](http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#sec251) and
|
<a href="http://mads-hartmann.com/ocaml/2015/01/05/gadt-ocaml.html">Mads Hartmann blog post about use cases for
|
||||||
[Mads Hartmann blog post about use cases for
|
GADTs</a>, <a href="https://andreas.github.io/2018/01/05/modeling-graphql-type-modifiers-with-gadts/">Andreas
|
||||||
GADTs](http://mads-hartmann.com/ocaml/2015/01/05/gadt-ocaml.html), [Andreas
|
|
||||||
Garnæs about using GADTs for GraphQL type
|
Garnæs about using GADTs for GraphQL type
|
||||||
modifiers](https://andreas.github.io/2018/01/05/modeling-graphql-type-modifiers-with-gadts/))
|
modifiers</a>)
|
||||||
to preserve a relation between key and value (and A record has a list of IPv4
|
to preserve a relation between key and value (and A record has a list of IPv4
|
||||||
addresses and a ttl as value) - similar to
|
addresses and a ttl as value) - similar to
|
||||||
[hmap](http://erratique.ch/software/hmap), but different: a closed key-value
|
<a href="http://erratique.ch/software/hmap">hmap</a>, but different: a closed key-value
|
||||||
mapping (the GADT), no int for each key and mutable state. Thanks to Justus
|
mapping (the GADT), no int for each key and mutable state. Thanks to Justus
|
||||||
Matthiesen for helping me with GADTs and this code. Look into the
|
Matthiesen for helping me with GADTs and this code. Look into the
|
||||||
[interface](https://github.com/roburio/udns/blob/master/src/dns_map.mli) and
|
<a href="https://github.com/roburio/udns/blob/master/src/dns_map.mli">interface</a> and
|
||||||
[implementation](https://github.com/roburio/udns/blob/master/src/dns_map.ml).
|
<a href="https://github.com/roburio/udns/blob/master/src/dns_map.ml">implementation</a>.</p>
|
||||||
|
<pre><code class="language-OCaml">(* an ordering relation, I dislike using int for that *)
|
||||||
|
|
||||||
```OCaml
|
|
||||||
(* an ordering relation, I dislike using int for that *)
|
|
||||||
module Order = struct
|
module Order = struct
|
||||||
type (_,_) t =
|
type (_,_) t =
|
||||||
| Lt : ('a, 'b) t
|
| Lt : ('a, 'b) t
|
||||||
|
|
@ -223,88 +188,79 @@ module Key = struct
|
||||||
| Cname : (int32 * Dns_name.t) t
|
| Cname : (int32 * Dns_name.t) t
|
||||||
|
|
||||||
(* we need a total order on our keys *)
|
(* we need a total order on our keys *)
|
||||||
let compare : type a b. a t -> b t -> (a, b) Order.t = fun t t' ->
|
let compare : type a b. a t -> b t -> (a, b) Order.t = fun t t' ->
|
||||||
let open Order in
|
let open Order in
|
||||||
match t, t' with
|
match t, t' with
|
||||||
| Cname, Cname -> Eq | Cname, _ -> Lt | _, Cname -> Gt
|
| Cname, Cname -> Eq | Cname, _ -> Lt | _, Cname -> Gt
|
||||||
| Ns, Ns -> Eq | Ns, _ -> Lt | _, Ns -> Gt
|
| Ns, Ns -> Eq | Ns, _ -> Lt | _, Ns -> Gt
|
||||||
| Soa, Soa -> Eq | Soa, _ -> Lt | _, Soa -> Gt
|
| Soa, Soa -> Eq | Soa, _ -> Lt | _, Soa -> Gt
|
||||||
| A, A -> Eq
|
| A, A -> Eq
|
||||||
end
|
end
|
||||||
|
|
||||||
type 'a key = 'a Key.t
|
type 'a key = 'a Key.t
|
||||||
|
|
||||||
(* our OCaml Map with an encapsulated constructor as key *)
|
(* our OCaml Map with an encapsulated constructor as key *)
|
||||||
type k = K : 'a key -> k
|
type k = K : 'a key -> k
|
||||||
module M = Map.Make(struct
|
module M = Map.Make(struct
|
||||||
type t = k
|
type t = k
|
||||||
(* the price I pay for not using int as three-state value *)
|
(* the price I pay for not using int as three-state value *)
|
||||||
let compare (K a) (K b) = match Key.compare a b with
|
let compare (K a) (K b) = match Key.compare a b with
|
||||||
| Order.Lt -> -1
|
| Order.Lt -> -1
|
||||||
| Order.Eq -> 0
|
| Order.Eq -> 0
|
||||||
| Order.Gt -> 1
|
| Order.Gt -> 1
|
||||||
end)
|
end)
|
||||||
|
|
||||||
(* v contains a key and value pair, wrapped by a single constructor *)
|
(* v contains a key and value pair, wrapped by a single constructor *)
|
||||||
type v = V : 'a key * 'a -> v
|
type v = V : 'a key * 'a -> v
|
||||||
|
|
||||||
(* t is the main type of a Dns_map, used by clients *)
|
(* t is the main type of a Dns_map, used by clients *)
|
||||||
type t = v M.t
|
type t = v M.t
|
||||||
|
|
||||||
(* retrieve a typed value out of the store *)
|
(* retrieve a typed value out of the store *)
|
||||||
let get : type a. a Key.t -> t -> a = fun k t ->
|
let get : type a. a Key.t -> t -> a = fun k t ->
|
||||||
match M.find (K k) t with
|
match M.find (K k) t with
|
||||||
| V (k', v) ->
|
| V (k', v) ->
|
||||||
(* this comparison is superfluous, just for the types *)
|
(* this comparison is superfluous, just for the types *)
|
||||||
match Key.compare k k' with
|
match Key.compare k k' with
|
||||||
| Order.Eq -> v
|
| Order.Eq -> v
|
||||||
| _ -> assert false
|
| _ -> assert false
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This helps me to programmaticaly retrieve tightly typed values from the cache,
|
||||||
This helps me to programmaticaly retrieve tightly typed values from the cache,
|
|
||||||
important when code depends on concrete values (i.e. when there are domain
|
important when code depends on concrete values (i.e. when there are domain
|
||||||
names, look these up as well and add as additional records). Look into [server/dns_server.ml](https://github.com/roburio/udns/blob/master/server/dns_server.ml)
|
names, look these up as well and add as additional records). Look into <a href="https://github.com/roburio/udns/blob/master/server/dns_server.ml">server/dns_server.ml</a></p>
|
||||||
|
<h3 id="dynamic-updates-notifications-and-authentication">Dynamic updates, notifications, and authentication</h3>
|
||||||
### Dynamic updates, notifications, and authentication
|
<p><a href="https://tools.ietf.org/html/rfc2136">Dynamic updates</a> specify in-protocol
|
||||||
|
record updates (supported for example by <code>nsupdate</code> from ISC bind-tools),
|
||||||
[Dynamic updates](https://tools.ietf.org/html/rfc2136) specify in-protocol
|
<a href="https://tools.ietf.org/html/rfc1996">notifications</a> are used by primary servers
|
||||||
record updates (supported for example by `nsupdate` from ISC bind-tools),
|
to notify secondary servers about updates, which then initiate a <a href="https://tools.ietf.org/html/rfc5936">zone
|
||||||
[notifications](https://tools.ietf.org/html/rfc1996) are used by primary servers
|
transfer</a> to retrieve up to date
|
||||||
to notify secondary servers about updates, which then initiate a [zone
|
data. <a href="https://tools.ietf.org/html/rfc2845">Shared hmac secrets</a> are used to
|
||||||
transfer](https://tools.ietf.org/html/rfc5936) to retrieve up to date
|
|
||||||
data. [Shared hmac secrets](https://tools.ietf.org/html/rfc2845) are used to
|
|
||||||
ensure that the transaction (update, zone transfer) was authorised. These are
|
ensure that the transaction (update, zone transfer) was authorised. These are
|
||||||
all protocol extensions, there is no need to use out-of-protocol solutions.
|
all protocol extensions, there is no need to use out-of-protocol solutions.</p>
|
||||||
|
<p>The server logic for update and zone transfer frames is slightly more complex,
|
||||||
The server logic for update and zone transfer frames is slightly more complex,
|
|
||||||
and includes a dependency upon an authenticator (implemented using the
|
and includes a dependency upon an authenticator (implemented using the
|
||||||
[nocrypto](https://github.com/mirleft/ocaml-nocrypto) library, and
|
<a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a> library, and
|
||||||
[ptime](http://erratique.ch/software/ptime)).
|
<a href="http://erratique.ch/software/ptime">ptime</a>).</p>
|
||||||
|
<h3 id="deployment-and-lets-encrypt">Deployment and Let's Encrypt</h3>
|
||||||
### Deployment and Let's Encrypt
|
<p>To deploy servers without much persistent data, an authentication schema is
|
||||||
|
|
||||||
To deploy servers without much persistent data, an authentication schema is
|
|
||||||
hardcoded in the dns-server: shared secrets are also stored as DNS entries
|
hardcoded in the dns-server: shared secrets are also stored as DNS entries
|
||||||
(DNSKEY), and `_transfer.zone`, `_update.zone`, and `_key-management.zone` names
|
(DNSKEY), and <code>_transfer.zone</code>, <code>_update.zone</code>, and <code>_key-management.zone</code> names
|
||||||
are introduced to encode the permissions. A `_transfer` key also needs to
|
are introduced to encode the permissions. A <code>_transfer</code> key also needs to
|
||||||
encode the IP address of the primary (to know where to request zone transfers)
|
encode the IP address of the primary (to know where to request zone transfers)
|
||||||
and secondary IP (to know where to send notifications).
|
and secondary IP (to know where to send notifications).</p>
|
||||||
|
<p>Please have a look at
|
||||||
Please have a look at
|
<a href="https://git.robur.io/?p=ns.robur.io.git;a=summary">ns.robur.io</a> and the <a href="https://github.com/roburio/udns/blob/master/mirage/examples">examples</a> for more details. The shared secrets are provided as boot parameter of the unikernel.</p>
|
||||||
[ns.robur.io](https://git.robur.io/?p=ns.robur.io.git;a=summary) and the [examples](https://github.com/roburio/udns/blob/master/mirage/examples) for more details. The shared secrets are provided as boot parameter of the unikernel.
|
<p>I hacked maker's
|
||||||
|
<a href="https://github.com/hannesm/ocaml-letsencrypt/tree/nsupdate">ocaml-letsencrypt</a>
|
||||||
I hacked maker's
|
|
||||||
[ocaml-letsencrypt](https://github.com/hannesm/ocaml-letsencrypt/tree/nsupdate)
|
|
||||||
library to use µDNS and sending update frames to the given IP address. I
|
library to use µDNS and sending update frames to the given IP address. I
|
||||||
already used this to have letsencrypt issue various certificates for my domains.
|
already used this to have letsencrypt issue various certificates for my domains.</p>
|
||||||
|
<p>There is no persistent storage of updates yet, but this can be realised by
|
||||||
There is no persistent storage of updates yet, but this can be realised by
|
|
||||||
implementing a secondary (which is notified on update) that writes every new
|
implementing a secondary (which is notified on update) that writes every new
|
||||||
zone to persistent storage (e.g. [disk](https://github.com/mirage/mirage-block)
|
zone to persistent storage (e.g. <a href="https://github.com/mirage/mirage-block">disk</a>
|
||||||
or [git](https://github.com/mirage/ocaml-git)). I also plan to have an
|
or <a href="https://github.com/mirage/ocaml-git">git</a>). I also plan to have an
|
||||||
automated Let's Encrypt certificate unikernel which listens for certificate
|
automated Let's Encrypt certificate unikernel which listens for certificate
|
||||||
signing requests and stores signed certificates in DNS. Luckily the year only
|
signing requests and stores signed certificates in DNS. Luckily the year only
|
||||||
started and there's plenty of time left.
|
started and there's plenty of time left.</p>
|
||||||
|
<p>I'm interested in feedback, either via <strike><a href="https://twitter.com/h4nnes">twitter</a></strike>
|
||||||
I'm interested in feedback, either via <strike>[twitter](https://twitter.com/h4nnes)</strike>
|
hannesm@mastodon.social or via eMail.</p>
|
||||||
hannesm@mastodon.social or via eMail.
|
</article></div></div></main></body></html>
|
||||||
148
Posts/Deploy
148
Posts/Deploy
|
|
@ -1,89 +1,59 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Deploying binary MirageOS unikernels
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying binary MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying binary MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2021-06-30 (last updated: 2023-11-20)</span><article><h2 id="introduction">Introduction</h2>
|
||||||
author: hannes
|
<p>MirageOS development focus has been a lot on tooling and the developer experience, but to accomplish <a href="https://robur.coop">our</a> goal to "get MirageOS into production", we need to lower the barrier. This means for us to release binary unikernels. As described <a href="/Posts/NGI">earlier</a>, we received a grant for "Deploying MirageOS" from <a href="https://pointer.ngi.eu">NGI Pointer</a> to work on the required infrastructure. This is joint work with <a href="https://reynir.dk/">Reynir</a>.</p>
|
||||||
tags: mirageos, deployment
|
<p>We provide at <a href="https://builds.robur.coop">builds.robur.coop</a> binary unikernel images (and supplementary software). Doing binary releases of MirageOS unikernels is challenging in two aspects: firstly to be useful for everyone, a binary unikernel should not contain any configuration (such as private keys, certificates, etc.). Secondly, the binaries should be <a href="https://reproducible-builds.org">reproducible</a>. This is crucial for security; everyone can reproduce the exact same binary and verify that our build service did only use the sources. No malware or backdoors included.</p>
|
||||||
abstract: Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder
|
<p>This post describes how you can deploy MirageOS unikernels without compiling it from source, then dives into the two issues outlined above - configuration and reproducibility - and finally describes how to setup your own reproducible build infrastructure for MirageOS, and how to bootstrap it.</p>
|
||||||
---
|
<h2 id="deploying-mirageos-unikernels-from-binary">Deploying MirageOS unikernels from binary</h2>
|
||||||
|
<p>To execute a MirageOS unikernel, apart from a hypervisor (Xen/KVM/Muen), a tender (responsible for allocating host system resources and passing these to the unikernel) is needed. Using virtio, this is conventionally done with qemu on Linux, but its code size (and attack surface) is huge. For MirageOS, we develop <a href="https://github.com/solo5/solo5">Solo5</a>, a minimal tender. It supports <em>hvt</em> - hardware virtualization (Linux KVM, FreeBSD BHyve, OpenBSD VMM), <em>spt</em> - sandboxed process (a tight seccomp ruleset (only a handful of system calls allowed, no hardware virtualization needed), Linux only). Apart from that, <a href="https://muen.sk"><em>muen</em></a> (a hypervisor developed in Ada), <em>virtio</em> (for some cloud deployments), and <em>xen</em> (PVHv2 or Qubes 4.0) - <a href="https://github.com/Solo5/solo5/blob/master/docs/building.md">read more</a>. We deploy our unikernels as hvt with FreeBSD BHyve as hypervisor.</p>
|
||||||
## Introduction
|
<p>On <a href="https://builds.robur.coop">builds.robur.coop</a>, next to the unikernel images, <a href="https://builds.robur.coop/job/solo5-hvt/"><em>solo5-hvt</em> packages</a> are provided - download the binary and install it. A <a href="https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/solo5">NixOS package</a> is already available - please note that <a href="https://github.com/Solo5/solo5/pull/494">soon</a> packaging will be much easier (and we will work on packages merged into distributions).</p>
|
||||||
|
<p>When the tender is installed, download a unikernel image (e.g. the <a href="https://builds.robur.coop/job/traceroute/build/latest/">traceroute</a> described in <a href="/Posts/Traceroute">an earlier post</a>), and execute it:</p>
|
||||||
MirageOS development focus has been a lot on tooling and the developer experience, but to accomplish [our](https://robur.coop) goal to "get MirageOS into production", we need to lower the barrier. This means for us to release binary unikernels. As described [earlier](/Posts/NGI), we received a grant for "Deploying MirageOS" from [NGI Pointer](https://pointer.ngi.eu) to work on the required infrastructure. This is joint work with [Reynir](https://reynir.dk/).
|
<pre><code>$ solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1
|
||||||
|
</code></pre>
|
||||||
We provide at [builds.robur.coop](https://builds.robur.coop) binary unikernel images (and supplementary software). Doing binary releases of MirageOS unikernels is challenging in two aspects: firstly to be useful for everyone, a binary unikernel should not contain any configuration (such as private keys, certificates, etc.). Secondly, the binaries should be [reproducible](https://reproducible-builds.org). This is crucial for security; everyone can reproduce the exact same binary and verify that our build service did only use the sources. No malware or backdoors included.
|
<p>If you plan to orchestrate MirageOS unikernels, you may be interested in <a href="https://github.com/roburio/albatross">albatross</a> - we provide <a href="https://builds.robur.coop/job/albatross/">binary packages as well for albatross</a>. An upcoming post will go into further details of how to setup albatross.</p>
|
||||||
|
<h2 id="mirageos-configuration">MirageOS configuration</h2>
|
||||||
This post describes how you can deploy MirageOS unikernels without compiling it from source, then dives into the two issues outlined above - configuration and reproducibility - and finally describes how to setup your own reproducible build infrastructure for MirageOS, and how to bootstrap it.
|
<p>A MirageOS unikernel has a specific purpose - composed of OCaml libraries - selected at compile time, which allows to only embed the required pieces. This reduces the attack surface drastically. At the same time, to be widely useful to multiple organisations, no configuration data must be embedded into the unikernel.</p>
|
||||||
|
<p>Early MirageOS unikernels such as <a href="https://github.com/mirage/mirage-www">mirage-www</a> embed content (blog posts, ..) and TLS certificates and private keys in the binary (using <a href="https://github.com/mirage/ocaml-crunch">crunch</a>). The <a href="https://github.com/mirage/qubes-mirage-firewall">Qubes firewall</a> (read the <a href="http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/">blog post by Thomas</a> for more information) used to include the firewall rules until <a href="https://github.com/mirage/qubes-mirage-firewall/releases/tag/v0.6">v0.6</a> in the binary, since <a href="https://github.com/mirage/qubes-mirage-firewall/tree/v0.7">v0.7</a> the rules are read dynamically from QubesDB. This is big usability improvement.</p>
|
||||||
## Deploying MirageOS unikernels from binary
|
<p>We have several possibilities to provide configuration information in MirageOS, on the one hand via boot parameters (can be pre-filled at development time, and further refined at configuration time, but those passed at boot time take precedence). Boot parameters have a length limitation.</p>
|
||||||
|
<p>Another option is to <a href="https://github.com/roburio/tlstunnel/">use a block device</a> - where the TLS reverse proxy stores the configuration, modifiable via a TCP control socket (authentication using a shared hmac secret).</p>
|
||||||
To execute a MirageOS unikernel, apart from a hypervisor (Xen/KVM/Muen), a tender (responsible for allocating host system resources and passing these to the unikernel) is needed. Using virtio, this is conventionally done with qemu on Linux, but its code size (and attack surface) is huge. For MirageOS, we develop [Solo5](https://github.com/solo5/solo5), a minimal tender. It supports *hvt* - hardware virtualization (Linux KVM, FreeBSD BHyve, OpenBSD VMM), *spt* - sandboxed process (a tight seccomp ruleset (only a handful of system calls allowed, no hardware virtualization needed), Linux only). Apart from that, [*muen*](https://muen.sk) (a hypervisor developed in Ada), *virtio* (for some cloud deployments), and *xen* (PVHv2 or Qubes 4.0) - [read more](https://github.com/Solo5/solo5/blob/master/docs/building.md). We deploy our unikernels as hvt with FreeBSD BHyve as hypervisor.
|
<p>Several other unikernels, such as <a href="https://github.com/Engil/Canopy">this website</a> and <a href="https://github.com/roburio/caldav">our CalDAV server</a>, store the content in a remote git repository. The git URI and credentials (private key seed, host key fingerprint) are passed via boot parameter.</p>
|
||||||
|
<p>Finally, another option that we take advantage of is to introduce a post-link step that rewrites the binary to embed configuration. The tool <a href="https://github.com/dinosaure/caravan">caravan</a> developed by Romain that does this rewrite is used by our <a href="https://github.com/roburio/openvpn/tree/robur/mirage-router">openvpn router</a> (<a href="https://builds.robur.coop/job/openvpn-router/build/latest/">binary</a>).</p>
|
||||||
On [builds.robur.coop](https://builds.robur.coop), next to the unikernel images, [*solo5-hvt* packages](https://builds.robur.coop/job/solo5-hvt/) are provided - download the binary and install it. A [NixOS package](https://github.com/NixOS/nixpkgs/tree/master/pkgs/os-specific/solo5) is already available - please note that [soon](https://github.com/Solo5/solo5/pull/494) packaging will be much easier (and we will work on packages merged into distributions).
|
<p>In the future, some configuration information - such as monitoring system, syslog sink, IP addresses - may be done via DHCP on one of the private network interfaces - this would mean that the DHCP server has some global configuration option, and the unikernels no longer require that many boot parameters. Another option we want to investigate is where the tender shares a file as read-only memory-mapped region from the host system to the guest system - but this is tricky considering all targets above (especially virtio and muen).</p>
|
||||||
|
<h2 id="behind-the-scenes-reproducible-builds">Behind the scenes: reproducible builds</h2>
|
||||||
When the tender is installed, download a unikernel image (e.g. the [traceroute](https://builds.robur.coop/job/traceroute/build/latest/) described in [an earlier post](/Posts/Traceroute)), and execute it:
|
<p>To provide a high level of assurance and trust, if you distribute binaries in 2021, you should have a recipe how they can be reproduced in a bit-by-bit identical way. This way, different organisations can run builders and rebuilders, and a user can decide to only use a binary if it has been reproduced by multiple organisations in different jurisdictions using different physical machines - to avoid malware being embedded in the binary.</p>
|
||||||
|
<p>For a reproduction to be successful, you need to collect the checksums of all sources that contributed to the built, together with other things (host system packages, environment variables, etc.). Of course, you can record the entire OS and sources as a tarball (or file system snapshot) and distribute that - but this may be suboptimal in terms of bandwidth requirements.</p>
|
||||||
```
|
<p>With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the <code>opam switch export</code> includes <a href="https://github.com/ocaml/opam/pull/4040">extra-files (patches)</a> and <a href="https://github.com/ocaml/opam/pull/4055">records the VCS version</a>. Based on this functionality, <a href="https://github.com/roburio/orb">orb</a>, an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are <code>PATH</code> (sanitized with an allow list, <code>/bin</code>, <code>/sbin</code>, with <code>/usr</code>, <code>/usr/local</code>, and <code>/opt</code> prefixes), and <code>HOME</code>. To enable Debian's <code>apt</code> to install packages, <code>DEBIAN_FRONTEND</code> is set to <code>noninteractive</code>. The <code>SWITCH_PATH</code> is recorded to allow orb to use the same path during a rebuild. The <code>SOURCE_DATE_EPOCH</code> is set to enable tools that record a timestamp to use a static one. The <code>OS*</code> variables are only used for recording the host OS and version.</p>
|
||||||
$ solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1
|
<p>The goal of reproducible builds can certainly be achieved in several ways, including to store all sources and used executables in a huge tarball (or docker container), which is preserved for rebuilders. The question of minimal trusted computing base and how such a container could be rebuild from sources in reproducible way are open.</p>
|
||||||
```
|
<p>The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe.</p>
|
||||||
|
<p>For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. <em>hvt</em>), and we occasionally development versions of opam packages, we use <a href="https://git.robur.coop/robur/unikernel-repo">the unikernel-repo</a> as overlay.</p>
|
||||||
If you plan to orchestrate MirageOS unikernels, you may be interested in [albatross](https://github.com/roburio/albatross) - we provide [binary packages as well for albatross](https://builds.robur.coop/job/albatross/). An upcoming post will go into further details of how to setup albatross.
|
<p>If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes.</p>
|
||||||
|
<p>Each unikernel (and non-unikernel) job (e.g. <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/">dns-primary</a> outputs some artifacts:</p>
|
||||||
## MirageOS configuration
|
<ul>
|
||||||
|
<li>the <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/f/bin/primary_git.hvt">binary image</a> (in <code>bin/</code>, unikernel image, OS package)
|
||||||
A MirageOS unikernel has a specific purpose - composed of OCaml libraries - selected at compile time, which allows to only embed the required pieces. This reduces the attack surface drastically. At the same time, to be widely useful to multiple organisations, no configuration data must be embedded into the unikernel.
|
</li>
|
||||||
|
<li>the <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/f/build-environment"><code>build-environment</code></a> containing the environment variables used for this build
|
||||||
Early MirageOS unikernels such as [mirage-www](https://github.com/mirage/mirage-www) embed content (blog posts, ..) and TLS certificates and private keys in the binary (using [crunch](https://github.com/mirage/ocaml-crunch)). The [Qubes firewall](https://github.com/mirage/qubes-mirage-firewall) (read the [blog post by Thomas](http://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/) for more information) used to include the firewall rules until [v0.6](https://github.com/mirage/qubes-mirage-firewall/releases/tag/v0.6) in the binary, since [v0.7](https://github.com/mirage/qubes-mirage-firewall/tree/v0.7) the rules are read dynamically from QubesDB. This is big usability improvement.
|
</li>
|
||||||
|
<li>the <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/f/system-packages"><code>system-packages</code></a> containing all packages installed on the host system
|
||||||
We have several possibilities to provide configuration information in MirageOS, on the one hand via boot parameters (can be pre-filled at development time, and further refined at configuration time, but those passed at boot time take precedence). Boot parameters have a length limitation.
|
</li>
|
||||||
|
<li>the <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/f/opam-switch"><code>opam-switch</code></a> that contains all opam packages, including git commit or tarball with checksum, and potentially extra patches, used for this build
|
||||||
Another option is to [use a block device](https://github.com/roburio/tlstunnel/) - where the TLS reverse proxy stores the configuration, modifiable via a TCP control socket (authentication using a shared hmac secret).
|
</li>
|
||||||
|
<li>a job script and console output
|
||||||
Several other unikernels, such as [this website](https://github.com/Engil/Canopy) and [our CalDAV server](https://github.com/roburio/caldav), store the content in a remote git repository. The git URI and credentials (private key seed, host key fingerprint) are passed via boot parameter.
|
</li>
|
||||||
|
</ul>
|
||||||
Finally, another option that we take advantage of is to introduce a post-link step that rewrites the binary to embed configuration. The tool [caravan](https://github.com/dinosaure/caravan) developed by Romain that does this rewrite is used by our [openvpn router](https://github.com/roburio/openvpn/tree/robur/mirage-router) ([binary](https://builds.robur.coop/job/openvpn-router/build/latest/)).
|
<p>To reproduce such a built, you need to get the same operating system (OS, OS_FAMILY, OS_DISTRIBUTION, OS_VERSION in build-environment), the same set of system packages, and then you can <code>orb rebuild</code> which sets the environment variables and installs the opam packages from the opam-switch.</p>
|
||||||
|
<p>You can <a href="https://builds.robur.coop/job/dns-primary-git/">browse</a> the different builds, and if there are checksum changes, you can browse to a diff between the opam switches to reason whether the checksum change was intentional (e.g. <a href="https://builds.robur.coop/compare/ba9ab091-9400-4e8d-ad37-cf1339114df8/23341f6b-cd26-48ab-9383-e71342455e81/opam-switch">here</a> the checksum of the unikernel changed when the x509 library was updated).</p>
|
||||||
In the future, some configuration information - such as monitoring system, syslog sink, IP addresses - may be done via DHCP on one of the private network interfaces - this would mean that the DHCP server has some global configuration option, and the unikernels no longer require that many boot parameters. Another option we want to investigate is where the tender shares a file as read-only memory-mapped region from the host system to the guest system - but this is tricky considering all targets above (especially virtio and muen).
|
<p>The opam reproducible build infrastructure is driven by:</p>
|
||||||
|
<ul>
|
||||||
## Behind the scenes: reproducible builds
|
<li><a href="https://github.com/roburio/orb">orb</a> conducting reproducible builds (<a href="https://builds.robur.coop/job/orb/">packages</a>)
|
||||||
|
</li>
|
||||||
To provide a high level of assurance and trust, if you distribute binaries in 2021, you should have a recipe how they can be reproduced in a bit-by-bit identical way. This way, different organisations can run builders and rebuilders, and a user can decide to only use a binary if it has been reproduced by multiple organisations in different jurisdictions using different physical machines - to avoid malware being embedded in the binary.
|
<li><a href="https://github.com/roburio/builder">builder</a> scheduling builds in contained environments (<a href="https://builds.robur.coop/job/builder/">packages</a>)
|
||||||
|
</li>
|
||||||
For a reproduction to be successful, you need to collect the checksums of all sources that contributed to the built, together with other things (host system packages, environment variables, etc.). Of course, you can record the entire OS and sources as a tarball (or file system snapshot) and distribute that - but this may be suboptimal in terms of bandwidth requirements.
|
<li><a href="https://git.robur.coop/robur/builder-web">builder-web</a> storing builds in a database and providing a HTTP interface (<a href="https://builds.robur.coop/job/builder-web/">packages</a>)
|
||||||
|
</li>
|
||||||
With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the `opam switch export` includes [extra-files (patches)](https://github.com/ocaml/opam/pull/4040) and [records the VCS version](https://github.com/ocaml/opam/pull/4055). Based on this functionality, [orb](https://github.com/roburio/orb), an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are `PATH` (sanitized with an allow list, `/bin`, `/sbin`, with `/usr`, `/usr/local`, and `/opt` prefixes), and `HOME`. To enable Debian's `apt` to install packages, `DEBIAN_FRONTEND` is set to `noninteractive`. The `SWITCH_PATH` is recorded to allow orb to use the same path during a rebuild. The `SOURCE_DATE_EPOCH` is set to enable tools that record a timestamp to use a static one. The `OS*` variables are only used for recording the host OS and version.
|
</ul>
|
||||||
|
<p>These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.</p>
|
||||||
The goal of reproducible builds can certainly be achieved in several ways, including to store all sources and used executables in a huge tarball (or docker container), which is preserved for rebuilders. The question of minimal trusted computing base and how such a container could be rebuild from sources in reproducible way are open.
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>Thanks to NGI funding we now have reproducible MirageOS binary builds available at <a href="https://builds.robur.coop">builds.robur.coop</a>. The underlying infrastructure is reproducible, available for multiple platforms (Ubuntu using docker, FreeBSD using jails), and can be easily bootstrapped from source (once you have OCaml and opam working, getting builder and orb should be easy). All components are open source software, mostly with permissive licenses.</p>
|
||||||
The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe.
|
<p>We also have an index over sha-256 checksum of binaries - in the case you find a running unikernel image where you forgot which exact packages were used, you can do a reverse lookup.</p>
|
||||||
|
<p>We are aware that the web interface can be improved (PRs welcome). We will also work on the rebuilder setup and run some rebuilds.</p>
|
||||||
For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. *hvt*), and we occasionally development versions of opam packages, we use [the unikernel-repo](https://git.robur.coop/robur/unikernel-repo) as overlay.
|
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes.
|
|
||||||
|
|
||||||
Each unikernel (and non-unikernel) job (e.g. [dns-primary](https://builds.robur.coop/job/dns-primary-git/build/latest/) outputs some artifacts:
|
|
||||||
- the [binary image](https://builds.robur.coop/job/dns-primary-git/build/latest/f/bin/primary_git.hvt) (in `bin/`, unikernel image, OS package)
|
|
||||||
- the [`build-environment`](https://builds.robur.coop/job/dns-primary-git/build/latest/f/build-environment) containing the environment variables used for this build
|
|
||||||
- the [`system-packages`](https://builds.robur.coop/job/dns-primary-git/build/latest/f/system-packages) containing all packages installed on the host system
|
|
||||||
- the [`opam-switch`](https://builds.robur.coop/job/dns-primary-git/build/latest/f/opam-switch) that contains all opam packages, including git commit or tarball with checksum, and potentially extra patches, used for this build
|
|
||||||
- a job script and console output
|
|
||||||
|
|
||||||
To reproduce such a built, you need to get the same operating system (OS, OS_FAMILY, OS_DISTRIBUTION, OS_VERSION in build-environment), the same set of system packages, and then you can `orb rebuild` which sets the environment variables and installs the opam packages from the opam-switch.
|
|
||||||
|
|
||||||
You can [browse](https://builds.robur.coop/job/dns-primary-git/) the different builds, and if there are checksum changes, you can browse to a diff between the opam switches to reason whether the checksum change was intentional (e.g. [here](https://builds.robur.coop/compare/ba9ab091-9400-4e8d-ad37-cf1339114df8/23341f6b-cd26-48ab-9383-e71342455e81/opam-switch) the checksum of the unikernel changed when the x509 library was updated).
|
|
||||||
|
|
||||||
The opam reproducible build infrastructure is driven by:
|
|
||||||
- [orb](https://github.com/roburio/orb) conducting reproducible builds ([packages](https://builds.robur.coop/job/orb/))
|
|
||||||
- [builder](https://github.com/roburio/builder) scheduling builds in contained environments ([packages](https://builds.robur.coop/job/builder/))
|
|
||||||
- [builder-web](https://git.robur.coop/robur/builder-web) storing builds in a database and providing a HTTP interface ([packages](https://builds.robur.coop/job/builder-web/))
|
|
||||||
|
|
||||||
These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
Thanks to NGI funding we now have reproducible MirageOS binary builds available at [builds.robur.coop](https://builds.robur.coop). The underlying infrastructure is reproducible, available for multiple platforms (Ubuntu using docker, FreeBSD using jails), and can be easily bootstrapped from source (once you have OCaml and opam working, getting builder and orb should be easy). All components are open source software, mostly with permissive licenses.
|
|
||||||
|
|
||||||
We also have an index over sha-256 checksum of binaries - in the case you find a running unikernel image where you forgot which exact packages were used, you can do a reverse lookup.
|
|
||||||
|
|
||||||
We are aware that the web interface can be improved (PRs welcome). We will also work on the rebuilder setup and run some rebuilds.
|
|
||||||
|
|
||||||
Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.
|
|
||||||
|
|
||||||
349
Posts/DnsServer
349
Posts/DnsServer
|
|
@ -1,77 +1,47 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Deploying authoritative OCaml-DNS servers as MirageOS unikernels
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying authoritative OCaml-DNS servers as MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying authoritative OCaml-DNS servers as MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2019-12-23 (last updated: 2023-03-02)</span><article><h2 id="goal">Goal</h2>
|
||||||
author: hannes
|
<p>Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as <code>nqsb.io</code> and <code>robur.coop</code>. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as <a href="/Posts/DNS">µDNS</a>.</p>
|
||||||
tags: mirageos, protocol, deployment
|
<h2 id="prerequisites">Prerequisites</h2>
|
||||||
abstract: A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.
|
<p>You need to own a domain, and be able to delegate the name service to your own servers.
|
||||||
---
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as `nqsb.io` and `robur.coop`. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as [µDNS](/Posts/DNS).
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
You need to own a domain, and be able to delegate the name service to your own servers.
|
|
||||||
You also need two spare public IPv4 addresses (in different /24 networks) for your name servers.
|
You also need two spare public IPv4 addresses (in different /24 networks) for your name servers.
|
||||||
A git server or remote repository reachable via git over ssh.
|
A git server or remote repository reachable via git over ssh.
|
||||||
Servers which support [solo5](https://github.com/solo5/solo5) guests, and have the corresponding tender installed.
|
Servers which support <a href="https://github.com/solo5/solo5">solo5</a> guests, and have the corresponding tender installed.
|
||||||
A computer with [opam](https://opam.ocaml.org) (>= 2.0.0) installed.
|
A computer with <a href="https://opam.ocaml.org">opam</a> (>= 2.0.0) installed.</p>
|
||||||
|
<h2 id="data-preparation">Data preparation</h2>
|
||||||
## Data preparation
|
<p>Figure out a way to get the DNS entries of your domain in a <a href="https://tools.ietf.org/html/rfc1034">"master file format"</a>, i.e. what bind uses.</p>
|
||||||
|
<p>This is a master file for the <code>mirage</code> domain, defining <code>$ORIGIN</code> to avoid typing the domain name after each hostname (use <code>@</code> if you need the domain name only; if you need to refer to a hostname in a different domain end it with a dot (<code>.</code>), i.e. <code>ns2.foo.com.</code>). The default time to live <code>$TTL</code> is an hour (3600 seconds).
|
||||||
Figure out a way to get the DNS entries of your domain in a ["master file format"](https://tools.ietf.org/html/rfc1034), i.e. what bind uses.
|
The zone contains a <a href="https://tools.ietf.org/html/rfc1035#section-3.3.13">start of authority (<code>SOA</code>) record</a> containing the nameserver, hostmaster, serial, refresh, retry, expiry, and minimum.
|
||||||
|
Also, a single <a href="https://tools.ietf.org/html/rfc1035#section-3.3.11">name server (<code>NS</code>) record</a> <code>ns1</code> is specified with an accompanying <a href="https://tools.ietf.org/html/rfc1035#section-3.4.1">address (<code>A</code>) records</a> pointing to their IPv4 address.</p>
|
||||||
This is a master file for the `mirage` domain, defining `$ORIGIN` to avoid typing the domain name after each hostname (use `@` if you need the domain name only; if you need to refer to a hostname in a different domain end it with a dot (`.`), i.e. `ns2.foo.com.`). The default time to live `$TTL` is an hour (3600 seconds).
|
<pre><code class="language-shell">git-repo> cat mirage
|
||||||
The zone contains a [start of authority (`SOA`) record](https://tools.ietf.org/html/rfc1035#section-3.3.13) containing the nameserver, hostmaster, serial, refresh, retry, expiry, and minimum.
|
|
||||||
Also, a single [name server (`NS`) record](https://tools.ietf.org/html/rfc1035#section-3.3.11) `ns1` is specified with an accompanying [address (`A`) records](https://tools.ietf.org/html/rfc1035#section-3.4.1) pointing to their IPv4 address.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
git-repo> cat mirage
|
|
||||||
$ORIGIN mirage.
|
$ORIGIN mirage.
|
||||||
$TTL 3600
|
$TTL 3600
|
||||||
@ SOA ns1 hostmaster 1 86400 7200 1048576 3600
|
@ SOA ns1 hostmaster 1 86400 7200 1048576 3600
|
||||||
@ NS ns1
|
@ NS ns1
|
||||||
ns1 A 127.0.0.1
|
ns1 A 127.0.0.1
|
||||||
www A 1.1.1.1
|
www A 1.1.1.1
|
||||||
git-repo> git add mirage && git commit -m initial && git push
|
git-repo> git add mirage && git commit -m initial && git push
|
||||||
```
|
</code></pre>
|
||||||
|
<h2 id="installation">Installation</h2>
|
||||||
## Installation
|
<p>On your development machine, you need to install various OCaml packages. You don't need privileged access if common tools (C compiler, make, libgmp) are already installed. You have <code>opam</code> installed.</p>
|
||||||
|
<p>Let's create a fresh <code>switch</code> for the DNS journey:</p>
|
||||||
On your development machine, you need to install various OCaml packages. You don't need privileged access if common tools (C compiler, make, libgmp) are already installed. You have `opam` installed.
|
<pre><code class="language-shell">$ opam init
|
||||||
|
|
||||||
Let's create a fresh `switch` for the DNS journey:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ opam init
|
|
||||||
$ opam update
|
$ opam update
|
||||||
$ opam switch create udns 4.14.1
|
$ opam switch create udns 4.14.1
|
||||||
# waiting a bit, a fresh OCaml compiler is getting bootstrapped
|
# waiting a bit, a fresh OCaml compiler is getting bootstrapped
|
||||||
$ eval `opam env` #sets some environment variables
|
$ eval `opam env` #sets some environment variables
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The last command set environment variables in your current shell session, please use the same shell for the commands following (or run <code>eval $(opam env)</code> in another shell and proceed in there - the output of <code>opam switch</code> sohuld point to <code>udns</code>).</p>
|
||||||
The last command set environment variables in your current shell session, please use the same shell for the commands following (or run `eval $(opam env)` in another shell and proceed in there - the output of `opam switch` sohuld point to `udns`).
|
<h3 id="validation-of-our-zonefile">Validation of our zonefile</h3>
|
||||||
|
<p>First let's check that OCaml-DNS can parse our zonefile:</p>
|
||||||
### Validation of our zonefile
|
<pre><code class="language-shell">$ opam install dns-cli #installs ~/.opam/udns/bin/ozone and other binaries
|
||||||
|
$ ozone <git-repo>/mirage # see ozone --help
|
||||||
First let's check that OCaml-DNS can parse our zonefile:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ opam install dns-cli #installs ~/.opam/udns/bin/ozone and other binaries
|
|
||||||
$ ozone <git-repo>/mirage # see ozone --help
|
|
||||||
successfully checked zone
|
successfully checked zone
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Great. Error reporting is not great, but line numbers are indicated (<code>ozone: zone parse problem at line 3: syntax error</code>), <a href="https://github.com/mirage/ocaml-dns/tree/v4.2.0/zone">lexer and parser are lex/yacc style</a> (PRs welcome).</p>
|
||||||
Great. Error reporting is not great, but line numbers are indicated (`ozone: zone parse problem at line 3: syntax error`), [lexer and parser are lex/yacc style](https://github.com/mirage/ocaml-dns/tree/v4.2.0/zone) (PRs welcome).
|
<p>FWIW, <code>ozone</code> accepts <code>--old <filename></code> to check whether an update from the old zone to the new is fine. This can be used as <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">pre-commit hook</a> in your git repository to avoid bad parse states in your name servers.</p>
|
||||||
|
<h3 id="getting-the-primary-up">Getting the primary up</h3>
|
||||||
FWIW, `ozone` accepts `--old <filename>` to check whether an update from the old zone to the new is fine. This can be used as [pre-commit hook](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) in your git repository to avoid bad parse states in your name servers.
|
<p>The next step is to compile the primary server and run it to serve the domain data. Since the git-via-ssh client is not yet released, we need to add a custom opam repository to this switch.</p>
|
||||||
|
<pre><code class="language-shell"># get the `mirage` application via opam
|
||||||
### Getting the primary up
|
|
||||||
|
|
||||||
The next step is to compile the primary server and run it to serve the domain data. Since the git-via-ssh client is not yet released, we need to add a custom opam repository to this switch.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# get the `mirage` application via opam
|
|
||||||
$ opam install lwt mirage
|
$ opam install lwt mirage
|
||||||
|
|
||||||
# get the source code of the unikernels
|
# get the source code of the unikernels
|
||||||
|
|
@ -98,19 +68,14 @@ $ host ns1.mirage 127.0.0.1
|
||||||
ns1.mirage has address 127.0.0.1
|
ns1.mirage has address 127.0.0.1
|
||||||
$ dig any mirage @127.0.0.1
|
$ dig any mirage @127.0.0.1
|
||||||
# a DNS packet printout with all records available for mirage
|
# a DNS packet printout with all records available for mirage
|
||||||
```
|
</code></pre>
|
||||||
|
<p>That's exciting, the DNS server serving answers from a remote git repository.</p>
|
||||||
That's exciting, the DNS server serving answers from a remote git repository.
|
<h3 id="securing-the-git-access-with-ssh">Securing the git access with ssh</h3>
|
||||||
|
<p>Let's authenticate the access by using ssh, so we feel ready to push data there as well. The primary-git unikernel already includes an experimental <a href="https://github.com/haesbaert/awa-ssh">ssh client</a>, all we need to do is setting up credentials - in the following a RSA keypair and the server fingerprint.</p>
|
||||||
### Securing the git access with ssh
|
<pre><code class="language-shell"># collect the RSA host key fingerprint
|
||||||
|
$ ssh-keyscan <git-server> > /tmp/git-server-public-keys
|
||||||
Let's authenticate the access by using ssh, so we feel ready to push data there as well. The primary-git unikernel already includes an experimental [ssh client](https://github.com/haesbaert/awa-ssh), all we need to do is setting up credentials - in the following a RSA keypair and the server fingerprint.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# collect the RSA host key fingerprint
|
|
||||||
$ ssh-keyscan <git-server> > /tmp/git-server-public-keys
|
|
||||||
$ ssh-keygen -l -E sha256 -f /tmp/git-server-public-keys | grep ED25519
|
$ ssh-keygen -l -E sha256 -f /tmp/git-server-public-keys | grep ED25519
|
||||||
256 SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBaD0 <git-server> (ED25519)
|
256 SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBaD0 <git-server> (ED25519)
|
||||||
# we're interested in the SHA256:yyy only
|
# we're interested in the SHA256:yyy only
|
||||||
|
|
||||||
# generate a ssh keypair
|
# generate a ssh keypair
|
||||||
|
|
@ -118,46 +83,30 @@ $ awa_gen_key --keytype ed25510 # installed by the make step above in ~/.opam/ud
|
||||||
private key: ed25519:nO7ervdJqzPfuvdM/J4qImipwVoI5gl53fpqgjZnv9w=
|
private key: ed25519:nO7ervdJqzPfuvdM/J4qImipwVoI5gl53fpqgjZnv9w=
|
||||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyliWbwWWXBc1+DRIzReLQ4UFiVGXJ6Paw1Jts+XQte awa@awa.local
|
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyliWbwWWXBc1+DRIzReLQ4UFiVGXJ6Paw1Jts+XQte awa@awa.local
|
||||||
# please run your own awa_gen_key, don't use the numbers above
|
# please run your own awa_gen_key, don't use the numbers above
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The public key needs is in standard OpenSSH format and needs to be added to the list of accepted keys on your server - the exact steps depend on your git server, if you're running your own with <a href="https://github.com/tv42/gitosis">gitosis</a>, add it as new public key file and grant that key access to the data repository. If you use gitlab or github, you may want to create a new user account and with the generated key.</p>
|
||||||
The public key needs is in standard OpenSSH format and needs to be added to the list of accepted keys on your server - the exact steps depend on your git server, if you're running your own with [gitosis](https://github.com/tv42/gitosis), add it as new public key file and grant that key access to the data repository. If you use gitlab or github, you may want to create a new user account and with the generated key.
|
<p>The private key is not displayed, but only the seed required to re-generate it, when using the same random number generator, in our case <a href="https://mirage.github.io/mirage-crypto/doc/mirage-crypto-rng/Mirage_crypto_rng/index.html">fortuna implemented by mirage-crypto</a> - used by both <code>awa_gen_key</code> and <code>primary-git</code>. The seed is provided as command-line argument while starting <code>primary-git</code>:</p>
|
||||||
|
<pre><code class="language-shell"># execute with git over ssh, authenticator from ssh-keyscan, seed from awa_gen_key
|
||||||
The private key is not displayed, but only the seed required to re-generate it, when using the same random number generator, in our case [fortuna implemented by mirage-crypto](https://mirage.github.io/mirage-crypto/doc/mirage-crypto-rng/Mirage_crypto_rng/index.html) - used by both `awa_gen_key` and `primary-git`. The seed is provided as command-line argument while starting `primary-git`:
|
$ ./primary-git --authenticator=SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBaD0 --ssh-key=ed25519:nO7ervdJqzPfuvdM/J4qImipwVoI5gl53fpqgjZnv9w= --remote=git@<git-server>:repo-name.git
|
||||||
|
|
||||||
```shell
|
|
||||||
# execute with git over ssh, authenticator from ssh-keyscan, seed from awa_gen_key
|
|
||||||
$ ./primary-git --authenticator=SHA256:a5kkkuo7MwTBkW+HDt4km0gGPUAX0y1bFcPMXKxBaD0 --ssh-key=ed25519:nO7ervdJqzPfuvdM/J4qImipwVoI5gl53fpqgjZnv9w= --remote=git@<git-server>:repo-name.git
|
|
||||||
# started up, you can try the host and dig commands from above if you like
|
# started up, you can try the host and dig commands from above if you like
|
||||||
```
|
</code></pre>
|
||||||
|
<p>To wrap up, we now have a primary authoritative name server for our zone running as Unix process, which clones a remote git repository via ssh on startup and then serves it.</p>
|
||||||
To wrap up, we now have a primary authoritative name server for our zone running as Unix process, which clones a remote git repository via ssh on startup and then serves it.
|
<h3 id="authenticated-data-updates">Authenticated data updates</h3>
|
||||||
|
<p>Our remote git repository is the source of truth, if you need to add a DNS entry to the zone, you git pull, edit the zone file, remember to increase the serial in the SOA line, run <code>ozone</code>, git commit and push to the repository.</p>
|
||||||
### Authenticated data updates
|
<p>So, the <code>primary-git</code> needs to be informed of git pushes. This requires a communication channel from the git server (or somewhere else, e.g. your laptop) to the DNS server. I prefer in-protocol solutions over adding yet another protocol stack, no way my DNS server will talk HTTP REST.</p>
|
||||||
|
<p>The DNS protocol has an extension for <a href="https://tools.ietf.org/html/rfc1996">notifications of zone changes</a> (as a DNS packet), usually used between the primary and secondary servers. The <code>primary-git</code> accepts these notify requests (i.e. bends the standard slightly), and upon receival pulls the remote git repository, and serves the fresh zone files. Since a git pull may be rather excessive in terms of CPU cycles and network bandwidth, only authenticated notifications are accepted.</p>
|
||||||
Our remote git repository is the source of truth, if you need to add a DNS entry to the zone, you git pull, edit the zone file, remember to increase the serial in the SOA line, run `ozone`, git commit and push to the repository.
|
<p>The DNS protocol specifies in another extension <a href="https://tools.ietf.org/html/rfc2845">authentication (DNS TSIG)</a> with transaction signatures on DNS packets including a timestamp and fudge to avoid replay attacks. As key material hmac secrets distribued to both the communication endpoints are used.</p>
|
||||||
|
<p>To recap, the primary server is configured with command line parameters (for remote repository url and ssh credentials), and serves data from a zonefile. If the secrets would be provided via command line, a restart would be necessary for adding and removing keys. If put into the zonefile, they would be publicly served on request. So instead, we'll use another file, still in zone file format, in the top-level domain <code>_keys</code>, i.e. the <code>mirage._keys</code> file contains keys for the <code>mirage</code> zone. All files ending in <code>._keys</code> are parsed with the normal parser, but put into an authentication store instead of the domain data store, which is served publically.</p>
|
||||||
So, the `primary-git` needs to be informed of git pushes. This requires a communication channel from the git server (or somewhere else, e.g. your laptop) to the DNS server. I prefer in-protocol solutions over adding yet another protocol stack, no way my DNS server will talk HTTP REST.
|
<p>For encoding hmac secrets into DNS zone file format, the <a href="https://tools.ietf.org/html/rfc4034#section-2"><code>DNSKEY</code></a> format is used (designed for DNSsec). The <a href="https://www.isc.org/bind/">bind</a> software comes with <code>dnssec-keygen</code> and <code>tsig-keygen</code> to generate DNSKEY output: flags is 0, protocol is 3, and algorithm identifier for SHA256 is 163 (SHA384 164, SHA512 165). This is reused by the OCaml DNS library. The key material itself is base64 encoded.</p>
|
||||||
|
<p>Access control and naming of keys follows the DNS domain name hierarchy - a key has the form name._operation.domain, and has access granted to domain and all subdomains of it. Two operations are supported: update and transfer. In the future there may be a dedicated notify operation, for now we'll use update. The name part is ignored for the update operation.</p>
|
||||||
The DNS protocol has an extension for [notifications of zone changes](https://tools.ietf.org/html/rfc1996) (as a DNS packet), usually used between the primary and secondary servers. The `primary-git` accepts these notify requests (i.e. bends the standard slightly), and upon receival pulls the remote git repository, and serves the fresh zone files. Since a git pull may be rather excessive in terms of CPU cycles and network bandwidth, only authenticated notifications are accepted.
|
<p>Since we now embedd secret information in the git repository, it is a good idea to restrict access to it, i.e. make it private and not publicly cloneable or viewable. Let's generate a first hmac secret and send a notify:</p>
|
||||||
|
<pre><code class="language-shell">$ dd if=/dev/random bs=1 count=32 | b64encode -
|
||||||
The DNS protocol specifies in another extension [authentication (DNS TSIG)](https://tools.ietf.org/html/rfc2845) with transaction signatures on DNS packets including a timestamp and fudge to avoid replay attacks. As key material hmac secrets distribued to both the communication endpoints are used.
|
|
||||||
|
|
||||||
To recap, the primary server is configured with command line parameters (for remote repository url and ssh credentials), and serves data from a zonefile. If the secrets would be provided via command line, a restart would be necessary for adding and removing keys. If put into the zonefile, they would be publicly served on request. So instead, we'll use another file, still in zone file format, in the top-level domain `_keys`, i.e. the `mirage._keys` file contains keys for the `mirage` zone. All files ending in `._keys` are parsed with the normal parser, but put into an authentication store instead of the domain data store, which is served publically.
|
|
||||||
|
|
||||||
For encoding hmac secrets into DNS zone file format, the [`DNSKEY`](https://tools.ietf.org/html/rfc4034#section-2) format is used (designed for DNSsec). The [bind](https://www.isc.org/bind/) software comes with `dnssec-keygen` and `tsig-keygen` to generate DNSKEY output: flags is 0, protocol is 3, and algorithm identifier for SHA256 is 163 (SHA384 164, SHA512 165). This is reused by the OCaml DNS library. The key material itself is base64 encoded.
|
|
||||||
|
|
||||||
Access control and naming of keys follows the DNS domain name hierarchy - a key has the form name._operation.domain, and has access granted to domain and all subdomains of it. Two operations are supported: update and transfer. In the future there may be a dedicated notify operation, for now we'll use update. The name part is ignored for the update operation.
|
|
||||||
|
|
||||||
Since we now embedd secret information in the git repository, it is a good idea to restrict access to it, i.e. make it private and not publicly cloneable or viewable. Let's generate a first hmac secret and send a notify:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ dd if=/dev/random bs=1 count=32 | b64encode -
|
|
||||||
begin-base64 644 -
|
begin-base64 644 -
|
||||||
kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
||||||
====
|
====
|
||||||
[..]
|
[..]
|
||||||
git-repo> echo "personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=" > mirage._keys
|
git-repo> echo "personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=" > mirage._keys
|
||||||
git-repo> git add mirage._keys && git commit -m "add hmac secret" && git push
|
git-repo> git add mirage._keys && git commit -m "add hmac secret" && git push
|
||||||
|
|
||||||
# now we need to restart the primary git to get the git repository with the key
|
# now we need to restart the primary git to get the git repository with the key
|
||||||
$ ./primary-git --ssh-key=... # arguments from above, remote git, host key fingerprint, private key seed
|
$ ./primary-git --ssh-key=... # arguments from above, remote git, host key fingerprint, private key seed
|
||||||
|
|
@ -166,43 +115,28 @@ $ ./primary-git --ssh-key=... # arguments from above, remote git, host key finge
|
||||||
$ onotify 127.0.0.1 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
$ onotify 127.0.0.1 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
||||||
# onotify was installed by dns-cli in ~/.opam/udns/bin/onotify, see --help for options
|
# onotify was installed by dns-cli in ~/.opam/udns/bin/onotify, see --help for options
|
||||||
# further changes to the hmac secrets don't require a restart anymore, a notify packet is sufficient :D
|
# further changes to the hmac secrets don't require a restart anymore, a notify packet is sufficient :D
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Ok, this onotify command line could be setup as a git post-commit hook, or run manually after each manual git push.</p>
|
||||||
Ok, this onotify command line could be setup as a git post-commit hook, or run manually after each manual git push.
|
<h3 id="secondary">Secondary</h3>
|
||||||
|
<p>It's time to figure out how to integrate the secondary name server. An already existing bind or something else that accepts notifications and issues zone transfers with hmac-sha256 secrets should work out of the box. If you encounter interoperability issues, please get in touch with me.</p>
|
||||||
### Secondary
|
<p>The <code>secondary</code> unikernel is available from another git repository:</p>
|
||||||
|
<pre><code class="language-shell"># get the secondary sources
|
||||||
It's time to figure out how to integrate the secondary name server. An already existing bind or something else that accepts notifications and issues zone transfers with hmac-sha256 secrets should work out of the box. If you encounter interoperability issues, please get in touch with me.
|
|
||||||
|
|
||||||
The `secondary` unikernel is available from another git repository:
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# get the secondary sources
|
|
||||||
$ git clone https://github.com/roburio/dns-secondary.git
|
$ git clone https://github.com/roburio/dns-secondary.git
|
||||||
$ cd dns-secondary
|
$ cd dns-secondary
|
||||||
```
|
</code></pre>
|
||||||
|
<p>It's only command line argument is a list of hmac secrets used for authenticating that the received data originates from the primary server. Data is initially transferred by a <a href="https://tools.ietf.org/html/rfc5936">full zone transfer (AXFR)</a>, later updates (upon refresh timer or notify request sent by the primary) use <a href="https://tools.ietf.org/html/rfc1995">incremental (IXFR)</a>. Zone transfer requests and data are authenticated with transaction signatures again.</p>
|
||||||
It's only command line argument is a list of hmac secrets used for authenticating that the received data originates from the primary server. Data is initially transferred by a [full zone transfer (AXFR)](https://tools.ietf.org/html/rfc5936), later updates (upon refresh timer or notify request sent by the primary) use [incremental (IXFR)](https://tools.ietf.org/html/rfc1995). Zone transfer requests and data are authenticated with transaction signatures again.
|
<p>Convenience by OCaml DNS is that transfer key names matter, and are of the form <primary-ip>.<secondary-ip>._transfer.domain, i.e. <code>1.1.1.1.2.2.2.2._transfer.mirage</code> if the primary server is 1.1.1.1, and the secondary 2.2.2.2. Encoding the IP address in the name allows both parties to start the communication: the secondary starts by requesting a SOA for all domains for which keys are provided on command line, and if an authoritative SOA answer is received, the AXFR is triggered. The primary server emits notification requests on startup and then on every zone change (i.e. via git pull) to all secondary IP addresses of transfer keys present for the specific zone in addition to the notifications to the NS records in the zone.</p>
|
||||||
|
<pre><code class="language-shell">$ mirage configure
|
||||||
Convenience by OCaml DNS is that transfer key names matter, and are of the form <primary-ip>.<secondary-ip>._transfer.domain, i.e. `1.1.1.1.2.2.2.2._transfer.mirage` if the primary server is 1.1.1.1, and the secondary 2.2.2.2. Encoding the IP address in the name allows both parties to start the communication: the secondary starts by requesting a SOA for all domains for which keys are provided on command line, and if an authoritative SOA answer is received, the AXFR is triggered. The primary server emits notification requests on startup and then on every zone change (i.e. via git pull) to all secondary IP addresses of transfer keys present for the specific zone in addition to the notifications to the NS records in the zone.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
$ mirage configure
|
|
||||||
$ make
|
$ make
|
||||||
$ ./dist/secondary
|
$ ./dist/secondary
|
||||||
```
|
</code></pre>
|
||||||
|
<h3 id="ip-addresses-and-routing">IP addresses and routing</h3>
|
||||||
### IP addresses and routing
|
<p>Both primary and secondary serve the data on the DNS port (53) on UDP and TCP. To run both on the same machine and bind them to different IP addresses, we'll use a layer 2 network (ethernet frames) with a host system software switch (bridge interface <code>service</code>), the unikernels as virtual machines (or seccomp-sandboxed) via the <a href="https://github.com/solo5/solo5">solo5</a> backend. Using xen is possible as well. As IP address range we'll use 10.0.42.0/24, and the host system uses the 10.0.42.1.</p>
|
||||||
|
<p>The primary git needs connectivity to the remote git repository, thus on a laptop in a private network we need network address translation (NAT) from the bridge where the unikernels speak to the Internet where the git repository resides.</p>
|
||||||
Both primary and secondary serve the data on the DNS port (53) on UDP and TCP. To run both on the same machine and bind them to different IP addresses, we'll use a layer 2 network (ethernet frames) with a host system software switch (bridge interface `service`), the unikernels as virtual machines (or seccomp-sandboxed) via the [solo5](https://github.com/solo5/solo5) backend. Using xen is possible as well. As IP address range we'll use 10.0.42.0/24, and the host system uses the 10.0.42.1.
|
<pre><code class="language-shell"># on FreeBSD:
|
||||||
|
|
||||||
The primary git needs connectivity to the remote git repository, thus on a laptop in a private network we need network address translation (NAT) from the bridge where the unikernels speak to the Internet where the git repository resides.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# on FreeBSD:
|
|
||||||
# configure NAT with pf, you need to have forwarding enabled
|
# configure NAT with pf, you need to have forwarding enabled
|
||||||
$ sysctl net.inet.ip.forwarding: 1
|
$ sysctl net.inet.ip.forwarding: 1
|
||||||
$ echo 'nat pass on wlan0 inet from 10.0.42.0/24 to any -> (wlan0)' >> /etc/pf.conf
|
$ echo 'nat pass on wlan0 inet from 10.0.42.0/24 to any -> (wlan0)' >> /etc/pf.conf
|
||||||
$ service pf restart
|
$ service pf restart
|
||||||
|
|
||||||
# make tap interfaces UP on open()
|
# make tap interfaces UP on open()
|
||||||
|
|
@ -221,14 +155,10 @@ $ ifconfig tap create
|
||||||
tap1
|
tap1
|
||||||
# add them to the bridge
|
# add them to the bridge
|
||||||
$ ifconfig service addm tap0 addm tap1
|
$ ifconfig service addm tap0 addm tap1
|
||||||
```
|
</code></pre>
|
||||||
|
<h3 id="primary-and-secondary-setup">Primary and secondary setup</h3>
|
||||||
### Primary and secondary setup
|
<p>Let's update our zone slightly to reflect the IP changes.</p>
|
||||||
|
<pre><code class="language-shell">git-repo> cat mirage
|
||||||
Let's update our zone slightly to reflect the IP changes.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
git-repo> cat mirage
|
|
||||||
$ORIGIN mirage.
|
$ORIGIN mirage.
|
||||||
$TTL 3600
|
$TTL 3600
|
||||||
@ SOA ns1 hostmaster 2 86400 7200 1048576 3600
|
@ SOA ns1 hostmaster 2 86400 7200 1048576 3600
|
||||||
|
|
@ -238,16 +168,13 @@ ns1 A 10.0.42.2
|
||||||
ns2 A 10.0.42.3
|
ns2 A 10.0.42.3
|
||||||
|
|
||||||
# we also need an additional transfer key
|
# we also need an additional transfer key
|
||||||
git-repo> cat mirage._keys
|
git-repo> cat mirage._keys
|
||||||
personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
||||||
10.0.42.2.10.0.42.3._transfer.mirage. DNSKEY 0 3 163 cDK6sKyvlt8UBerZlmxuD84ih2KookJGDagJlLVNo20=
|
10.0.42.2.10.0.42.3._transfer.mirage. DNSKEY 0 3 163 cDK6sKyvlt8UBerZlmxuD84ih2KookJGDagJlLVNo20=
|
||||||
git-repo> git commit -m "updates" . && git push
|
git-repo> git commit -m "updates" . && git push
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see <a href="https://mirage.io/wiki/hello-world#Building-for-Another-Backend">other targets</a> for further information).</p>
|
||||||
Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see [other targets](https://mirage.io/wiki/hello-world#Building-for-Another-Backend) for further information).
|
<pre><code class="language-shell"># back to primary
|
||||||
|
|
||||||
```shell
|
|
||||||
# back to primary
|
|
||||||
$ cd ../dns-primary-git
|
$ cd ../dns-primary-git
|
||||||
$ mirage configure -t hvt # or e.g. -t spt (and solo5-spt below)
|
$ mirage configure -t hvt # or e.g. -t spt (and solo5-spt below)
|
||||||
# installs backend-specific opam packages, recompiles some
|
# installs backend-specific opam packages, recompiles some
|
||||||
|
|
@ -271,64 +198,62 @@ $ dig any mirage @10.0.42.3
|
||||||
|
|
||||||
# testing an update and propagation
|
# testing an update and propagation
|
||||||
# edit mirage zone, add a new record and increment the serial number
|
# edit mirage zone, add a new record and increment the serial number
|
||||||
git-repo> echo "foo A 127.0.0.1" >> mirage
|
git-repo> echo "foo A 127.0.0.1" >> mirage
|
||||||
git-repo> vi mirage <- increment serial
|
git-repo> vi mirage <- increment serial
|
||||||
git-repo> git commit -m 'add foo' . && git push
|
git-repo> git commit -m 'add foo' . && git push
|
||||||
$ onotify 10.0.42.2 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
$ onotify 10.0.42.2 mirage --key=personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg=
|
||||||
|
|
||||||
# now check that it worked
|
# now check that it worked
|
||||||
$ dig foo.mirage @10.0.42.2 # primary
|
$ dig foo.mirage @10.0.42.2 # primary
|
||||||
$ dig foo.mirage @10.0.42.3 # secondary got notified and transferred the zone
|
$ dig foo.mirage @10.0.42.3 # secondary got notified and transferred the zone
|
||||||
```
|
</code></pre>
|
||||||
|
<p>You can also check the behaviour when restarting either of the VMs, whenever the primary is available the zone is synchronised. If the primary is down, the secondary still serves the zone. When the secondary is started while the primary is down, it won't serve any data until the primary is online (the secondary polls periodically, the primary sends notifies on startup).</p>
|
||||||
You can also check the behaviour when restarting either of the VMs, whenever the primary is available the zone is synchronised. If the primary is down, the secondary still serves the zone. When the secondary is started while the primary is down, it won't serve any data until the primary is online (the secondary polls periodically, the primary sends notifies on startup).
|
<h3 id="dynamic-data-updates-via-dns-pushed-to-git">Dynamic data updates via DNS, pushed to git</h3>
|
||||||
|
<p>DNS is a rich protocol, and it also has builtin <a href="https://tools.ietf.org/html/rfc2136">updates</a> that are supported by OCaml DNS, again authenticated with hmac-sha256 and shared secrets. Bind provides the command-line utility <code>nsupdate</code> to send these update packets, a simple <code>oupdate</code> unix utility is available as well (i.e. for integration of dynamic DNS clients). You know the drill, add a shared secret to the primary, git push, notify the primary, and voila we can dynamically in-protocol update. An update received by the primary via this way will trigger a git push to the remote git repository, and notifications to the secondary servers as described above.</p>
|
||||||
### Dynamic data updates via DNS, pushed to git
|
<pre><code class="language-shell"># being lazy, I reuse the key above
|
||||||
|
|
||||||
DNS is a rich protocol, and it also has builtin [updates](https://tools.ietf.org/html/rfc2136) that are supported by OCaml DNS, again authenticated with hmac-sha256 and shared secrets. Bind provides the command-line utility `nsupdate` to send these update packets, a simple `oupdate` unix utility is available as well (i.e. for integration of dynamic DNS clients). You know the drill, add a shared secret to the primary, git push, notify the primary, and voila we can dynamically in-protocol update. An update received by the primary via this way will trigger a git push to the remote git repository, and notifications to the secondary servers as described above.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
# being lazy, I reuse the key above
|
|
||||||
$ oupdate 10.0.42.2 personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg= my-other.mirage 1.2.3.4
|
$ oupdate 10.0.42.2 personal._update.mirage:SHA256:kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv2xg= my-other.mirage 1.2.3.4
|
||||||
|
|
||||||
# let's observe the remote git
|
# let's observe the remote git
|
||||||
git-repo> git pull
|
git-repo> git pull
|
||||||
# there should be a new commit generated by the primary
|
# there should be a new commit generated by the primary
|
||||||
git-repo> git log
|
git-repo> git log
|
||||||
|
|
||||||
# test it, should return 1.2.3.4
|
# test it, should return 1.2.3.4
|
||||||
$ dig my-other.mirage @10.0.42.2
|
$ dig my-other.mirage @10.0.42.2
|
||||||
$ dig my-other.mirage @10.0.42.3
|
$ dig my-other.mirage @10.0.42.3
|
||||||
```
|
</code></pre>
|
||||||
|
<p>So we can deploy further <code>oupdate</code> (or <code>nsupdate</code>) clients, distribute hmac secrets, and have the DNS zone updated. The source of truth is still the git repository, where the primary-git pushes to. Merge conflicts and timing of pushes is not yet dealt with. They are unlikely to happen since the primary is notified on pushes and should have up-to-date data in storage. Sorry, I'm unsure about the error semantics, try it yourself.</p>
|
||||||
So we can deploy further `oupdate` (or `nsupdate`) clients, distribute hmac secrets, and have the DNS zone updated. The source of truth is still the git repository, where the primary-git pushes to. Merge conflicts and timing of pushes is not yet dealt with. They are unlikely to happen since the primary is notified on pushes and should have up-to-date data in storage. Sorry, I'm unsure about the error semantics, try it yourself.
|
<h3 id="lets-encrypt">Let's encrypt!</h3>
|
||||||
|
<p><a href="https://letsencrypt.org/">Let's encrypt</a> is a certificate authority (CA), which certificate is shipped as trust anchor in web browsers. They specified a protocol for <a href="https://tools.ietf.org/html/draft-ietf-acme-acme-05">automated certificate management environment (ACME)</a>, used to get X509 certificates for your services. In the protocol, a certificate signing request (publickey and hostname) is sent to let's encrypt servers, which sends a challenge to proof the ownership of the hostnames. One widely-used way to solve this challenge is running a web server, another is to serve it as text record from the authoritative DNS server.</p>
|
||||||
### Let's encrypt!
|
<p>Since I avoid persistent storage when possible, and also don't want to integrate a HTTP client stack in the primary server, I developed a third unikernel that acts as (hidden) secondary server, performs the tedious HTTP communication with let's encrypt servers, and stores all data in the public DNS zone.</p>
|
||||||
|
<p>For encoding of certificates, the DANE working group specified <a href="https://tools.ietf.org/html/rfc6698.html#section-7.1">TLSA</a> records in DNS. They are quadruples of usage, selector, matching type, and ASN.1 DER-encoded material. We set usage to 3 (domain-issued certificate), matching type to 0 (no hash), and selector to 0 (full certificate) or 255 (private usage) for certificate signing requests. The interaction is as follows:</p>
|
||||||
[Let's encrypt](https://letsencrypt.org/) is a certificate authority (CA), which certificate is shipped as trust anchor in web browsers. They specified a protocol for [automated certificate management environment (ACME)](https://tools.ietf.org/html/draft-ietf-acme-acme-05), used to get X509 certificates for your services. In the protocol, a certificate signing request (publickey and hostname) is sent to let's encrypt servers, which sends a challenge to proof the ownership of the hostnames. One widely-used way to solve this challenge is running a web server, another is to serve it as text record from the authoritative DNS server.
|
<ol>
|
||||||
|
<li>Primary, secondary, and let's encrypt unikernels are running
|
||||||
Since I avoid persistent storage when possible, and also don't want to integrate a HTTP client stack in the primary server, I developed a third unikernel that acts as (hidden) secondary server, performs the tedious HTTP communication with let's encrypt servers, and stores all data in the public DNS zone.
|
</li>
|
||||||
|
<li>A service (<code>ocertify</code>, <code>unikernels/certificate</code>, or the <code>dns-certify.mirage</code> library) demands a TLS certificate, and has a hmac-secret for the primary DNS
|
||||||
For encoding of certificates, the DANE working group specified [TLSA](https://tools.ietf.org/html/rfc6698.html#section-7.1) records in DNS. They are quadruples of usage, selector, matching type, and ASN.1 DER-encoded material. We set usage to 3 (domain-issued certificate), matching type to 0 (no hash), and selector to 0 (full certificate) or 255 (private usage) for certificate signing requests. The interaction is as follows:
|
</li>
|
||||||
|
<li>The service generates a certificate signing request with the desired hostname(s), and performs an nsupdate with TLSA 255 <DER encoded signing-request>
|
||||||
1. Primary, secondary, and let's encrypt unikernels are running
|
</li>
|
||||||
2. A service (`ocertify`, `unikernels/certificate`, or the `dns-certify.mirage` library) demands a TLS certificate, and has a hmac-secret for the primary DNS
|
<li>The primary accepts the update, pushes the new zone to git, and sends notifies to secondary and let's encrypt unikernels which (incrementally) transfer the zone
|
||||||
3. The service generates a certificate signing request with the desired hostname(s), and performs an nsupdate with TLSA 255 <DER encoded signing-request>
|
</li>
|
||||||
4. The primary accepts the update, pushes the new zone to git, and sends notifies to secondary and let's encrypt unikernels which (incrementally) transfer the zone
|
<li>The let's encrypt unikernel notices while transferring the zone a signing request without a certificate, starts HTTP interaction with let's encrypt
|
||||||
5. The let's encrypt unikernel notices while transferring the zone a signing request without a certificate, starts HTTP interaction with let's encrypt
|
</li>
|
||||||
6. The let's encrypt unikernel solves the challenge, sends the response as update of a TXT record to the primary nameserver
|
<li>The let's encrypt unikernel solves the challenge, sends the response as update of a TXT record to the primary nameserver
|
||||||
7. The primary pushes the TXT record to git, and notifies secondaries (which transfer the zone)
|
</li>
|
||||||
8. The let's encrypt servers request the TXT record from either or both authoritative name servers
|
<li>The primary pushes the TXT record to git, and notifies secondaries (which transfer the zone)
|
||||||
9. The let's encrypt unikernel polls for the issued certificate and send an update to the primary TLSA 0 <DER encoded certificate>
|
</li>
|
||||||
10. The primary pushes the certificate to git, notifies secondaries (which transfer the zone)
|
<li>The let's encrypt servers request the TXT record from either or both authoritative name servers
|
||||||
11. The service polls TLSA records for the hostname, and use it upon retrieval
|
</li>
|
||||||
|
<li>The let's encrypt unikernel polls for the issued certificate and send an update to the primary TLSA 0 <DER encoded certificate>
|
||||||
Note that neither the signing request nor the certificate contain private key material, thus it is fine to serve them publically. Please also note, that the service polls for the certificate for the hostname in DNS, which is valid (start and end date) certificate and uses the same public key, this certificate is used and steps 3-10 are not executed.
|
</li>
|
||||||
|
<li>The primary pushes the certificate to git, notifies secondaries (which transfer the zone)
|
||||||
The let's encrypt unikernel does not serve anything, it is a reactive system which acts upon notification from the primary. Thus, it can be executed in a private address space (with a NAT). Since the OCaml DNS server stack needs to push notifications to it, it preserves all incoming signed SOA requests as candidates for notifications on update. The let's encrypt unikernel ensures to always have a connection to the primary to receive notifications.
|
</li>
|
||||||
|
<li>The service polls TLSA records for the hostname, and use it upon retrieval
|
||||||
```shell
|
</li>
|
||||||
# getting let's encrypt up and running
|
</ol>
|
||||||
|
<p>Note that neither the signing request nor the certificate contain private key material, thus it is fine to serve them publically. Please also note, that the service polls for the certificate for the hostname in DNS, which is valid (start and end date) certificate and uses the same public key, this certificate is used and steps 3-10 are not executed.</p>
|
||||||
|
<p>The let's encrypt unikernel does not serve anything, it is a reactive system which acts upon notification from the primary. Thus, it can be executed in a private address space (with a NAT). Since the OCaml DNS server stack needs to push notifications to it, it preserves all incoming signed SOA requests as candidates for notifications on update. The let's encrypt unikernel ensures to always have a connection to the primary to receive notifications.</p>
|
||||||
|
<pre><code class="language-shell"># getting let's encrypt up and running
|
||||||
$ cd ..
|
$ cd ..
|
||||||
$ git clone https://github.com/roburio/dns-letsencrypt-secondary.git
|
$ git clone https://github.com/roburio/dns-letsencrypt-secondary.git
|
||||||
$ cd dns-letsencrypt-secondary
|
$ cd dns-letsencrypt-secondary
|
||||||
|
|
@ -340,18 +265,12 @@ $ solo5-hvt --net:service=tap2 -- letsencrypt.hvt --keys=...
|
||||||
|
|
||||||
# test it
|
# test it
|
||||||
$ ocertify 10.0.42.2 foo.mirage
|
$ ocertify 10.0.42.2 foo.mirage
|
||||||
```
|
</code></pre>
|
||||||
|
<p>For actual testing with let's encrypt servers you need to have the primary and secondary deployed on your remote hosts, and your domain needs to be delegated to these servers. Good luck. And ensure you have backup your git repository.</p>
|
||||||
For actual testing with let's encrypt servers you need to have the primary and secondary deployed on your remote hosts, and your domain needs to be delegated to these servers. Good luck. And ensure you have backup your git repository.
|
<p>As fine print, while this tutorial was about the <code>mirage</code> zone, you can stick any number of zones into the git repository. If you use a <code>_keys</code> file (without any domain prefix), you can configure hmac secrets for all zones, i.e. something to use in your let's encrypt unikernel and secondary unikernel. Dynamic addition of zones is supported, just create a new zonefile and notify the primary, the secondary will be notified and pick it up. The primary responds to a signed SOA for the root zone (i.e. requested by the secondary) with the SOA response (not authoritative), and additionally notifications for all domains of the primary.</p>
|
||||||
|
<h3 id="conclusion-and-thanks">Conclusion and thanks</h3>
|
||||||
As fine print, while this tutorial was about the `mirage` zone, you can stick any number of zones into the git repository. If you use a `_keys` file (without any domain prefix), you can configure hmac secrets for all zones, i.e. something to use in your let's encrypt unikernel and secondary unikernel. Dynamic addition of zones is supported, just create a new zonefile and notify the primary, the secondary will be notified and pick it up. The primary responds to a signed SOA for the root zone (i.e. requested by the secondary) with the SOA response (not authoritative), and additionally notifications for all domains of the primary.
|
<p>This tutorial presented how to use the OCaml DNS based unikernels to run authoritative name servers for your domain, using a git repository as the source of truth, dynamic authenticated updates, and let's encrypt certificate issuing.</p>
|
||||||
|
<p>There are further steps to take, such as monitoring (<code>mirage configure --monitoring</code>), which use a second network interface for reporting syslog and metrics to telegraf / influx / grafana. Some DNS features are still missing, most prominently DNSSec.</p>
|
||||||
### Conclusion and thanks
|
<p>I'd like to thank all people involved in this software stack, without other key components, including <a href="https://github.com/mirage/ocaml-git">git</a>, <a href="https://github.com/mirage/mirage-crypto">mirage-crypto</a>, <a href="https://github.com/mirage/awa-ssh">awa-ssh</a>, <a href="https://github.com/solo5/sol5">solo5</a>, <a href="https://github.com/mirage/mirage">mirage</a>, <a href="https://github.com/mmaker/ocaml-letsencrypt">ocaml-letsencrypt</a>, and more.</p>
|
||||||
|
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
This tutorial presented how to use the OCaml DNS based unikernels to run authoritative name servers for your domain, using a git repository as the source of truth, dynamic authenticated updates, and let's encrypt certificate issuing.
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
There are further steps to take, such as monitoring (`mirage configure --monitoring`), which use a second network interface for reporting syslog and metrics to telegraf / influx / grafana. Some DNS features are still missing, most prominently DNSSec.
|
|
||||||
|
|
||||||
I'd like to thank all people involved in this software stack, without other key components, including [git](https://github.com/mirage/ocaml-git), [mirage-crypto](https://github.com/mirage/mirage-crypto), [awa-ssh](https://github.com/mirage/awa-ssh), [solo5](https://github.com/solo5/sol5), [mirage](https://github.com/mirage/mirage), [ocaml-letsencrypt](https://github.com/mmaker/ocaml-letsencrypt), and more.
|
|
||||||
|
|
||||||
If you want to support our work on MirageOS unikernels, please [donate to robur](https://robur.coop/Donate). I'm interested in feedback, either via [twitter](https://twitter.com/h4nnes), [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
137
Posts/EC
137
Posts/EC
|
|
@ -1,92 +1,45 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Cryptography updates in OCaml and MirageOS
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Cryptography updates in OCaml and MirageOS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Cryptography updates in OCaml and MirageOS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/tls" class="tag">tls</a></div><span class="date">Published: 2021-04-23 (last updated: 2021-11-19)</span><article><h2 id="introduction">Introduction</h2>
|
||||||
author: hannes
|
<p>Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a> stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.</p>
|
||||||
tags: mirageos, security, tls
|
<h2 id="elliptic-curve-cryptography">Elliptic curve cryptography</h2>
|
||||||
abstract: Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.
|
<p><a href="https://mirage.io/blog/tls-1-3-mirageos">Since May 2020</a>, our <a href="https://usenix15.nqsb.io">OCaml-TLS</a> stack supports TLS 1.3 (since tls version 0.12.0 on opam).</p>
|
||||||
---
|
<p>TLS 1.3 requires elliptic curve cryptography - which was not available in <a href="https://github.com/mirage/mirage-crypto">mirage-crypto</a> (the maintained fork of <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a>).</p>
|
||||||
|
<p>There are two major uses of elliptic curves: <a href="https://en.wikipedia.org/wiki/Elliptic-curve_Diffie%E2%80%93Hellman">key exchange (ECDH)</a> for establishing a shared secret over an insecure channel, and <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm">digital signature (ECDSA)</a> for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)</p>
|
||||||
## Introduction
|
<p>Elliptic curve cryptoraphy is <a href="https://eprint.iacr.org/2020/615">vulnerable</a> <a href="https://raccoon-attack.com/">to</a> <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-5407">various</a> <a href="https://github.com/mimoo/timing_attack_ecdsa_tls">timing</a> <a href="https://minerva.crocs.fi.muni.cz/">attacks</a> - have a read of the <a href="https://blog.trailofbits.com/2020/06/11/ecdsa-handle-with-care/">overview article on ECDSA</a>. When implementing elliptic curve cryptography, it is best to avoid these known attacks. Gladly, there are some projects which address these issues by construction.</p>
|
||||||
|
<p>In addition, to use the code in MirageOS, it should be boring C code: no heap allocations, only using a very small amount of C library functions -- the code needs to be compiled in an environment with <a href="https://github.com/mirage/ocaml-freestanding/tree/v0.6.4/nolibc">nolibc</a>.</p>
|
||||||
Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified [fiat-crypto](https://github.com/mit-plv/fiat-crypto/) stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.
|
<p>Two projects started in semantics, to solve the issue from the grounds up: <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a> and <a href="https://github.com/project-everest/hacl-star/">hacl-star</a>: their approach is to use a proof system (<a href="https://coq.inria.fr">Coq</a> or <a href="https://www.fstar-lang.org/">F*</a> to verify that the code executes in constant time, not depending on data input. Both projects provide as output of their proof systems C code.</p>
|
||||||
|
<p>For our initial TLS 1.3 stack, <a href="https://github.com/pascutto/">Clément</a>, <a href="https://github.com/NathanReb/">Nathan</a> and <a href="https://github.com/emillon/">Etienne</a> developed <a href="https://github.com/mirage/fiat">fiat-p256</a> and <a href="https://github.com/mirage/hacl">hacl_x5519</a>. Both were one-shot interfaces for a narrow use case (ECDH for NIST P-256 and X25519), worked well for their purpose, and allowed to gather some experience from the development side.</p>
|
||||||
## Elliptic curve cryptography
|
<h3 id="changed-requirements">Changed requirements</h3>
|
||||||
|
<p>Revisiting our cryptography stack with the elliptic curve perspective had several reasons, on the one side the customer project <a href="https://www.nitrokey.com/products/nethsm">NetHSM</a> asked for feasibility of ECDSA/EdDSA for various elliptic curves, on the other side <a href="https://github.com/mirage/ocaml-dns/pull/251">DNSSec</a> uses elliptic curve cryptography (ECDSA), and also <a href="https://www.wireguard.com/">wireguard</a> relies on elliptic curve cryptography. The number of X.509 certificates using elliptic curves is increasing, and we don't want to leave our TLS stack in a state where it can barely talk to a growing number of services on the Internet.</p>
|
||||||
[Since May 2020](https://mirage.io/blog/tls-1-3-mirageos), our [OCaml-TLS](https://usenix15.nqsb.io) stack supports TLS 1.3 (since tls version 0.12.0 on opam).
|
<p>Looking at <a href="https://github.com/project-everest/hacl-star/"><em>hacl-star</em></a>, their <a href="https://hacl-star.github.io/Supported.html">support</a> is limited to P-256 and Curve25519, any new curve requires writing F*. Another issue with hacl-star is C code quality: their C code does neither <a href="https://github.com/mirage/hacl/issues/46">compile with older C compilers (found on Oracle Linux 7 / CentOS 7)</a>, nor when enabling all warnings (> 150 are generated). We consider the C compiler as useful resource to figure out undefined behaviour (and other problems), and when shipping C code we ensure that it compiles with <code>-Wall -Wextra -Wpedantic --std=c99 -Werror</code>. The hacl project <a href="https://github.com/mirage/hacl/tree/master/src/kremlin">ships</a> a bunch of header files and helper functions to work on all platforms, which is a clunky <code>ifdef</code> desert. The hacl approach is to generate a whole algorithm solution: from arithmetic primitives, group operations, up to cryptographic protocol - everything included.</p>
|
||||||
|
<p>In contrast, <a href="https://github.com/mit-plv/fiat-crypto/"><em>fiat-crypto</em></a> is a Coq development, which as part of compilation (proof verification) generates executables (via OCaml code extraction from Coq). These executables are used to generate modular arithmetic (as C code) given a curve description. The <a href="https://github.com/mirage/mirage-crypto/tree/main/ec/native">generated C code</a> is highly portable, independent of platform (word size is taken as input) - it only requires a <code><stdint.h></code>, and compiles with all warnings enabled (once <a href="https://github.com/mit-plv/fiat-crypto/pull/906">a minor PR</a> got merged). Supporting a new curve is simple: generate the arithmetic code using fiat-crypto with the new curve description. The downside is that group operations and protocol needs to implemented elsewhere (and is not part of the proven code) - gladly this is pretty straightforward to do, especially in high-level languages.</p>
|
||||||
TLS 1.3 requires elliptic curve cryptography - which was not available in [mirage-crypto](https://github.com/mirage/mirage-crypto) (the maintained fork of [nocrypto](https://github.com/mirleft/ocaml-nocrypto)).
|
<h3 id="working-with-fiat-crypto">Working with fiat-crypto</h3>
|
||||||
|
<p>As mentioned, our initial <a href="https://github.com/mirage/fiat">fiat-p256</a> binding provided ECDH for the NIST P-256 curve. Also, BoringSSL uses fiat-crypto for ECDH, and developed the code for group operations and cryptographic protocol on top of it.</p>
|
||||||
There are two major uses of elliptic curves: [key exchange (ECDH)](https://en.wikipedia.org/wiki/Elliptic-curve_Diffie%E2%80%93Hellman) for establishing a shared secret over an insecure channel, and [digital signature (ECDSA)](https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm) for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)
|
<p>The work needed was (a) ECDSA support and (b) supporting more curves (let's focus on NIST curves). For ECDSA, the algorithm requires modular arithmetics in the field of the group order (in addition to the prime). We generate these primitives with fiat-crypto (named <code>npYYY_AA</code>) - that required <a href="https://github.com/mit-plv/fiat-crypto/commit/e31a36d5f1b20134e67ccc5339d88f0ff3cb0f86">a small fix in decoding hex</a>. Fiat-crypto also provides inversion <a href="https://github.com/mit-plv/fiat-crypto/pull/670">since late October 2020</a>, <a href="https://eprint.iacr.org/2021/549">paper</a> - which allowed to reduce our code base taken from BoringSSL. The ECDSA protocol was easy to implement in OCaml using the generated arithmetics.</p>
|
||||||
|
<p>Addressing the issue of more curves was also easy to achieve, the C code (group operations) are macros that are instantiated for each curve - the OCaml code are functors that are applied with each curve description.</p>
|
||||||
Elliptic curve cryptoraphy is [vulnerable](https://eprint.iacr.org/2020/615) [to](https://raccoon-attack.com/) [various](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-5407) [timing](https://github.com/mimoo/timing_attack_ecdsa_tls) [attacks](https://minerva.crocs.fi.muni.cz/) - have a read of the [overview article on ECDSA](https://blog.trailofbits.com/2020/06/11/ecdsa-handle-with-care/). When implementing elliptic curve cryptography, it is best to avoid these known attacks. Gladly, there are some projects which address these issues by construction.
|
<p>Thanks to the test vectors (as structured data) from <a href="https://github.com/google/wycheproof/">wycheproof</a> (and again thanks to Etienne, Nathan, and Clément for their OCaml code decodin them), I feel confident that our elliptic curve code works as desired.</p>
|
||||||
|
<p>What was left is X25519 and Ed25519 - dropping the hacl dependency entirely felt appealing (less C code to maintain from fewer projects). This turned out to require more C code, which we took from BoringSSL. It may be desirable to reduce the imported C code, or to wait until a project on top of fiat-crypto which provides proven cryptographic protocols is in a usable state.</p>
|
||||||
In addition, to use the code in MirageOS, it should be boring C code: no heap allocations, only using a very small amount of C library functions -- the code needs to be compiled in an environment with [nolibc](https://github.com/mirage/ocaml-freestanding/tree/v0.6.4/nolibc).
|
<p>To avoid performance degradation, I distilled some <a href="https://github.com/mirage/mirage-crypto/pull/107#issuecomment-799701703">X25519 benchmarks</a>, turns out the fiat-crypto and hacl performance is very similar.</p>
|
||||||
|
<h3 id="achievements">Achievements</h3>
|
||||||
Two projects started in semantics, to solve the issue from the grounds up: [fiat-crypto](https://github.com/mit-plv/fiat-crypto/) and [hacl-star](https://github.com/project-everest/hacl-star/): their approach is to use a proof system ([Coq](https://coq.inria.fr) or [F*](https://www.fstar-lang.org/) to verify that the code executes in constant time, not depending on data input. Both projects provide as output of their proof systems C code.
|
<p>The new opam package <a href="https://mirage.github.io/mirage-crypto/doc/mirage-crypto-ec/Mirage_crypto_ec/index.html">mirage-crypto-ec</a> is released, which includes the C code generated by fiat-crypto (including <a href="https://github.com/mit-plv/fiat-crypto/pull/670">inversion</a>), <a href="https://github.com/mirage/mirage-crypto/blob/main/ec/native/point_operations.h">point operations</a> from BoringSSL, and some <a href="https://github.com/mirage/mirage-crypto/blob/main/ec/mirage_crypto_ec.ml">OCaml code</a> for invoking these functions and doing bounds checks, and whether points are on the curve. The OCaml code are some functors that take the curve description (consisting of parameters, C function names, byte length of value) and provide Diffie-Hellman (Dh) and digital signature algorithm (Dsa) modules. The nonce for ECDSA is computed deterministically, as suggested by <a href="https://tools.ietf.org/html/rfc6979">RFC 6979</a>, to avoid private key leakage.</p>
|
||||||
|
<p>The code has been developed in <a href="https://github.com/mirage/mirage-crypto/pull/101">NIST curves</a>, <a href="https://github.com/mirage/mirage-crypto/pull/106">removing blinding</a> (since we use operations that are verified to be constant-time), <a href="https://github.com/mirage/mirage-crypto/pull/108">added missing length checks</a> (reported by <a href="https://github.com/greg42">Greg</a>), <a href="https://github.com/mirage/mirage-crypto/pull/107">curve25519</a>, <a href="https://github.com/mirage/mirage-crypto/pull/117">a fix for signatures that do not span the entire byte size (discovered while adapting X.509)</a>, <a href="https://github.com/mirage/mirage-crypto/pull/118">fix X25519 when the input has offset <> 0</a>. It works on x86 and arm, both 32 and 64 bit (checked by CI). The development was partially sponsored by Nitrokey.</p>
|
||||||
For our initial TLS 1.3 stack, [Clément](https://github.com/pascutto/), [Nathan](https://github.com/NathanReb/) and [Etienne](https://github.com/emillon/) developed [fiat-p256](https://github.com/mirage/fiat) and [hacl_x5519](https://github.com/mirage/hacl). Both were one-shot interfaces for a narrow use case (ECDH for NIST P-256 and X25519), worked well for their purpose, and allowed to gather some experience from the development side.
|
<p>What is left to do, apart from further security reviews, is <a href="https://github.com/mirage/mirage-crypto/issues/109">performance improvements</a>, <a href="https://github.com/mirage/mirage-crypto/issues/112">Ed448/X448 support</a>, and <a href="https://github.com/mirage/mirage-crypto/issues/105">investigating deterministic k for P521</a>. Pull requests are welcome.</p>
|
||||||
|
<p>When you use the code, and encounter any issues, please <a href="https://github.com/mirage/mirage-crypto/issues">report them</a>.</p>
|
||||||
### Changed requirements
|
<h2 id="layer-up---x.509-now-with-ecdsa-eddsa-and-pkcs-12-support-and-a-long-standing-issue-fixed">Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed</h2>
|
||||||
|
<p>With the sign and verify primitives, the next step is to interoperate with other tools that generate and use these public and private keys. This consists of serialisation to and deserialisation from common data formats (ASN.1 DER and PEM encoding), and support for handling X.509 certificates with elliptic curve keys. Since X.509 0.12.0, it supports EC private and public keys, including certificate validation and issuance.</p>
|
||||||
Revisiting our cryptography stack with the elliptic curve perspective had several reasons, on the one side the customer project [NetHSM](https://www.nitrokey.com/products/nethsm) asked for feasibility of ECDSA/EdDSA for various elliptic curves, on the other side [DNSSec](https://github.com/mirage/ocaml-dns/pull/251) uses elliptic curve cryptography (ECDSA), and also [wireguard](https://www.wireguard.com/) relies on elliptic curve cryptography. The number of X.509 certificates using elliptic curves is increasing, and we don't want to leave our TLS stack in a state where it can barely talk to a growing number of services on the Internet.
|
<p>Releasing X.509 also included to go through the issue tracker and attempt to solve the existing issues. This time, the <a href="https://github.com/mirleft/ocaml-x509/issues/69">"country name is encoded as UTF8String, while RFC demands PrintableString"</a> filed more than 5 years ago by <a href="https://github.com/reynir">Reynir</a>, re-reported by <a href="https://github.com/paurkedal">Petter</a> in early 2017, and again by <a href="https://github.com/NightBlues">Vadim</a> in late 2020, <a href="https://github.com/mirleft/ocaml-x509/pull/140">was fixed by Vadim</a>.</p>
|
||||||
|
<p>Another long-standing pull request was support for <a href="https://tools.ietf.org/html/rfc7292">PKCS 12</a>, the archive format for certificate and private key bundles. This has <a href="https://github.com/mirleft/ocaml-x509/pull/114">been developed and merged</a>. PKCS 12 is a widely used and old format (e.g. when importing / exporting cryptographic material in your browser, used by OpenVPN, ...). Its specification uses RC2 and 3DES (see <a href="https://unmitigatedrisk.com/?p=654">this nice article</a>), which are the default algorithms used by <code>openssl pkcs12</code>.</p>
|
||||||
Looking at [*hacl-star*](https://github.com/project-everest/hacl-star/), their [support](https://hacl-star.github.io/Supported.html) is limited to P-256 and Curve25519, any new curve requires writing F*. Another issue with hacl-star is C code quality: their C code does neither [compile with older C compilers (found on Oracle Linux 7 / CentOS 7)](https://github.com/mirage/hacl/issues/46), nor when enabling all warnings (> 150 are generated). We consider the C compiler as useful resource to figure out undefined behaviour (and other problems), and when shipping C code we ensure that it compiles with `-Wall -Wextra -Wpedantic --std=c99 -Werror`. The hacl project [ships](https://github.com/mirage/hacl/tree/master/src/kremlin) a bunch of header files and helper functions to work on all platforms, which is a clunky `ifdef` desert. The hacl approach is to generate a whole algorithm solution: from arithmetic primitives, group operations, up to cryptographic protocol - everything included.
|
<h2 id="one-more-layer-up---tls">One more layer up - TLS</h2>
|
||||||
|
<p>In TLS we are finally able to use ECDSA (and EdDSA) certificates and private keys, this resulted in slightly more complex configuration - the constraints between supported groups, signature algorithms, ciphersuite, and certificates are intricate:</p>
|
||||||
In contrast, [*fiat-crypto*](https://github.com/mit-plv/fiat-crypto/) is a Coq development, which as part of compilation (proof verification) generates executables (via OCaml code extraction from Coq). These executables are used to generate modular arithmetic (as C code) given a curve description. The [generated C code](https://github.com/mirage/mirage-crypto/tree/main/ec/native) is highly portable, independent of platform (word size is taken as input) - it only requires a `<stdint.h>`, and compiles with all warnings enabled (once [a minor PR](https://github.com/mit-plv/fiat-crypto/pull/906) got merged). Supporting a new curve is simple: generate the arithmetic code using fiat-crypto with the new curve description. The downside is that group operations and protocol needs to implemented elsewhere (and is not part of the proven code) - gladly this is pretty straightforward to do, especially in high-level languages.
|
<p>The ciphersuite (in TLS before 1.3) specifies which key exchange mechanism to use, but also which signature algorithm to use (RSA/ECDSA). The supported groups client hello extension specifies which elliptic curves are supported by the client. The signature algorithm hello extension (TLS 1.2 and above) specifies the signature algorithm. In the end, at load time the TLS configuration is validated and groups, ciphersuites, and signature algorithms are condensed depending on configured server certificates. At session initiation time, once the client reports what it supports, these parameters are further cut down to eventually find some suitable cryptographic parameters for this session.</p>
|
||||||
|
<p>From the user perspective, earlier the certificate bundle and private key was a pair of <code>X509.Certificate.t list</code> and <code>Mirage_crypto_pk.Rsa.priv</code>, now the second part is a <code>X509.Private_key.t</code> - all provided constructors have been updates (notably <code>X509_lwt.private_of_pems</code> and <code>Tls_mirage.X509.certificate</code>).</p>
|
||||||
### Working with fiat-crypto
|
<h2 id="finally-conduit-and-mirage">Finally, conduit and mirage</h2>
|
||||||
|
<p>Thanks to <a href="https://github.com/dinosaure">Romain</a>, conduit* 4.0.0 was released which supports the modified API of X.509 and TLS. Romain also developed patches and released mirage 3.10.3 which supports the above mentioned work.</p>
|
||||||
As mentioned, our initial [fiat-p256](https://github.com/mirage/fiat) binding provided ECDH for the NIST P-256 curve. Also, BoringSSL uses fiat-crypto for ECDH, and developed the code for group operations and cryptographic protocol on top of it.
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>Elliptic curve cryptography is now available in OCaml using verified cryptographic primitives from the fiat-crypto project - <code>opam install mirage-crypto-ec</code>. X.509 since 0.12.0 and TLS since 0.13.0 and MirageOS since 3.10.3 support this new development which gives rise to smaller EC keys. Our old bindings, fiat-p256 and hacl_x25519 have been archived and will no longer be maintained.</p>
|
||||||
The work needed was (a) ECDSA support and (b) supporting more curves (let's focus on NIST curves). For ECDSA, the algorithm requires modular arithmetics in the field of the group order (in addition to the prime). We generate these primitives with fiat-crypto (named `npYYY_AA`) - that required [a small fix in decoding hex](https://github.com/mit-plv/fiat-crypto/commit/e31a36d5f1b20134e67ccc5339d88f0ff3cb0f86). Fiat-crypto also provides inversion [since late October 2020](https://github.com/mit-plv/fiat-crypto/pull/670), [paper](https://eprint.iacr.org/2021/549) - which allowed to reduce our code base taken from BoringSSL. The ECDSA protocol was easy to implement in OCaml using the generated arithmetics.
|
<p>Thanks to everyone involved on this journey: reporting issues, sponsoring parts of the work, helping with integration, developing initial prototypes, and keep motivating me to continue this until the release is done.</p>
|
||||||
|
<p>In the future, it may be possible to remove zarith and gmp from the dependency chain, and provide EC-only TLS servers and clients for MirageOS. The benefit will be much less C code (libgmp-freestanding.a is 1.5MB in size) in our trusted code base.</p>
|
||||||
Addressing the issue of more curves was also easy to achieve, the C code (group operations) are macros that are instantiated for each curve - the OCaml code are functors that are applied with each curve description.
|
<p>Another potential project that is very close now is a certificate authority developed in MirageOS - now that EC keys, PKCS 12, revocation lists, ... are implemented.</p>
|
||||||
|
<h2 id="footer">Footer</h2>
|
||||||
Thanks to the test vectors (as structured data) from [wycheproof](https://github.com/google/wycheproof/) (and again thanks to Etienne, Nathan, and Clément for their OCaml code decodin them), I feel confident that our elliptic curve code works as desired.
|
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
What was left is X25519 and Ed25519 - dropping the hacl dependency entirely felt appealing (less C code to maintain from fewer projects). This turned out to require more C code, which we took from BoringSSL. It may be desirable to reduce the imported C code, or to wait until a project on top of fiat-crypto which provides proven cryptographic protocols is in a usable state.
|
|
||||||
|
|
||||||
To avoid performance degradation, I distilled some [X25519 benchmarks](https://github.com/mirage/mirage-crypto/pull/107#issuecomment-799701703), turns out the fiat-crypto and hacl performance is very similar.
|
|
||||||
|
|
||||||
### Achievements
|
|
||||||
|
|
||||||
The new opam package [mirage-crypto-ec](https://mirage.github.io/mirage-crypto/doc/mirage-crypto-ec/Mirage_crypto_ec/index.html) is released, which includes the C code generated by fiat-crypto (including [inversion](https://github.com/mit-plv/fiat-crypto/pull/670)), [point operations](https://github.com/mirage/mirage-crypto/blob/main/ec/native/point_operations.h) from BoringSSL, and some [OCaml code](https://github.com/mirage/mirage-crypto/blob/main/ec/mirage_crypto_ec.ml) for invoking these functions and doing bounds checks, and whether points are on the curve. The OCaml code are some functors that take the curve description (consisting of parameters, C function names, byte length of value) and provide Diffie-Hellman (Dh) and digital signature algorithm (Dsa) modules. The nonce for ECDSA is computed deterministically, as suggested by [RFC 6979](https://tools.ietf.org/html/rfc6979), to avoid private key leakage.
|
|
||||||
|
|
||||||
The code has been developed in [NIST curves](https://github.com/mirage/mirage-crypto/pull/101), [removing blinding](https://github.com/mirage/mirage-crypto/pull/106) (since we use operations that are verified to be constant-time), [added missing length checks](https://github.com/mirage/mirage-crypto/pull/108) (reported by [Greg](https://github.com/greg42)), [curve25519](https://github.com/mirage/mirage-crypto/pull/107), [a fix for signatures that do not span the entire byte size (discovered while adapting X.509)](https://github.com/mirage/mirage-crypto/pull/117), [fix X25519 when the input has offset <> 0](https://github.com/mirage/mirage-crypto/pull/118). It works on x86 and arm, both 32 and 64 bit (checked by CI). The development was partially sponsored by Nitrokey.
|
|
||||||
|
|
||||||
What is left to do, apart from further security reviews, is [performance improvements](https://github.com/mirage/mirage-crypto/issues/109), [Ed448/X448 support](https://github.com/mirage/mirage-crypto/issues/112), and [investigating deterministic k for P521](https://github.com/mirage/mirage-crypto/issues/105). Pull requests are welcome.
|
|
||||||
|
|
||||||
When you use the code, and encounter any issues, please [report them](https://github.com/mirage/mirage-crypto/issues).
|
|
||||||
|
|
||||||
## Layer up - X.509 now with ECDSA / EdDSA and PKCS 12 support, and a long-standing issue fixed
|
|
||||||
|
|
||||||
With the sign and verify primitives, the next step is to interoperate with other tools that generate and use these public and private keys. This consists of serialisation to and deserialisation from common data formats (ASN.1 DER and PEM encoding), and support for handling X.509 certificates with elliptic curve keys. Since X.509 0.12.0, it supports EC private and public keys, including certificate validation and issuance.
|
|
||||||
|
|
||||||
Releasing X.509 also included to go through the issue tracker and attempt to solve the existing issues. This time, the ["country name is encoded as UTF8String, while RFC demands PrintableString"](https://github.com/mirleft/ocaml-x509/issues/69) filed more than 5 years ago by [Reynir](https://github.com/reynir), re-reported by [Petter](https://github.com/paurkedal) in early 2017, and again by [Vadim](https://github.com/NightBlues) in late 2020, [was fixed by Vadim](https://github.com/mirleft/ocaml-x509/pull/140).
|
|
||||||
|
|
||||||
Another long-standing pull request was support for [PKCS 12](https://tools.ietf.org/html/rfc7292), the archive format for certificate and private key bundles. This has [been developed and merged](https://github.com/mirleft/ocaml-x509/pull/114). PKCS 12 is a widely used and old format (e.g. when importing / exporting cryptographic material in your browser, used by OpenVPN, ...). Its specification uses RC2 and 3DES (see [this nice article](https://unmitigatedrisk.com/?p=654)), which are the default algorithms used by `openssl pkcs12`.
|
|
||||||
|
|
||||||
## One more layer up - TLS
|
|
||||||
|
|
||||||
In TLS we are finally able to use ECDSA (and EdDSA) certificates and private keys, this resulted in slightly more complex configuration - the constraints between supported groups, signature algorithms, ciphersuite, and certificates are intricate:
|
|
||||||
|
|
||||||
The ciphersuite (in TLS before 1.3) specifies which key exchange mechanism to use, but also which signature algorithm to use (RSA/ECDSA). The supported groups client hello extension specifies which elliptic curves are supported by the client. The signature algorithm hello extension (TLS 1.2 and above) specifies the signature algorithm. In the end, at load time the TLS configuration is validated and groups, ciphersuites, and signature algorithms are condensed depending on configured server certificates. At session initiation time, once the client reports what it supports, these parameters are further cut down to eventually find some suitable cryptographic parameters for this session.
|
|
||||||
|
|
||||||
From the user perspective, earlier the certificate bundle and private key was a pair of `X509.Certificate.t list` and `Mirage_crypto_pk.Rsa.priv`, now the second part is a `X509.Private_key.t` - all provided constructors have been updates (notably `X509_lwt.private_of_pems` and `Tls_mirage.X509.certificate`).
|
|
||||||
|
|
||||||
## Finally, conduit and mirage
|
|
||||||
|
|
||||||
Thanks to [Romain](https://github.com/dinosaure), conduit* 4.0.0 was released which supports the modified API of X.509 and TLS. Romain also developed patches and released mirage 3.10.3 which supports the above mentioned work.
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
Elliptic curve cryptography is now available in OCaml using verified cryptographic primitives from the fiat-crypto project - `opam install mirage-crypto-ec`. X.509 since 0.12.0 and TLS since 0.13.0 and MirageOS since 3.10.3 support this new development which gives rise to smaller EC keys. Our old bindings, fiat-p256 and hacl_x25519 have been archived and will no longer be maintained.
|
|
||||||
|
|
||||||
Thanks to everyone involved on this journey: reporting issues, sponsoring parts of the work, helping with integration, developing initial prototypes, and keep motivating me to continue this until the release is done.
|
|
||||||
|
|
||||||
In the future, it may be possible to remove zarith and gmp from the dependency chain, and provide EC-only TLS servers and clients for MirageOS. The benefit will be much less C code (libgmp-freestanding.a is 1.5MB in size) in our trusted code base.
|
|
||||||
|
|
||||||
Another potential project that is very close now is a certificate authority developed in MirageOS - now that EC keys, PKCS 12, revocation lists, ... are implemented.
|
|
||||||
|
|
||||||
## Footer
|
|
||||||
|
|
||||||
If you want to support our work on MirageOS unikernels, please [donate to robur](https://robur.coop/Donate). I'm interested in feedback, either via [twitter](https://twitter.com/h4nnes), [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
220
Posts/Functoria
220
Posts/Functoria
|
|
@ -1,161 +1,119 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Configuration DSL step-by-step
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Configuration DSL step-by-step</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Configuration DSL step-by-step" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Configuration DSL step-by-step</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-05-10 (last updated: 2021-11-19)</span><article><p>Sorry for being late again with this article, I had other ones planned, but am not yet satisfied with content and code, will have to wait another week.</p>
|
||||||
author: hannes
|
<h2 id="mirageos-configuration">MirageOS configuration</h2>
|
||||||
tags: mirageos, background
|
<p>As described in an <a href="/Posts/OperatingSystem">earlier post</a>, MirageOS is a library operating system which generates single address space custom kernels (so called unikernels) for each application. The application code is (mostly) independent on the used backend. To achieve this, the language which expresses the configuration of a MirageOS unikernel is rather complex, and has to deal with package dependencies, setup of layers (network stack starting at the (virtual) ethernet device, or sockets), logging, tracing.</p>
|
||||||
abstract: how to actually configure the system
|
<p>The abstraction over concrete implementation of e.g. the network stack is done by providing a module signature in the <a href="https://github.com/mirage/mirage/tree/master/types">mirage-types</a> package. The socket-based network stack, the tap device based network stack, and the Xen virtual network device based network stack implement this signature (depending on other module signatures). The unikernel contains code which applies those dependent modules to instantiate a custom-tailored network stack for the specific configuration. A developer should only describe what their requirements are, the user who wants to deploy it should provide the concrete configuration. And the developer should not need to manually instantiate the network stack for all possible configurations, this is what the mirage tool should embed.</p>
|
||||||
---
|
<p>Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code. This turned out to be error prone. In 2015 <a href="https://github.com/Drup">Drup</a> developed <a href="https://github.com/mirage/functoria">Functoria</a>, a domain-specific language (DSL) to organize functor applications, primarily for MirageOS. It has been introduced in <a href="https://mirage.io/blog/introducing-functoria">a blog post</a>. It is not limited to MirageOS (although this is the primary user right now).</p>
|
||||||
|
<p>Functoria has been included in MirageOS since its <a href="https://github.com/mirage/mirage/releases/tag/v2.7.0">2.7.0 release</a> at the end of February 2016. Functoria provides support for command line arguments which can then either be passed at configuration time or at boot time to the unikernel (such as IP address configuration) using the <a href="http://erratique.ch/software/cmdliner">cmdliner library</a> underneath (and includes dynamic man pages, help, sensible command line parsing, and even visualisation (<code>mirage describe</code>) of the configuration and data dependencies).</p>
|
||||||
Sorry for being late again with this article, I had other ones planned, but am not yet satisfied with content and code, will have to wait another week.
|
<p>I won't go into details about command line arguments in here, please have a look at the <a href="https://mirage.io/blog/introducing-functoria">functoria blog post</a> in case you're interested. Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel (<a href="http://marrakech2016.mirage.io">running here</a>, <a href="https://github.com/mirage/marrakech2016">source</a>). Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown. Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).</p>
|
||||||
|
<h3 id="unikernel">Unikernel</h3>
|
||||||
## MirageOS configuration
|
<p>Similar to <a href="/Posts/nqsbWebsite">my nqsb.io website post</a>, this unikernel only has a single resource and thus does not need to do any parsing (or even call <code>read</code>). The main function is <code>start</code>:</p>
|
||||||
|
<pre><code class="language-OCaml">let start stack _ =
|
||||||
As described in an [earlier post](/Posts/OperatingSystem), MirageOS is a library operating system which generates single address space custom kernels (so called unikernels) for each application. The application code is (mostly) independent on the used backend. To achieve this, the language which expresses the configuration of a MirageOS unikernel is rather complex, and has to deal with package dependencies, setup of layers (network stack starting at the (virtual) ethernet device, or sockets), logging, tracing.
|
|
||||||
|
|
||||||
The abstraction over concrete implementation of e.g. the network stack is done by providing a module signature in the [mirage-types](https://github.com/mirage/mirage/tree/master/types) package. The socket-based network stack, the tap device based network stack, and the Xen virtual network device based network stack implement this signature (depending on other module signatures). The unikernel contains code which applies those dependent modules to instantiate a custom-tailored network stack for the specific configuration. A developer should only describe what their requirements are, the user who wants to deploy it should provide the concrete configuration. And the developer should not need to manually instantiate the network stack for all possible configurations, this is what the mirage tool should embed.
|
|
||||||
|
|
||||||
Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code. This turned out to be error prone. In 2015 [Drup](https://github.com/Drup) developed [Functoria](https://github.com/mirage/functoria), a domain-specific language (DSL) to organize functor applications, primarily for MirageOS. It has been introduced in [a blog post](https://mirage.io/blog/introducing-functoria). It is not limited to MirageOS (although this is the primary user right now).
|
|
||||||
|
|
||||||
Functoria has been included in MirageOS since its [2.7.0 release](https://github.com/mirage/mirage/releases/tag/v2.7.0) at the end of February 2016. Functoria provides support for command line arguments which can then either be passed at configuration time or at boot time to the unikernel (such as IP address configuration) using the [cmdliner library](http://erratique.ch/software/cmdliner) underneath (and includes dynamic man pages, help, sensible command line parsing, and even visualisation (`mirage describe`) of the configuration and data dependencies).
|
|
||||||
|
|
||||||
I won't go into details about command line arguments in here, please have a look at the [functoria blog post](https://mirage.io/blog/introducing-functoria) in case you're interested. Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel ([running here](http://marrakech2016.mirage.io), [source](https://github.com/mirage/marrakech2016)). Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown. Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).
|
|
||||||
|
|
||||||
### Unikernel
|
|
||||||
|
|
||||||
Similar to [my nqsb.io website post](/Posts/nqsbWebsite), this unikernel only has a single resource and thus does not need to do any parsing (or even call `read`). The main function is `start`:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let start stack _ =
|
|
||||||
S.listen_tcpv4 stack ~port:80 (serve rendered) ;
|
S.listen_tcpv4 stack ~port:80 (serve rendered) ;
|
||||||
S.listen stack
|
S.listen stack
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Where <code>S</code> is a <a href="https://github.com/mirage/mirage/blob/355edc987135aff640b240594ae4af31815922e5/types/V1.mli#L668-L765">V1_LWT.STACKV4</a>, a complete TCP/IP stack for IPv4. The functions we are using are <a href="https://github.com/mirage/mirage/blob/355edc987135aff640b240594ae4af31815922e5/types/V1.mli#L754-L759">listen_tcpv4</a>, which needs a <code>stack</code>, <code>port</code> and a <code>callback</code> (and should be called <code>register_tcp_callback</code>), and <a href="https://github.com/mirage/mirage/blob/355edc987135aff640b240594ae4af31815922e5/types/V1.mli#L761-L764">listen</a> which polls for incoming frames.</p>
|
||||||
Where `S` is a [V1_LWT.STACKV4](https://github.com/mirage/mirage/blob/355edc987135aff640b240594ae4af31815922e5/types/V1.mli#L668-L765), a complete TCP/IP stack for IPv4. The functions we are using are [listen_tcpv4](https://github.com/mirage/mirage/blob/355edc987135aff640b240594ae4af31815922e5/types/V1.mli#L754-L759), which needs a `stack`, `port` and a `callback` (and should be called `register_tcp_callback`), and [listen](https://github.com/mirage/mirage/blob/355edc987135aff640b240594ae4af31815922e5/types/V1.mli#L761-L764) which polls for incoming frames.
|
<p>Our callback is <code>serve rendered</code>, where <code>serve</code> is defined as:</p>
|
||||||
|
<pre><code class="language-OCaml">let serve data tcp =
|
||||||
Our callback is `serve rendered`, where `serve` is defined as:
|
TCP.writev tcp [ header; data ] >>= fun _ ->
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let serve data tcp =
|
|
||||||
TCP.writev tcp [ header; data ] >>= fun _ ->
|
|
||||||
TCP.close tcp
|
TCP.close tcp
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Upon an incoming TCP connection, the list consisting of <code>header ; data</code> is written to the connection, which is subsequently closed.</p>
|
||||||
Upon an incoming TCP connection, the list consisting of `header ; data` is written to the connection, which is subsequently closed.
|
<p>The function <code>header</code> is very similar to our previous one, splicing a proper HTTP header together:</p>
|
||||||
|
<pre><code class="language-OCaml">let http_header ~status xs =
|
||||||
The function `header` is very similar to our previous one, splicing a proper HTTP header together:
|
let headers = List.map (fun (k, v) -> k ^ ": " ^ v) xs in
|
||||||
|
let lines = status :: headers @ [ "\r\n" ] in
|
||||||
```OCaml
|
Cstruct.of_string (String.concat "\r\n" lines)
|
||||||
let http_header ~status xs =
|
|
||||||
let headers = List.map (fun (k, v) -> k ^ ": " ^ v) xs in
|
|
||||||
let lines = status :: headers @ [ "\r\n" ] in
|
|
||||||
Cstruct.of_string (String.concat "\r\n" lines)
|
|
||||||
|
|
||||||
let header = http_header
|
let header = http_header
|
||||||
~status:"HTTP/1.1 200 OK"
|
~status:"HTTP/1.1 200 OK"
|
||||||
[ ("Content-Type", "text/html; charset=UTF-8") ;
|
[ ("Content-Type", "text/html; charset=UTF-8") ;
|
||||||
("Connection", "close") ]
|
("Connection", "close") ]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>And the <code>rendered</code> function consists of some hardcoded HTML, and references to two other modules, <code>Style.data</code> and <code>Content.data</code>:</p>
|
||||||
And the `rendered` function consists of some hardcoded HTML, and references to two other modules, `Style.data` and `Content.data`:
|
<pre><code class="language-OCaml">let rendered =
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let rendered =
|
|
||||||
Cstruct.of_string
|
Cstruct.of_string
|
||||||
(String.concat "" [
|
(String.concat "" [
|
||||||
"<html><head>" ;
|
"<html><head>" ;
|
||||||
"<title>1st MirageOS hackathon: 11-16th March 2016, Marrakech, Morocco</title>" ;
|
"<title>1st MirageOS hackathon: 11-16th March 2016, Marrakech, Morocco</title>" ;
|
||||||
"<style>" ; Style.data ; "</style>" ;
|
"<style>" ; Style.data ; "</style>" ;
|
||||||
"</head>" ;
|
"</head>" ;
|
||||||
"<body><div id=\"content\">" ;
|
"<body><div id=\"content\">" ;
|
||||||
Content.data ;
|
Content.data ;
|
||||||
"</div></body></html>" ])
|
"</div></body></html>" ])
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This puts together the pieces we need for a simple HTML site. This unikernel does not have any external dependencies, if we assume that the mirage toolchain, the types, and the network implementation are already provided (the latter two are implicitly added by the mirage tool depending on the configuration, the first you'll have to install manually <code>opam install mirage</code>).</p>
|
||||||
This puts together the pieces we need for a simple HTML site. This unikernel does not have any external dependencies, if we assume that the mirage toolchain, the types, and the network implementation are already provided (the latter two are implicitly added by the mirage tool depending on the configuration, the first you'll have to install manually `opam install mirage`).
|
<p>But wait, where do <code>Style</code> and <code>Content</code> come from? There are no <code>ml</code> modules in the repository. Instead, there is a <a href="https://github.com/mirage/marrakech2016/blob/master/data/content.md">content.md</a> and <a href="https://github.com/mirage/marrakech2016/blob/master/data/style.css">style.css</a> in the <code>data</code> subdirectory.</p>
|
||||||
|
<h3 id="configuration">Configuration</h3>
|
||||||
But wait, where do `Style` and `Content` come from? There are no `ml` modules in the repository. Instead, there is a [content.md](https://github.com/mirage/marrakech2016/blob/master/data/content.md) and [style.css](https://github.com/mirage/marrakech2016/blob/master/data/style.css) in the `data` subdirectory.
|
<p>We use the builtin configuration time magic of functoria to translate these into OCaml modules, in such a way that our unikernel does not need to embed code to render markdown to HTML and carry along a markdown data file.</p>
|
||||||
|
<p>Inside of <code>config.ml</code>, let's look again at the bottom:</p>
|
||||||
### Configuration
|
<pre><code class="language-OCaml">let () =
|
||||||
|
register "marrakech2016" [
|
||||||
We use the builtin configuration time magic of functoria to translate these into OCaml modules, in such a way that our unikernel does not need to embed code to render markdown to HTML and carry along a markdown data file.
|
|
||||||
|
|
||||||
Inside of `config.ml`, let's look again at the bottom:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let () =
|
|
||||||
register "marrakech2016" [
|
|
||||||
foreign
|
foreign
|
||||||
~deps:[abstract config_shell]
|
~deps:[abstract config_shell]
|
||||||
"Unikernel.Main"
|
"Unikernel.Main"
|
||||||
( stackv4 @-> job )
|
( stackv4 @-> job )
|
||||||
$ net
|
$ net
|
||||||
]
|
]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The function <a href="https://mirage.github.io/mirage/Mirage.html#VALregister">register</a> is provided by the mirage tool, it will execute the list of jobs using the given name. To construct a <code>job</code>, we use the <a href="https://mirage.github.io/functoria/Functoria.html#VALforeign">foreign</a> combinator, which might have dependencies (here, a list with the single element <code>config_shell</code> explained later, using the <a href="https://mirage.github.io/functoria/Functoria_key.html#VALabstract">abstract</a> combinator), the name of the main function (<code>Unikernel.main</code>), a <a href="https://mirage.github.io/functoria/Functoria.html#TYPEtyp">typ</a> (here constructed using the <a href="https://mirage.github.io/functoria/Functoria.html#VAL%28@-%3E%29">@-></a> combinator, from a <a href="https://mirage.github.io/mirage/Mirage.html#TYPEstackv4">stackv4</a> to a <a href="https://mirage.github.io/functoria/Functoria.html#TYPEjob">job</a>), and this applied (using the <a href="https://mirage.github.io/functoria/Functoria.html#VAL%28$%29">$</a> combinator) to the <code>net</code> (an actual implementation of <code>stackv4</code>).</p>
|
||||||
The function [register](https://mirage.github.io/mirage/Mirage.html#VALregister) is provided by the mirage tool, it will execute the list of jobs using the given name. To construct a `job`, we use the [foreign](https://mirage.github.io/functoria/Functoria.html#VALforeign) combinator, which might have dependencies (here, a list with the single element `config_shell` explained later, using the [abstract](https://mirage.github.io/functoria/Functoria_key.html#VALabstract) combinator), the name of the main function (`Unikernel.main`), a [typ](https://mirage.github.io/functoria/Functoria.html#TYPEtyp) (here constructed using the [@->](https://mirage.github.io/functoria/Functoria.html#VAL%28@-%3E%29) combinator, from a [stackv4](https://mirage.github.io/mirage/Mirage.html#TYPEstackv4) to a [job](https://mirage.github.io/functoria/Functoria.html#TYPEjob)), and this applied (using the [$](https://mirage.github.io/functoria/Functoria.html#VAL%28$%29) combinator) to the `net` (an actual implementation of `stackv4`).
|
<p>The <code>net</code> implementation is as following:</p>
|
||||||
|
<pre><code class="language-OCaml">let address addr nm gw =
|
||||||
The `net` implementation is as following:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let address addr nm gw =
|
|
||||||
let f = Ipaddr.V4.of_string_exn in
|
let f = Ipaddr.V4.of_string_exn in
|
||||||
{ address = f addr ; netmask = f nm ; gateways = [f gw] }
|
{ address = f addr ; netmask = f nm ; gateways = [f gw] }
|
||||||
|
|
||||||
let server = address "198.167.222.204" "255.255.255.0" "198.167.222.1"
|
let server = address "198.167.222.204" "255.255.255.0" "198.167.222.1"
|
||||||
|
|
||||||
let net =
|
let net =
|
||||||
if_impl Key.is_xen
|
if_impl Key.is_xen
|
||||||
(direct_stackv4_with_static_ipv4 default_console tap0 server)
|
(direct_stackv4_with_static_ipv4 default_console tap0 server)
|
||||||
(socket_stackv4 default_console [Ipaddr.V4.any])
|
(socket_stackv4 default_console [Ipaddr.V4.any])
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Depending on whether we're running on unix or xen, either a socket stack (for testing) or the concrete IP configuration for deployment (using <a href="https://mirage.github.io/functoria/Functoria.html#VALif_impl">if_impl</a> and <a href="https://mirage.github.io/mirage/Mirage_key.html#VALis_xen">is_xen</a> from our DSLs).</p>
|
||||||
Depending on whether we're running on unix or xen, either a socket stack (for testing) or the concrete IP configuration for deployment (using [if_impl](https://mirage.github.io/functoria/Functoria.html#VALif_impl) and [is_xen](https://mirage.github.io/mirage/Mirage_key.html#VALis_xen) from our DSLs).
|
<p>So far nothing too surprising, only some combinators of the functoria DSL which let us describe the possible configuration options.</p>
|
||||||
|
<p>Let us look into <code>config_shell</code>, which embeds the markdown and CSS into OCaml modules at configuration time:</p>
|
||||||
So far nothing too surprising, only some combinators of the functoria DSL which let us describe the possible configuration options.
|
<pre><code class="language-OCaml">type sh = ShellConfig
|
||||||
|
|
||||||
Let us look into `config_shell`, which embeds the markdown and CSS into OCaml modules at configuration time:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
type sh = ShellConfig
|
|
||||||
|
|
||||||
let config_shell = impl @@ object
|
let config_shell = impl @@ object
|
||||||
inherit base_configurable
|
inherit base_configurable
|
||||||
|
|
||||||
method configure i =
|
method configure i =
|
||||||
let open Functoria_app.Cmd in
|
let open Functoria_app.Cmd in
|
||||||
let (>>=) = Rresult.(>>=) in
|
let (>>=) = Rresult.(>>=) in
|
||||||
let dir = Info.root i in
|
let dir = Info.root i in
|
||||||
run "echo 'let data = {___|' > style.ml" >>= fun () ->
|
run "echo 'let data = {___|' > style.ml" >>= fun () ->
|
||||||
run "cat data/style.css >> style.ml" >>= fun () ->
|
run "cat data/style.css >> style.ml" >>= fun () ->
|
||||||
run "echo '|___}' >> style.ml" >>= fun () ->
|
run "echo '|___}' >> style.ml" >>= fun () ->
|
||||||
run "echo 'let data = {___|' > content.ml" >>= fun () ->
|
run "echo 'let data = {___|' > content.ml" >>= fun () ->
|
||||||
run "omd data/content.md >> content.ml" >>= fun () ->
|
run "omd data/content.md >> content.ml" >>= fun () ->
|
||||||
run "echo '|___}' >> content.ml"
|
run "echo '|___}' >> content.ml"
|
||||||
|
|
||||||
method clean i = Functoria_app.Cmd.run "rm -f style.ml content.ml"
|
method clean i = Functoria_app.Cmd.run "rm -f style.ml content.ml"
|
||||||
|
|
||||||
method module_name = "Functoria_runtime"
|
method module_name = "Functoria_runtime"
|
||||||
method name = "shell_config"
|
method name = "shell_config"
|
||||||
method ty = Type ShellConfig
|
method ty = Type ShellConfig
|
||||||
end
|
end
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Functoria uses classes internally, and we extend the <a href="https://mirage.github.io/functoria/Functoria.base_configurable-c.html">base_configurable</a> class, which extends <a href="https://mirage.github.io/functoria/Functoria.configurable-c.html">configurable</a> with some sensible defaults.</p>
|
||||||
Functoria uses classes internally, and we extend the [base_configurable](https://mirage.github.io/functoria/Functoria.base_configurable-c.html) class, which extends [configurable](https://mirage.github.io/functoria/Functoria.configurable-c.html) with some sensible defaults.
|
<p>The important bits are what actually happens during <code>configure</code> and <code>clean</code>: execution of some shell commands (<code>echo</code>, <code>omd</code>, and <code>rm</code>) using the <a href="https://mirage.github.io/functoria/Functoria_app.html">functoria application builder</a> interface. Some information is as well exposed via the <a href="https://mirage.github.io/functoria/Functoria_info.html">Functoria_info</a> module.</p>
|
||||||
|
<h3 id="wrapup">Wrapup</h3>
|
||||||
The important bits are what actually happens during `configure` and `clean`: execution of some shell commands (`echo`, `omd`, and `rm`) using the [functoria application builder](https://mirage.github.io/functoria/Functoria_app.html) interface. Some information is as well exposed via the [Functoria_info](https://mirage.github.io/functoria/Functoria_info.html) module.
|
<p>We walked through the configuration magic of MirageOS, which is a domain-specific language designed for MirageOS demands. We can run arbitrary commands at compile time, and do not need to escape into external files, such as Makefile or shell scripts, but can embed them in our <code>config.ml</code>.</p>
|
||||||
|
<p>I'm interested in feedback, either via
|
||||||
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
### Wrapup
|
<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
|
||||||
|
<ul>
|
||||||
We walked through the configuration magic of MirageOS, which is a domain-specific language designed for MirageOS demands. We can run arbitrary commands at compile time, and do not need to escape into external files, such as Makefile or shell scripts, but can embed them in our `config.ml`.
|
<li>now using Html5.P.print instead of string concatenation, as suggested by Drup (both on <a href="https://github.com/mirleft/nqsb.io/commit/f16291b67d203bf6b2ebc0c5c8479b7cfd153683">nqsb.io</a> and in <a href="https://github.com/Engil/Canopy/pull/46">Canopy</a>)
|
||||||
|
</li>
|
||||||
I'm interested in feedback, either via
|
<li>Canopy updated and created timestamps (for <a href="https://github.com/Engil/Canopy/pull/48">irmin-0.10</a> and <a href="https://github.com/Engil/Canopy/pull/43">irmin-0.11</a>)
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
</li>
|
||||||
|
<li>another <a href="https://github.com/mirage/mirage-http/pull/24">resource leak in mirage-http</a>
|
||||||
## Other updates in the MirageOS ecosystem
|
</li>
|
||||||
|
<li><a href="https://github.com/mirage/mirage-platform/pull/165">mirage-platform now has 4.03 support</a> and <a href="https://github.com/mirage/mirage-platform/issues/118">strtod</a> (finally :)
|
||||||
- now using Html5.P.print instead of string concatenation, as suggested by Drup (both on [nqsb.io](https://github.com/mirleft/nqsb.io/commit/f16291b67d203bf6b2ebc0c5c8479b7cfd153683) and in [Canopy](https://github.com/Engil/Canopy/pull/46))
|
</li>
|
||||||
- Canopy updated and created timestamps (for [irmin-0.10](https://github.com/Engil/Canopy/pull/48) and [irmin-0.11](https://github.com/Engil/Canopy/pull/43))
|
<li><a href="https://mirage.io/blog/2016-spring-hackathon">blog posts about retreat in marrakech</a>
|
||||||
- another [resource leak in mirage-http](https://github.com/mirage/mirage-http/pull/24)
|
</li>
|
||||||
- [mirage-platform now has 4.03 support](https://github.com/mirage/mirage-platform/pull/165) and [strtod](https://github.com/mirage/mirage-platform/issues/118) (finally :)
|
<li><a href="https://github.com/Cumulus/Syndic">syndic 1.5.0 release</a> now using ptime instead of calendar
|
||||||
- [blog posts about retreat in marrakech](https://mirage.io/blog/2016-spring-hackathon)
|
</li>
|
||||||
- [syndic 1.5.0 release](https://github.com/Cumulus/Syndic) now using ptime instead of calendar
|
</ul>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
419
Posts/Jackline
419
Posts/Jackline
|
|
@ -1,101 +1,80 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Jackline, a secure terminal-based XMPP client
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Jackline, a secure terminal-based XMPP client</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Jackline, a secure terminal-based XMPP client" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Jackline, a secure terminal-based XMPP client</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/UI" class="tag">UI</a><a href="/tags/security" class="tag">security</a></div><span class="date">Published: 2017-01-30 (last updated: 2021-09-08)</span><article><p><img src="/static/img/jackline2.png" alt="screenshot" /></p>
|
||||||
author: hannes
|
<p>Back in 2014, when we implemented <a href="https://nqsb.io">TLS</a> in OCaml, at some point
|
||||||
tags: UI, security
|
|
||||||
abstract: implement it once to know you can do it. implement it a second time and you get readable code. implementing it a third time from scratch may lead to useful libraries.
|
|
||||||
---
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Back in 2014, when we implemented [TLS](https://nqsb.io) in OCaml, at some point
|
|
||||||
I was bored with TLS. I usually need at least two projects (but not more than 5) at the same time to
|
I was bored with TLS. I usually need at least two projects (but not more than 5) at the same time to
|
||||||
procrastinate the one I should do with the other one - it is always more fun to
|
procrastinate the one I should do with the other one - it is always more fun to
|
||||||
do what you're not supposed to do. I started to implement another security
|
do what you're not supposed to do. I started to implement another security
|
||||||
protocol ([Off-the-record](https://otr.cypherpunks.ca/), resulted in
|
protocol (<a href="https://otr.cypherpunks.ca/">Off-the-record</a>, resulted in
|
||||||
[ocaml-otr](https://hannesm.github.io/ocaml-otr/doc/Otr.html)) on my own,
|
<a href="https://hannesm.github.io/ocaml-otr/doc/Otr.html">ocaml-otr</a>) on my own,
|
||||||
applying what I learned while co-developing TLS with David. I was eager to
|
applying what I learned while co-developing TLS with David. I was eager to
|
||||||
actually deploy our TLS stack: using it with a web server (see [this post](/Posts/nqsbWebsite)) is fun, but only using one half
|
actually deploy our TLS stack: using it with a web server (see <a href="/Posts/nqsbWebsite">this post</a>) is fun, but only using one half
|
||||||
of the state machine (server side) and usually short-lived connections
|
of the state machine (server side) and usually short-lived connections
|
||||||
(discovers lots of issues with connection establishment) - not the client side
|
(discovers lots of issues with connection establishment) - not the client side
|
||||||
and no long living connection (which may discover other kinds of issues, such as
|
and no long living connection (which may discover other kinds of issues, such as
|
||||||
leaking memory).
|
leaking memory).</p>
|
||||||
|
<p>To use the stack, I needed to find an application I use on a daily basis (thus
|
||||||
To use the stack, I needed to find an application I use on a daily basis (thus
|
|
||||||
I'm eager to get it up and running if it fails to work). Mail client or web
|
I'm eager to get it up and running if it fails to work). Mail client or web
|
||||||
client are just a bit too big for a spare time project (maybe not ;). Another
|
client are just a bit too big for a spare time project (maybe not ;). Another
|
||||||
communication protocol I use daily is jabber, or
|
communication protocol I use daily is jabber, or
|
||||||
[XMPP](https://en.wikipedia.org/wiki/Xmpp). Back then I used
|
<a href="https://en.wikipedia.org/wiki/Xmpp">XMPP</a>. Back then I used
|
||||||
[mcabber](https://mcabber.com) inside a terminal, which is a curses based client
|
<a href="https://mcabber.com">mcabber</a> inside a terminal, which is a curses based client
|
||||||
written in C.
|
written in C.</p>
|
||||||
|
<p>I started to develop <a href="https://github.com/hannesm/jackline">jackline</a> (first
|
||||||
I started to develop [jackline](https://github.com/hannesm/jackline) (first
|
|
||||||
commit is 13th November 2014), a terminal based XMPP client in
|
commit is 13th November 2014), a terminal based XMPP client in
|
||||||
[OCaml](/Posts/OCaml). This is a report of a
|
<a href="/Posts/OCaml">OCaml</a>. This is a report of a
|
||||||
work-in-progress (unreleased, but publicly available!) software project. I'm
|
work-in-progress (unreleased, but publicly available!) software project. I'm
|
||||||
not happy with the code base, but neverthelss consider it to be a successful
|
not happy with the code base, but neverthelss consider it to be a successful
|
||||||
project: dozens of friends are using it (no exact numbers), I got [contributions from other people](https://github.com/hannesm/jackline/graphs/contributors)
|
project: dozens of friends are using it (no exact numbers), I got <a href="https://github.com/hannesm/jackline/graphs/contributors">contributions from other people</a>
|
||||||
(more than 25 commits from more than 8 individuals), I use it on a daily basis
|
(more than 25 commits from more than 8 individuals), I use it on a daily basis
|
||||||
for lots of personal communication.
|
for lots of personal communication.</p>
|
||||||
|
<h2 id="what-is-xmpp">What is XMPP?</h2>
|
||||||
## What is XMPP?
|
<p>The eXtensible Messaging and Presence Protocol (previously known as Jabber)
|
||||||
|
describes (these days as <a href="https://tools.ietf.org/html/rfc6120">RFC 6120</a>) a
|
||||||
The eXtensible Messaging and Presence Protocol (previously known as Jabber)
|
|
||||||
describes (these days as [RFC 6120](https://tools.ietf.org/html/rfc6120)) a
|
|
||||||
communication protocol based on XML fragments, which enables near real-time
|
communication protocol based on XML fragments, which enables near real-time
|
||||||
exchange of structured (and extensible) data between two network entities.
|
exchange of structured (and extensible) data between two network entities.</p>
|
||||||
|
<p>The landscape of instant messaging used to contain ICQ, AOL instant messenger,
|
||||||
The landscape of instant messaging used to contain ICQ, AOL instant messenger,
|
|
||||||
and MSN messenger. In 1999, people defined a completely open protocol standard,
|
and MSN messenger. In 1999, people defined a completely open protocol standard,
|
||||||
then named Jabber, since 2011 official RFCs. It is a federated (similar to
|
then named Jabber, since 2011 official RFCs. It is a federated (similar to
|
||||||
eMail) near-real time extensible messaging system (including presence
|
eMail) near-real time extensible messaging system (including presence
|
||||||
information) used for instant messaging. Extensions include end-to-end
|
information) used for instant messaging. Extensions include end-to-end
|
||||||
encryption, multi-user chat, audio transport, ... Unicode support is builtin,
|
encryption, multi-user chat, audio transport, ... Unicode support is builtin,
|
||||||
everything is UTF8 encoded.
|
everything is UTF8 encoded.</p>
|
||||||
|
<p>There are various open jabber servers where people can register accounts, as
|
||||||
There are various open jabber servers where people can register accounts, as
|
|
||||||
well as closed ones. Google Talk used to federate (until 2014) into XMPP,
|
well as closed ones. Google Talk used to federate (until 2014) into XMPP,
|
||||||
Facebook chat used to be based on XMPP. Those big companies wanted something
|
Facebook chat used to be based on XMPP. Those big companies wanted something
|
||||||
"more usable" (where they're more in control, reliable message delivery via
|
"more usable" (where they're more in control, reliable message delivery via
|
||||||
caching in the server and mandatory delivery receipts, multiple devices all
|
caching in the server and mandatory delivery receipts, multiple devices all
|
||||||
getting the same messages), and thus moved away from the open standard.
|
getting the same messages), and thus moved away from the open standard.</p>
|
||||||
|
<h3 id="xmpp-security">XMPP Security</h3>
|
||||||
### XMPP Security
|
<p>Authentication is done via a TLS channel (where your client should authenticate
|
||||||
|
|
||||||
Authentication is done via a TLS channel (where your client should authenticate
|
|
||||||
the server), and SASL that the server authenticates your client. I
|
the server), and SASL that the server authenticates your client. I
|
||||||
[investigated in 2008](https://berlin.ccc.de/~hannes/secure-instant-messaging.pdf) (in German)
|
<a href="https://berlin.ccc.de/~hannes/secure-instant-messaging.pdf">investigated in 2008</a> (in German)
|
||||||
which clients and servers use which authentication methods (I hope the state of
|
which clients and servers use which authentication methods (I hope the state of
|
||||||
certificate verification improved in the last decade).
|
certificate verification improved in the last decade).</p>
|
||||||
|
<p>End-to-end encryption is achievable using OpenPGP (rarely used in my group of
|
||||||
End-to-end encryption is achievable using OpenPGP (rarely used in my group of
|
friends) via XMPP, or <a href="https://otr.cypherpunks.ca/">Off-the-record</a>, which was
|
||||||
friends) via XMPP, or [Off-the-record](https://otr.cypherpunks.ca/), which was
|
|
||||||
pioneered over XMPP, and is still in wide use - it gave rise to forward secrecy:
|
pioneered over XMPP, and is still in wide use - it gave rise to forward secrecy:
|
||||||
if your long-term (stored on disk) asymmetric keys get seized or stolen, they
|
if your long-term (stored on disk) asymmetric keys get seized or stolen, they
|
||||||
are not sufficient to decrypt recorded sessions (you can't derive the session
|
are not sufficient to decrypt recorded sessions (you can't derive the session
|
||||||
key from the asymmetric keys) -- but the encrypted channel is still
|
key from the asymmetric keys) -- but the encrypted channel is still
|
||||||
authenticated (once you verified the public key via a different channel or a
|
authenticated (once you verified the public key via a different channel or a
|
||||||
shared secret, using the [Socialist millionaires problem](https://en.wikipedia.org/wiki/Socialist_millionaire)).
|
shared secret, using the <a href="https://en.wikipedia.org/wiki/Socialist_millionaire">Socialist millionaires problem</a>).</p>
|
||||||
|
<p>OTR does not support offline messages (the session keys may already be destroyed
|
||||||
OTR does not support offline messages (the session keys may already be destroyed
|
|
||||||
by the time the communication partner reconnects and receives the stored
|
by the time the communication partner reconnects and receives the stored
|
||||||
messages), and thus recently [omemo](https://conversations.im/omemo/) was
|
messages), and thus recently <a href="https://conversations.im/omemo/">omemo</a> was
|
||||||
developed. Other messaging protocols (Signal, Threema) are not really open,
|
developed. Other messaging protocols (Signal, Threema) are not really open,
|
||||||
support no federation, but have good support for group encryption and offline
|
support no federation, but have good support for group encryption and offline
|
||||||
messaging. (There is a [nice overview over secure messaging and threats.](https://www.cypherpunks.ca/~iang/pubs/secmessaging-oakland15.pdf))
|
messaging. (There is a <a href="https://www.cypherpunks.ca/~iang/pubs/secmessaging-oakland15.pdf">nice overview over secure messaging and threats.</a>)</p>
|
||||||
|
<p>There is (AFAIK) no encrypted group messaging via XMPP; also the XMPP server
|
||||||
There is (AFAIK) no encrypted group messaging via XMPP; also the XMPP server
|
|
||||||
contains lots of sensible data: your address book (buddy list), together with
|
contains lots of sensible data: your address book (buddy list), together with
|
||||||
offline messages, nicknames you gave to your buddies, subscription information,
|
offline messages, nicknames you gave to your buddies, subscription information,
|
||||||
and information every time you connect (research of privacy preserving presence
|
and information every time you connect (research of privacy preserving presence
|
||||||
protocols has been done, but is not widely used AFAIK,
|
protocols has been done, but is not widely used AFAIK,
|
||||||
e.g. [DP5](http://cacr.uwaterloo.ca/techreports/2014/cacr2014-10.pdf)).
|
e.g. <a href="http://cacr.uwaterloo.ca/techreports/2014/cacr2014-10.pdf">DP5</a>).</p>
|
||||||
|
<h3 id="xmpp-client-landscape">XMPP client landscape</h3>
|
||||||
### XMPP client landscape
|
<p>See <a href="https://en.wikipedia.org/wiki/Comparison_of_XMPP_clients">wikipedia</a> for an
|
||||||
|
extensive comparison (which does not mention jackline :P).</p>
|
||||||
See [wikipedia](https://en.wikipedia.org/wiki/Comparison_of_XMPP_clients) for an
|
<p>A more opinionated analysis is that you were free to choose between C - where
|
||||||
extensive comparison (which does not mention jackline :P).
|
|
||||||
|
|
||||||
A more opinionated analysis is that you were free to choose between C - where
|
|
||||||
all code has to do manual memory management and bounds checking - with ncurses
|
all code has to do manual memory management and bounds checking - with ncurses
|
||||||
(or GTK) and OpenSSL (or GnuTLS) using libpurple (or some other barely
|
(or GTK) and OpenSSL (or GnuTLS) using libpurple (or some other barely
|
||||||
maintained library which tries to unify all instant messaging protocols), or
|
maintained library which tries to unify all instant messaging protocols), or
|
||||||
|
|
@ -103,39 +82,33 @@ Python - where you barely know upfront what it will do at runtime - with GTK and
|
||||||
some OpenSSL, or even JavaScript - where external scripts can dynamically modify
|
some OpenSSL, or even JavaScript - where external scripts can dynamically modify
|
||||||
the prototype of everything at runtime (and thus modify code arbitrarily,
|
the prototype of everything at runtime (and thus modify code arbitrarily,
|
||||||
violating invariants) - calling out to C libraries (NSS, maybe libpurple, who
|
violating invariants) - calling out to C libraries (NSS, maybe libpurple, who
|
||||||
knows?).
|
knows?).</p>
|
||||||
|
<p>Due to complex APIs of transport layer security, certificate verification is
|
||||||
Due to complex APIs of transport layer security, certificate verification is
|
<a href="https://pidgin.im/news/security/?id=91">still not always done correctly</a> (that's just
|
||||||
[still not always done correctly](https://pidgin.im/news/security/?id=91) (that's just
|
|
||||||
one example, you'll find more) - even if, it may not allow custom trust anchors
|
one example, you'll find more) - even if, it may not allow custom trust anchors
|
||||||
or certificate fingerprint based verification - which are crucial for a
|
or certificate fingerprint based verification - which are crucial for a
|
||||||
federated operations without a centralised trust authority.
|
federated operations without a centralised trust authority.</p>
|
||||||
|
<p>Large old code basis usually gather dust and getting bitrot - and if you add
|
||||||
Large old code basis usually gather dust and getting bitrot - and if you add
|
|
||||||
patch by patch from random people on the Internet, you've to deal with the most
|
patch by patch from random people on the Internet, you've to deal with the most
|
||||||
common bug: insufficient checking of input (or output data, [if you encrypt only the plain body, but not the marked up one](https://dev.gajim.org/gajim/gajim-plugins/issues/145)). In some
|
common bug: insufficient checking of input (or output data, <a href="https://dev.gajim.org/gajim/gajim-plugins/issues/145">if you encrypt only the plain body, but not the marked up one</a>). In some
|
||||||
programming languages this easily [leads to execution of remote code](https://pidgin.im/news/security/?id=64), other programming languages steal the
|
programming languages this easily <a href="https://pidgin.im/news/security/?id=64">leads to execution of remote code</a>, other programming languages steal the
|
||||||
work from programmers by deploying automated memory management (finally machines
|
work from programmers by deploying automated memory management (finally machines
|
||||||
take our work away! :)) - also named garbage collection, often used together
|
take our work away! :)) - also named garbage collection, often used together
|
||||||
with automated bounds checking -- this doesn't mean that you're safe - there are
|
with automated bounds checking -- this doesn't mean that you're safe - there are
|
||||||
still logical flaws, and integer overflows (and funny things which happen at
|
still logical flaws, and integer overflows (and funny things which happen at
|
||||||
resource starvation), etc.
|
resource starvation), etc.</p>
|
||||||
|
<h3 id="goals-and-non-goals">Goals and non-goals</h3>
|
||||||
### Goals and non-goals
|
<p>My upfront motivation was to write and use an XMPP client tailored to my needs.
|
||||||
|
|
||||||
My upfront motivation was to write and use an XMPP client tailored to my needs.
|
|
||||||
I personally don't use many graphical applications (coding in emacs, mail via
|
I personally don't use many graphical applications (coding in emacs, mail via
|
||||||
thunderbird, firefox, mplayer, mupdf), but stick mostly to terminal
|
thunderbird, firefox, mplayer, mupdf), but stick mostly to terminal
|
||||||
applications. I additionally don't use any terminal multiplexer (saw too many
|
applications. I additionally don't use any terminal multiplexer (saw too many
|
||||||
active `screen` sessions on remote servers where people left root shells open).
|
active <code>screen</code> sessions on remote servers where people left root shells open).</p>
|
||||||
|
<p>The <a href="https://github.com/hannesm/jackline/commit/9322ceefa9a331fa92a6bf253e8d8f010da2229c">goal was from the beginning</a>
|
||||||
The [goal was from the beginning](https://github.com/hannesm/jackline/commit/9322ceefa9a331fa92a6bf253e8d8f010da2229c)
|
to write a "minimalistic graphical user interface for a secure (fail hard)
|
||||||
to write a "minimalistic graphical user interface for a secure (fail hard)
|
and trustworthy XMPP client". By <em>fail hard</em> I mean exactly that: if it can't
|
||||||
and trustworthy XMPP client". By *fail hard* I mean exactly that: if it can't
|
|
||||||
authenticate the server, don't send the password. If there is no
|
authenticate the server, don't send the password. If there is no
|
||||||
end-to-end encrypted session, don't send the message.
|
end-to-end encrypted session, don't send the message.</p>
|
||||||
|
<p>As a user of (unreleased) software, there is a single property which I like to
|
||||||
As a user of (unreleased) software, there is a single property which I like to
|
|
||||||
preserve: continue to support all data written to persistent storage. Even
|
preserve: continue to support all data written to persistent storage. Even
|
||||||
during large refactorings, ensure that data on the user's disk will also be
|
during large refactorings, ensure that data on the user's disk will also be
|
||||||
correctly parsed. There is nothing worse than having to manually configure an
|
correctly parsed. There is nothing worse than having to manually configure an
|
||||||
|
|
@ -143,57 +116,48 @@ application after update. The solution is straightforward: put a version in
|
||||||
every file you write, and keep readers for all versions ever written around.
|
every file you write, and keep readers for all versions ever written around.
|
||||||
My favourite marshalling format (human readable, structured) are still
|
My favourite marshalling format (human readable, structured) are still
|
||||||
S-expressions - luckily there is a
|
S-expressions - luckily there is a
|
||||||
[sexplib](https://github.com/janestreet/sexplib) in OCaml for handling these.
|
<a href="https://github.com/janestreet/sexplib">sexplib</a> in OCaml for handling these.
|
||||||
Additionally, once the initial configuration file has been created (e.g. interactively with the application), the application
|
Additionally, once the initial configuration file has been created (e.g. interactively with the application), the application
|
||||||
does no further writes to the config file. Users can make arbitrary modifications to the file,
|
does no further writes to the config file. Users can make arbitrary modifications to the file,
|
||||||
and restart the application (and they can make changes while the application is running).
|
and restart the application (and they can make changes while the application is running).</p>
|
||||||
|
<p>I also appreciate another property of software: don't ever transmit any data or
|
||||||
I also appreciate another property of software: don't ever transmit any data or
|
|
||||||
open a network connection unless initiated by the user (this means no autoconnect on startup, or user is typing indications). Don't be obviously
|
open a network connection unless initiated by the user (this means no autoconnect on startup, or user is typing indications). Don't be obviously
|
||||||
fingerprintable. A more mainstream demand is surely that software should not
|
fingerprintable. A more mainstream demand is surely that software should not
|
||||||
phone home - that's why I don't know how many people are using jackline, reports
|
phone home - that's why I don't know how many people are using jackline, reports
|
||||||
based on friends opinions are hundreds of users, I personally know at least
|
based on friends opinions are hundreds of users, I personally know at least
|
||||||
several dozens.
|
several dozens.</p>
|
||||||
|
<p>As written <a href="/Posts/OperatingSystem">earlier</a>, I often take
|
||||||
As written [earlier](/Posts/OperatingSystem), I often take
|
|
||||||
a look at the trusted computing base of a computer system. Jackline's trusted
|
a look at the trusted computing base of a computer system. Jackline's trusted
|
||||||
computing base consists of the client software itself, its OCaml dependencies
|
computing base consists of the client software itself, its OCaml dependencies
|
||||||
(including OTR, TLS, tty library, ...), then the OCaml runtime system, which
|
(including OTR, TLS, tty library, ...), then the OCaml runtime system, which
|
||||||
uses some parts of libc, and a whole UNIX kernel underneath -- one goal is to
|
uses some parts of libc, and a whole UNIX kernel underneath -- one goal is to
|
||||||
have jackline running as a unikernel (then you connect via SSH or telnet and
|
have jackline running as a unikernel (then you connect via SSH or telnet and
|
||||||
TLS).
|
TLS).</p>
|
||||||
|
<p>There are only a few features I need in an XMPP client: single account, strict
|
||||||
There are only a few features I need in an XMPP client: single account, strict
|
|
||||||
validation, delivery receipts, notification callback, being able to deal with
|
validation, delivery receipts, notification callback, being able to deal with
|
||||||
friends logged in multiple times with wrongly set priorities - and end-to-end
|
friends logged in multiple times with wrongly set priorities - and end-to-end
|
||||||
encryption. I don't need inline HTML, avatar images, my currently running
|
encryption. I don't need inline HTML, avatar images, my currently running
|
||||||
music, leaking timezone information, etc. I explicitly don't want to import any
|
music, leaking timezone information, etc. I explicitly don't want to import any
|
||||||
private key material from other clients and libraries, because I want to ensure
|
private key material from other clients and libraries, because I want to ensure
|
||||||
that the key was generated by a good random number generator (read [David's blog article](https://mirage.io/blog/mirage-entropy) on randomness and entropy).
|
that the key was generated by a good random number generator (read <a href="https://mirage.io/blog/mirage-entropy">David's blog article</a> on randomness and entropy).</p>
|
||||||
|
<p>The security story is crucial: always do strict certificate validation, fail
|
||||||
The security story is crucial: always do strict certificate validation, fail
|
|
||||||
hard, make it noticable by the user if they're doing insecure communication.
|
hard, make it noticable by the user if they're doing insecure communication.
|
||||||
Only few people are into reading out loud their OTR public key fingerprint, and
|
Only few people are into reading out loud their OTR public key fingerprint, and
|
||||||
SMP is not trivial -- thus jackline records the known public keys together with
|
SMP is not trivial -- thus jackline records the known public keys together with
|
||||||
a set of resources used, a session count, and blurred timestamps (accuracy: day)
|
a set of resources used, a session count, and blurred timestamps (accuracy: day)
|
||||||
when the publickey was initially used and when it was used the last time.
|
when the publickey was initially used and when it was used the last time.</p>
|
||||||
|
<p>I'm pragmatic - if there is some server (or client) deployed out there which
|
||||||
I'm pragmatic - if there is some server (or client) deployed out there which
|
violates (my interpretation of) the specification, I'm happy to <a href="https://github.com/hannesm/ocaml-otr/issues/10">implement workarounds</a>. Initially I
|
||||||
violates (my interpretation of) the specification, I'm happy to [implement workarounds](https://github.com/hannesm/ocaml-otr/issues/10). Initially I
|
worked roughly one day a week on jackline.</p>
|
||||||
worked roughly one day a week on jackline.
|
<p>To not release the software for some years was something I learned from the
|
||||||
|
<a href="https://common-lisp.net/project/slime/">slime</a> project (<a href="https://www.youtube.com/watch?v=eZDWJfB9XY4">watch Luke's presentation from 2013</a>) - if
|
||||||
To not release the software for some years was something I learned from the
|
|
||||||
[slime](https://common-lisp.net/project/slime/) project ([watch Luke's presentation from 2013](https://www.youtube.com/watch?v=eZDWJfB9XY4)) - if
|
|
||||||
there's someone complaining about an issue, fix it within 10 minutes and ask
|
there's someone complaining about an issue, fix it within 10 minutes and ask
|
||||||
them to update. This only works if each user compiles the git version anyways.
|
them to update. This only works if each user compiles the git version anyways.</p>
|
||||||
|
<h2 id="user-interface">User interface</h2>
|
||||||
## User interface
|
<p><img src="/static/img/jackline.png" alt="other screenshot" /></p>
|
||||||
|
<p>Stated goal is <em>minimalistic</em>. No heavy use of colours. Visibility on
|
||||||
|
|
||||||
|
|
||||||
Stated goal is *minimalistic*. No heavy use of colours. Visibility on
|
|
||||||
both black and white background (btw, as a Unix process there is no way to find
|
both black and white background (btw, as a Unix process there is no way to find
|
||||||
out your background colour (or is there?)). The focus is also *security* - and
|
out your background colour (or is there?)). The focus is also <em>security</em> - and
|
||||||
that's where I used colours from the beginning: red is unencrypted (non
|
that's where I used colours from the beginning: red is unencrypted (non
|
||||||
end-to-end, there's always the transport layer encryption) communication, green
|
end-to-end, there's always the transport layer encryption) communication, green
|
||||||
is encrypted communication. Verification status of the public key uses the same
|
is encrypted communication. Verification status of the public key uses the same
|
||||||
|
|
@ -201,11 +165,10 @@ colours: red for not verified, green for verified. Instead of colouring each
|
||||||
message individually, I use the encryption status of the active contact
|
message individually, I use the encryption status of the active contact
|
||||||
(highlighted in the contact list, where messages you type now will be sent to)
|
(highlighted in the contact list, where messages you type now will be sent to)
|
||||||
to colour the entire frame. This results in a remarkable visual indication and
|
to colour the entire frame. This results in a remarkable visual indication and
|
||||||
(at least I) think twice before presssing `return` in a red terminal. Messages
|
(at least I) think twice before presssing <code>return</code> in a red terminal. Messages
|
||||||
were initially white/black, but got a bit fancier over time: incoming messages
|
were initially white/black, but got a bit fancier over time: incoming messages
|
||||||
are bold, multi user messages mentioning your nick are underlined.
|
are bold, multi user messages mentioning your nick are underlined.</p>
|
||||||
|
<p>The graphical design is mainly inspired by mcabber, as mentioned earlier. There
|
||||||
The graphical design is mainly inspired by mcabber, as mentioned earlier. There
|
|
||||||
are four components: the contact list in the upper left, chat window upper
|
are four components: the contact list in the upper left, chat window upper
|
||||||
right, log window on the bottom (surrounded by two status bars), and a readline
|
right, log window on the bottom (surrounded by two status bars), and a readline
|
||||||
input. The sizes are configurable (via commands and key shortcuts). A
|
input. The sizes are configurable (via commands and key shortcuts). A
|
||||||
|
|
@ -213,172 +176,162 @@ different view is having the chat window fullscreen (or only the received
|
||||||
messages) - useful for copy and pasting fragments. Navigation is done in the
|
messages) - useful for copy and pasting fragments. Navigation is done in the
|
||||||
contact list. There is a single active contact (colours are inverted in the
|
contact list. There is a single active contact (colours are inverted in the
|
||||||
contact list, and the contact is mentioned in the status bar), whose chat
|
contact list, and the contact is mentioned in the status bar), whose chat
|
||||||
messages are displayed.
|
messages are displayed.</p>
|
||||||
|
<p>There is not much support for customisation - some people demanded to have a
|
||||||
There is not much support for customisation - some people demanded to have a
|
|
||||||
7bit ASCII version (I output some unicode characters for layout). Recently I
|
7bit ASCII version (I output some unicode characters for layout). Recently I
|
||||||
added support to customise the colours. I tried to ensure it looks fine on both
|
added support to customise the colours. I tried to ensure it looks fine on both
|
||||||
black and white background.
|
black and white background.</p>
|
||||||
|
<h2 id="code">Code</h2>
|
||||||
## Code
|
<p>Initially I targeted GTK with OCaml, but that excursion only lasted <a href="https://github.com/hannesm/jackline/commit/17b674130f7b1fcf2542eb5e0911a40b81fc724e">two weeks</a>,
|
||||||
|
when I switched to a <a href="https://github.com/diml/lambda-term">lambda-term</a> terminal
|
||||||
Initially I targeted GTK with OCaml, but that excursion only lasted [two weeks](https://github.com/hannesm/jackline/commit/17b674130f7b1fcf2542eb5e0911a40b81fc724e),
|
interface.</p>
|
||||||
when I switched to a [lambda-term](https://github.com/diml/lambda-term) terminal
|
<h3 id="ui">UI</h3>
|
||||||
interface.
|
<p>The lambda-term interface survived for a good year (until <a href="https://github.com/hannesm/jackline/pull/117">7th Feb 2016</a>),
|
||||||
|
when I started to use <a href="https://github.com/pqwy/notty">notty</a> - developed by
|
||||||
### UI
|
David - using a decent <a href="http://erratique.ch/software/uutf">unicode library</a>.</p>
|
||||||
|
<p>Notty back then was under heavy development, I spend several hours rebasing
|
||||||
The lambda-term interface survived for a good year (until [7th Feb 2016](https://github.com/hannesm/jackline/pull/117)),
|
|
||||||
when I started to use [notty](https://github.com/pqwy/notty) - developed by
|
|
||||||
David - using a decent [unicode library](http://erratique.ch/software/uutf).
|
|
||||||
|
|
||||||
Notty back then was under heavy development, I spend several hours rebasing
|
|
||||||
jackline to updates in notty. What I got out of it is proper unicode support:
|
jackline to updates in notty. What I got out of it is proper unicode support:
|
||||||
the symbol 茶 gets two characters width (see screenshot at top of page), and the layouting keeps track
|
the symbol 茶 gets two characters width (see screenshot at top of page), and the layouting keeps track
|
||||||
how many characters are already written on the terminal.
|
how many characters are already written on the terminal.</p>
|
||||||
|
<p>I recommend to look into <a href="https://github.com/pqwy/notty">notty</a> if you want to
|
||||||
I recommend to look into [notty](https://github.com/pqwy/notty) if you want to
|
do terminal graphics in OCaml!</p>
|
||||||
do terminal graphics in OCaml!
|
<h3 id="application-logic-and-state">Application logic and state</h3>
|
||||||
|
<p>Stepping back, an XMPP client reacts to two input sources: the user input
|
||||||
### Application logic and state
|
|
||||||
|
|
||||||
Stepping back, an XMPP client reacts to two input sources: the user input
|
|
||||||
(including terminal resize), and network input (or failure). The output is a
|
(including terminal resize), and network input (or failure). The output is a
|
||||||
screen (80x25 characters) image. Each input event can trigger output events on
|
screen (80x25 characters) image. Each input event can trigger output events on
|
||||||
the display and the network.
|
the display and the network.</p>
|
||||||
|
<p>I used to use multiple threads and locking between shared data for these kinds
|
||||||
I used to use multiple threads and locking between shared data for these kinds
|
|
||||||
of applications: there can go something wrong when network and user input
|
of applications: there can go something wrong when network and user input
|
||||||
happens at the same time, or what if the output is interrupted by more input
|
happens at the same time, or what if the output is interrupted by more input
|
||||||
(which happens e.g. during copy and paste).
|
(which happens e.g. during copy and paste).</p>
|
||||||
|
<p>Initially I used lots of shared data and had hope, but this was clearly not a
|
||||||
Initially I used lots of shared data and had hope, but this was clearly not a
|
|
||||||
good solution. Nowadays I use mailboxes, and separate tasks which wait for
|
good solution. Nowadays I use mailboxes, and separate tasks which wait for
|
||||||
receiving a message: one task which writes persistent data (session counts,
|
receiving a message: one task which writes persistent data (session counts,
|
||||||
verified fingerprints) periodically to ask, another which writes on change to
|
verified fingerprints) periodically to ask, another which writes on change to
|
||||||
disk, an error handler
|
disk, an error handler
|
||||||
([`init_system`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/bin/jackline.ml#L3-L25))
|
(<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/bin/jackline.ml#L3-L25"><code>init_system</code></a>)
|
||||||
which resets the state upon a connection failure, another task which waits for
|
which resets the state upon a connection failure, another task which waits for
|
||||||
user input
|
user input
|
||||||
([`read_terminal`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_input.ml#L169)),
|
(<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_input.ml#L169"><code>read_terminal</code></a>),
|
||||||
one waiting for network input ([`Connect`, including reconnecting timers](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L202)),
|
one waiting for network input (<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L202"><code>Connect</code>, including reconnecting timers</a>),
|
||||||
one to call out the notification hooks
|
one to call out the notification hooks
|
||||||
([`Notify`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L100)),
|
(<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L100"><code>Notify</code></a>),
|
||||||
etc. The main task is simple: wait for input, process input (producing a new
|
etc. The main task is simple: wait for input, process input (producing a new
|
||||||
state), render the state, and recursively call itself
|
state), render the state, and recursively call itself
|
||||||
([`loop`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_client.ml#L371)).
|
(<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_client.ml#L371"><code>loop</code></a>).</p>
|
||||||
|
<p>Only recently I solved the copy and paste issue by <a href="https://github.com/hannesm/jackline/commit/cab34acab004023911997ec9aee8b00a976af7e4">delaying all redraws by 40ms</a>,
|
||||||
Only recently I solved the copy and paste issue by [delaying all redraws by 40ms](https://github.com/hannesm/jackline/commit/cab34acab004023911997ec9aee8b00a976af7e4),
|
and canceling if another redraw is scheduled.</p>
|
||||||
and canceling if another redraw is scheduled.
|
<p>The whole
|
||||||
|
<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L29-L58"><code>state</code></a>
|
||||||
The whole
|
contains some user interface parameters (<code>/buddywith</code>, <code>/logheight</code>, ..), as
|
||||||
[`state`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L29-L58)
|
|
||||||
contains some user interface parameters (`/buddywith`, `/logheight`, ..), as
|
|
||||||
well as the contact map, which contain users, which have sessions, each
|
well as the contact map, which contain users, which have sessions, each
|
||||||
containing chat messages.
|
containing chat messages.</p>
|
||||||
|
<p>The code base is just below 6000 lines of code (way too big ;), and nowadays
|
||||||
The code base is just below 6000 lines of code (way too big ;), and nowadays
|
supports multi-user chat, sane multi-resource interaction (press <code>enter</code> to show
|
||||||
supports multi-user chat, sane multi-resource interaction (press `enter` to show
|
|
||||||
all available resources of a contact and message each individually in case you
|
all available resources of a contact and message each individually in case you
|
||||||
need to), configurable colours, tab completions for nicknames and commands,
|
need to), configurable colours, tab completions for nicknames and commands,
|
||||||
per-user input history, emacs keybindings. It even works with the XMPP gateway
|
per-user input history, emacs keybindings. It even works with the XMPP gateway
|
||||||
provided by slack (some startup doing a centralised groupchat with picture embedding and
|
provided by slack (some startup doing a centralised groupchat with picture embedding and
|
||||||
animated cats).
|
animated cats).</p>
|
||||||
|
<h3 id="road-ahead">Road ahead</h3>
|
||||||
### Road ahead
|
<p>Common feature requests are: <a href="https://github.com/hannesm/jackline/issues/153">omemo support</a>,
|
||||||
|
<a href="https://github.com/hannesm/jackline/issues/104">IRC support</a>,
|
||||||
Common feature requests are: [omemo support](https://github.com/hannesm/jackline/issues/153),
|
<a href="https://github.com/hannesm/jackline/issues/115">support for multiple accounts</a>
|
||||||
[IRC support](https://github.com/hannesm/jackline/issues/104),
|
|
||||||
[support for multiple accounts](https://github.com/hannesm/jackline/issues/115)
|
|
||||||
(tbh, these are all
|
(tbh, these are all
|
||||||
things I'd like to have as well).
|
things I'd like to have as well).</p>
|
||||||
|
<p>But there's some mess to clean up:</p>
|
||||||
But there's some mess to clean up:
|
<ol>
|
||||||
|
<li>
|
||||||
1. The [XMPP library](https://github.com/ermine/xmpp) makes heavy use of
|
<p>The <a href="https://github.com/ermine/xmpp">XMPP library</a> makes heavy use of
|
||||||
functors (to abstract over the concrete IO, etc.), and embeds IO deep inside it.
|
functors (to abstract over the concrete IO, etc.), and embeds IO deep inside it.
|
||||||
I do prefer (see e.g. [our TLS paper](https://usenix15.nqsb.io), or [my ARP post](/Posts/ARP)) these days to have a pure interface for
|
I do prefer (see e.g. <a href="https://usenix15.nqsb.io">our TLS paper</a>, or <a href="/Posts/ARP">my ARP post</a>) these days to have a pure interface for
|
||||||
the protocol implementation, providing explicit input (state, event, data), and
|
the protocol implementation, providing explicit input (state, event, data), and
|
||||||
output (state, action, potentially data to send on network, potentially data to
|
output (state, action, potentially data to send on network, potentially data to
|
||||||
process by the application). The [sasl implementation](https://github.com/hannesm/xmpp/blob/eee18bd3dd343550169969c0b45548eafd51cfe1/src/sasl.ml)
|
process by the application). The <a href="https://github.com/hannesm/xmpp/blob/eee18bd3dd343550169969c0b45548eafd51cfe1/src/sasl.ml">sasl implementation</a>
|
||||||
is partial and deeply embedded. The XML parser is as well deeply embedded (and
|
is partial and deeply embedded. The XML parser is as well deeply embedded (and
|
||||||
[has some issues](https://github.com/hannesm/jackline/issues/8#issuecomment-67773044)).
|
<a href="https://github.com/hannesm/jackline/issues/8#issuecomment-67773044">has some issues</a>).
|
||||||
The library needs to be torn apart (something I procrastinate since more than
|
The library needs to be torn apart (something I procrastinate since more than
|
||||||
a year). Once it is pure, the application can have full control over when to
|
a year). Once it is pure, the application can have full control over when to
|
||||||
call IO (and esp use the same protocol implementation as well for registering a
|
call IO (and esp use the same protocol implementation as well for registering a
|
||||||
new account - [currently not supported](https://github.com/hannesm/jackline/issues/12)).
|
new account - <a href="https://github.com/hannesm/jackline/issues/12">currently not supported</a>).</p>
|
||||||
|
</li>
|
||||||
2. On the frontend side (the `cli` subfolder), there is too much knowledge of
|
<li>
|
||||||
|
<p>On the frontend side (the <code>cli</code> subfolder), there is too much knowledge of
|
||||||
XMPP. It should be more general, and be reusable (some bits and pieces are
|
XMPP. It should be more general, and be reusable (some bits and pieces are
|
||||||
notty utilities, such as wrapping a string to fit into a text box of specific
|
notty utilities, such as wrapping a string to fit into a text box of specific
|
||||||
width, see
|
width, see
|
||||||
[`split_unicode`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_support.ml#L22)).
|
<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_support.ml#L22"><code>split_unicode</code></a>).</p>
|
||||||
|
</li>
|
||||||
3. The command processing engine itself is 1300 lines (including ad-hoc string
|
<li>
|
||||||
|
<p>The command processing engine itself is 1300 lines (including ad-hoc string
|
||||||
parsing)
|
parsing)
|
||||||
([`Cli_commands`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_commands.ml)),
|
(<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_commands.ml"><code>Cli_commands</code></a>),
|
||||||
best to replaced by a more decent command abstraction.
|
best to replaced by a more decent command abstraction.</p>
|
||||||
|
</li>
|
||||||
4. A big record of functions
|
<li>
|
||||||
([`user_data`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/src/xmpp_callbacks.ml#L46))
|
<p>A big record of functions
|
||||||
is passed (during `/connect` in
|
(<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/src/xmpp_callbacks.ml#L46"><code>user_data</code></a>)
|
||||||
[`handle_connect`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_commands.ml#L221-L582))
|
is passed (during <code>/connect</code> in
|
||||||
from the UI to the XMPP task to inject messages and errors.
|
<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_commands.ml#L221-L582"><code>handle_connect</code></a>)
|
||||||
|
from the UI to the XMPP task to inject messages and errors.</p>
|
||||||
5. The global variable
|
</li>
|
||||||
[`xmpp_session`](https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L200)
|
<li>
|
||||||
should be part of the earlier mentioned `cli_state`, also contacts should be a map, not a Hashtbl (took me some time to learn).
|
<p>The global variable
|
||||||
|
<a href="https://github.com/hannesm/jackline/blob/ec8f8c01d6503bf52be263cd319ef21f2b62ff2e/cli/cli_state.ml#L200"><code>xmpp_session</code></a>
|
||||||
6. Having jackline self-hosted as a MirageOS unikernel. I've implemented a a
|
should be part of the earlier mentioned <code>cli_state</code>, also contacts should be a map, not a Hashtbl (took me some time to learn).</p>
|
||||||
[telnet](https://github.com/hannesm/telnet) server, there is a
|
</li>
|
||||||
[notty branch](https://github.com/pqwy/notty/tree/mirage) be used with the telnet
|
<li>
|
||||||
server. But there is (right now) no good story for persistent mutable storage.
|
<p>Having jackline self-hosted as a MirageOS unikernel. I've implemented a a
|
||||||
|
<a href="https://github.com/hannesm/telnet">telnet</a> server, there is a
|
||||||
7. Jackline predates some very elegant libraries, such as
|
<a href="https://github.com/pqwy/notty/tree/mirage">notty branch</a> be used with the telnet
|
||||||
[logs](http://erratique.ch/software/logs) and
|
server. But there is (right now) no good story for persistent mutable storage.</p>
|
||||||
[astring](http://erratique.ch/software/astring), even
|
</li>
|
||||||
[result](http://caml.inria.fr/pub/docs/manual-ocaml/libref/Pervasives.html#TYPEresult) - since 4.03 part of Pervasives - is not used.
|
<li>
|
||||||
Clearly, other libraries (such as TLS) do not yet use result.
|
<p>Jackline predates some very elegant libraries, such as
|
||||||
|
<a href="http://erratique.ch/software/logs">logs</a> and
|
||||||
8. After looking in more depths at the logs library, and at user interfaces - I
|
<a href="http://erratique.ch/software/astring">astring</a>, even
|
||||||
|
<a href="http://caml.inria.fr/pub/docs/manual-ocaml/libref/Pervasives.html#TYPEresult">result</a> - since 4.03 part of Pervasives - is not used.
|
||||||
|
Clearly, other libraries (such as TLS) do not yet use result.</p>
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<p>After looking in more depths at the logs library, and at user interfaces - I
|
||||||
envision the graphical parts to be (mostly!?) a viewer of logs, and a command
|
envision the graphical parts to be (mostly!?) a viewer of logs, and a command
|
||||||
shell (using a control interface, maybe
|
shell (using a control interface, maybe
|
||||||
[9p](https://github.com/mirage/ocaml-9p/)): Multiple layers (of a protocol),
|
<a href="https://github.com/mirage/ocaml-9p/">9p</a>): Multiple layers (of a protocol),
|
||||||
slightly related (by [tags](http://erratique.ch/software/logs/doc/Logs.Tag.html) - such as the OTR session), and have the layers be visible to users (see also
|
slightly related (by <a href="http://erratique.ch/software/logs/doc/Logs.Tag.html">tags</a> - such as the OTR session), and have the layers be visible to users (see also
|
||||||
[tlstools](https://github.com/mirleft/tlstools)), a slightly different interface
|
<a href="https://github.com/mirleft/tlstools">tlstools</a>), a slightly different interface
|
||||||
of similarly structured data. In jackline I'd like to e.g. see all messages of
|
of similarly structured data. In jackline I'd like to e.g. see all messages of
|
||||||
a single OTR session (see [issue](https://github.com/hannesm/jackline/issues/111)), or hide the presence messages in a multi-user chat,
|
a single OTR session (see <a href="https://github.com/hannesm/jackline/issues/111">issue</a>), or hide the presence messages in a multi-user chat,
|
||||||
investigate the high-level message, its XML encoded stanza, TLS encrypted
|
investigate the high-level message, its XML encoded stanza, TLS encrypted
|
||||||
frames, the TCP flow, all down to the ethernet frames send over the wire - also
|
frames, the TCP flow, all down to the ethernet frames send over the wire - also
|
||||||
viewable as sequence diagram and other suitable (terminal) presentations (TCP
|
viewable as sequence diagram and other suitable (terminal) presentations (TCP
|
||||||
window size maybe in a size over time diagram).
|
window size maybe in a size over time diagram).</p>
|
||||||
|
</li>
|
||||||
9. Once the API between the sources (contacts, hosts) and the UI (what to
|
<li>
|
||||||
|
<p>Once the API between the sources (contacts, hosts) and the UI (what to
|
||||||
display, where and how to trigger notifications, where and how to handle global
|
display, where and how to trigger notifications, where and how to handle global
|
||||||
changes (such as reconnect)) is clear and implemented, commands need to be
|
changes (such as reconnect)) is clear and implemented, commands need to be
|
||||||
reinvented (some, such as navigation commands and emacs keybindings, are generic
|
reinvented (some, such as navigation commands and emacs keybindings, are generic
|
||||||
to the user interface, others are specific to XMPP and/or OTR): a new transport
|
to the user interface, others are specific to XMPP and/or OTR): a new transport
|
||||||
(IRC) or end-to-end crypto protocol (omemo) - should be easy to integrate (with
|
(IRC) or end-to-end crypto protocol (omemo) - should be easy to integrate (with
|
||||||
similar minimal UI features and colours).
|
similar minimal UI features and colours).</p>
|
||||||
|
</li>
|
||||||
### Conclusion
|
</ol>
|
||||||
|
<h3 id="conclusion">Conclusion</h3>
|
||||||
Jackline started as a procrastination project, and still is one. I only develop
|
<p>Jackline started as a procrastination project, and still is one. I only develop
|
||||||
on jackline if I enjoy it. I'm not scared to try new approaches in jackline,
|
on jackline if I enjoy it. I'm not scared to try new approaches in jackline,
|
||||||
and either reverting them or rewriting some chunks of code again. It is a
|
and either reverting them or rewriting some chunks of code again. It is a
|
||||||
project where I publish early and push often. I've met several people (whom I
|
project where I publish early and push often. I've met several people (whom I
|
||||||
don't think I know personally) in the multi-user chatroom
|
don't think I know personally) in the multi-user chatroom
|
||||||
`jackline@conference.jabber.ccc.de`, and fixed bugs, discussed features.
|
<code>jackline@conference.jabber.ccc.de</code>, and fixed bugs, discussed features.</p>
|
||||||
|
<p>When introducing <a href="https://github.com/hannesm/jackline/commit/40bec5efba81061cc41df891cadd282120e16816">customisable colours</a>,
|
||||||
When introducing [customisable colours](https://github.com/hannesm/jackline/commit/40bec5efba81061cc41df891cadd282120e16816),
|
|
||||||
the proximity to a log viewer became again clear to me - configurable colours
|
the proximity to a log viewer became again clear to me - configurable colours
|
||||||
are for severities such as `Success`, `Warning`, `Info`, `Error`, `Presence` -
|
are for severities such as <code>Success</code>, <code>Warning</code>, <code>Info</code>, <code>Error</code>, <code>Presence</code> -
|
||||||
maybe I really should get started on implementing a log viewer.
|
maybe I really should get started on implementing a log viewer.</p>
|
||||||
|
<p>I would like to have more community contributions to jackline, but the lack of
|
||||||
I would like to have more community contributions to jackline, but the lack of
|
|
||||||
documentation (there aren't even a lot of interface files), mixed with a
|
documentation (there aren't even a lot of interface files), mixed with a
|
||||||
non-mainstream programming language, and a convoluted code base, makes me want
|
non-mainstream programming language, and a convoluted code base, makes me want
|
||||||
some code cleanups first, or maybe starting from scratch.
|
some code cleanups first, or maybe starting from scratch.</p>
|
||||||
|
<p>I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a> or
|
||||||
I'm interested in feedback, either via [twitter](https://twitter.com/h4nnes) or
|
on the <a href="https://github.com/hannesm/jackline">jackline repository on GitHub</a>.</p>
|
||||||
on the [jackline repository on GitHub](https://github.com/hannesm/jackline).
|
</article></div></div></main></body></html>
|
||||||
|
|
@ -1,103 +1,77 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Who maintains package X?
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Who maintains package X?</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Who maintains package X?" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Who maintains package X?</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/package signing" class="tag">package signing</a><a href="/tags/security" class="tag">security</a></div><span class="date">Published: 2017-02-16 (last updated: 2017-03-09)</span><article><p>A very important data point for conex, the new opam signing utility, is who is authorised for a given package. We
|
||||||
author: hannes
|
|
||||||
tags: package signing, security
|
|
||||||
abstract: We describe why manual gathering of metadata is out of date, and version control systems are awesome.
|
|
||||||
---
|
|
||||||
|
|
||||||
A very important data point for conex, the new opam signing utility, is who is authorised for a given package. We
|
|
||||||
could have written this manually down, or force each author to create a
|
could have written this manually down, or force each author to create a
|
||||||
pull request for their packages, but this would be a long process and not
|
pull request for their packages, but this would be a long process and not
|
||||||
easy: the main opam repository has around 1500 unique packages, and 350
|
easy: the main opam repository has around 1500 unique packages, and 350
|
||||||
contributors. Fortunately, it is a git repository with 5 years of history, and
|
contributors. Fortunately, it is a git repository with 5 years of history, and
|
||||||
over 6900 pull requests. Each opam file may also contain a `maintainers` entry,
|
over 6900 pull requests. Each opam file may also contain a <code>maintainers</code> entry,
|
||||||
a list of strings (usually a mail address).
|
a list of strings (usually a mail address).</p>
|
||||||
|
<p>The data sources we correlate are the <code>maintainers</code> entry in opam file, and who
|
||||||
The data sources we correlate are the `maintainers` entry in opam file, and who
|
actually committed in the opam repository. This is inspired by <a href="https://github.com/ocaml/opam/issues/2693">some GitHub
|
||||||
actually committed in the opam repository. This is inspired by [some GitHub
|
discussion</a>.</p>
|
||||||
discussion](https://github.com/ocaml/opam/issues/2693).
|
<h3 id="github-id-and-email-address">GitHub id and email address</h3>
|
||||||
|
<p>For simplicity, since conex uses any (unique) identifier for authors, and the opam
|
||||||
### GitHub id and email address
|
|
||||||
|
|
||||||
For simplicity, since conex uses any (unique) identifier for authors, and the opam
|
|
||||||
repository is hosted on GitHub, we use a GitHub id as author identifier.
|
repository is hosted on GitHub, we use a GitHub id as author identifier.
|
||||||
Maintainer information is an email address, thus we need a mapping between them.
|
Maintainer information is an email address, thus we need a mapping between them.</p>
|
||||||
|
<p>We wrote a <a href="https://raw.githubusercontent.com/hannesm/conex/master/analysis/loop-prs.sh">shell
|
||||||
We wrote a [shell
|
script</a>
|
||||||
script](https://raw.githubusercontent.com/hannesm/conex/master/analysis/loop-prs.sh)
|
|
||||||
to find all PR merges, their GitHub id (in a brittle way: using the name of the
|
to find all PR merges, their GitHub id (in a brittle way: using the name of the
|
||||||
git remote), and email address of the last commit. It also saves a diff of the
|
git remote), and email address of the last commit. It also saves a diff of the
|
||||||
PR for later. This results in 6922 PRs (opam repository version 38d908dcbc58d07467fbc00698083fa4cbd94f9d).
|
PR for later. This results in 6922 PRs (opam repository version 38d908dcbc58d07467fbc00698083fa4cbd94f9d).</p>
|
||||||
|
<p>The metadata output is processed by
|
||||||
The metadata output is processed by
|
<a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L134-L156">github_mail</a>:
|
||||||
[github_mail](https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L134-L156):
|
we ignore PRs from GitHub organisations <code>PR.ignore_github</code>, where commits
|
||||||
we ignore PRs from GitHub organisations `PR.ignore_github`, where commits
|
<code>PR.ignore_pr</code> are picked from a different author (manually), bad mail addresses,
|
||||||
`PR.ignore_pr` are picked from a different author (manually), bad mail addresses,
|
and <a href="https://github.com/yallop">Jeremy's</a> mail address (it is added to too many GitHub ids otherwise). The
|
||||||
and [Jeremy's](https://github.com/yallop) mail address (it is added to too many GitHub ids otherwise). The
|
goal is to have a for an email address a single GitHub id. 329 authors with 416 mail addresses are mapped.</p>
|
||||||
goal is to have a for an email address a single GitHub id. 329 authors with 416 mail addresses are mapped.
|
<h3 id="maintainer-in-opam">Maintainer in opam</h3>
|
||||||
|
<p>As mentioned, lots of packages contain a <code>maintainers</code> entry. In
|
||||||
### Maintainer in opam
|
<a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L40-L68"><code>maintainers</code></a>
|
||||||
|
we extract the mail addresses of the <a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L70-L94">most recently released opam
|
||||||
As mentioned, lots of packages contain a `maintainers` entry. In
|
file</a>.
|
||||||
[`maintainers`](https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L40-L68)
|
|
||||||
we extract the mail addresses of the [most recently released opam
|
|
||||||
file](https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L70-L94).
|
|
||||||
Some hardcoded matches are teams which do not properly maintain the maintainers
|
Some hardcoded matches are teams which do not properly maintain the maintainers
|
||||||
field (such as mirage and xapi-project ;). We're open for suggestions to extend
|
field (such as mirage and xapi-project ;). We're open for suggestions to extend
|
||||||
this massaging to the needs. Additionally, the contact at ocamlpro mail address
|
this massaging to the needs. Additionally, the contact at ocamlpro mail address
|
||||||
was used for all packages before the maintainers entry was introduced (based on
|
was used for all packages before the maintainers entry was introduced (based on
|
||||||
a discussion with Louis Gesbert). 132 packages with empty maintainers.
|
a discussion with Louis Gesbert). 132 packages with empty maintainers.</p>
|
||||||
|
<h3 id="fitness">Fitness</h3>
|
||||||
### Fitness
|
<p>Combining these two data sources, we hoped to find a strict small set of whom to
|
||||||
|
|
||||||
Combining these two data sources, we hoped to find a strict small set of whom to
|
|
||||||
authorise for which package. Turns out some people use different mail addresses
|
authorise for which package. Turns out some people use different mail addresses
|
||||||
for git commits and opam maintainer entries, which [are be easily
|
for git commits and opam maintainer entries, which <a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L233-L269">are be easily
|
||||||
fixed](https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L233-L269).
|
fixed</a>.</p>
|
||||||
|
<p>While <a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L169-L205">processing the full diffs of each
|
||||||
While [processing the full diffs of each
|
PR</a>
|
||||||
PR](https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L169-L205)
|
|
||||||
(using the diff parser of conex mentioned above), ignoring the 44% done by
|
(using the diff parser of conex mentioned above), ignoring the 44% done by
|
||||||
[janitors](https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L158-L165)
|
<a href="https://github.com/hannesm/conex/blob/dbdfc5337c97d62edc74f1c546023bcb5e719343/analysis/maintainer.ml#L158-L165">janitors</a>
|
||||||
(a manually created set by looking at log data, please report if wrong), we
|
(a manually created set by looking at log data, please report if wrong), we
|
||||||
categorise the modifications: authorised modification (the GitHub id is
|
categorise the modifications: authorised modification (the GitHub id is
|
||||||
authorised for the package), modification by an author to a team-owned package
|
authorised for the package), modification by an author to a team-owned package
|
||||||
(propose to add this author to the team), modification of a package where no
|
(propose to add this author to the team), modification of a package where no
|
||||||
GitHub id is authorised, and unauthorised modification. We also ignore packages
|
GitHub id is authorised, and unauthorised modification. We also ignore packages
|
||||||
which are no longer in the opam repository.
|
which are no longer in the opam repository.</p>
|
||||||
|
<p>2766 modifications were authorised, 418 were team-owned, 452 were to packages
|
||||||
2766 modifications were authorised, 418 were team-owned, 452 were to packages
|
with no maintainer, and 570 unauthorised. This results in 125 unowned packages.</p>
|
||||||
with no maintainer, and 570 unauthorised. This results in 125 unowned packages.
|
<p>Out of the 452 modifications to packages with no maintainer, 75 are a global
|
||||||
|
one-to-one author to package relation, and are directly authorised.</p>
|
||||||
Out of the 452 modifications to packages with no maintainer, 75 are a global
|
<p>Inference of team members is an overapproximation (everybody who committed
|
||||||
one-to-one author to package relation, and are directly authorised.
|
|
||||||
|
|
||||||
Inference of team members is an overapproximation (everybody who committed
|
|
||||||
changes to their packages), additionally the janitors are missing. We will have
|
changes to their packages), additionally the janitors are missing. We will have
|
||||||
to fill these manually.
|
to fill these manually.</p>
|
||||||
|
<pre><code>alt-ergo -> OCamlPro-Iguernlala UnixJunkie backtracking bobot nobrowser
|
||||||
```
|
janestreet -> backtracking hannesm j0sh rgrinberg smondet
|
||||||
alt-ergo -> OCamlPro-Iguernlala UnixJunkie backtracking bobot nobrowser
|
mirage -> MagnusS dbuenzli djs55 hannesm hnrgrgr jonludlam mato mor1 pgj pqwy pw374 rdicosmo rgrinberg ruhatch sg2342 talex5 yomimono
|
||||||
janestreet -> backtracking hannesm j0sh rgrinberg smondet
|
ocsigen -> balat benozol dbuenzli hhugo hnrgrgr jpdeplaix mfp pveber scjung slegrand45 smondet vasilisp
|
||||||
mirage -> MagnusS dbuenzli djs55 hannesm hnrgrgr jonludlam mato mor1 pgj pqwy pw374 rdicosmo rgrinberg ruhatch sg2342 talex5 yomimono
|
xapi-project -> dbuenzli djs55 euanh mcclurmc rdicosmo simonjbeaumont yomimono
|
||||||
ocsigen -> balat benozol dbuenzli hhugo hnrgrgr jpdeplaix mfp pveber scjung slegrand45 smondet vasilisp
|
</code></pre>
|
||||||
xapi-project -> dbuenzli djs55 euanh mcclurmc rdicosmo simonjbeaumont yomimono
|
<h3 id="alternative-approach-github-urls">Alternative approach: GitHub urls</h3>
|
||||||
```
|
<p>An alternative approach (attempted earlier) working only for GitHub hosted projects, is to authorise
|
||||||
|
<a href="https://github.com/hannesm/conex/blob/github/analysis/maintainer.ml#L37-L91">the use of the user part of the GitHub repository
|
||||||
|
URL</a>.
|
||||||
### Alternative approach: GitHub urls
|
|
||||||
|
|
||||||
An alternative approach (attempted earlier) working only for GitHub hosted projects, is to authorise
|
|
||||||
[the use of the user part of the GitHub repository
|
|
||||||
URL](https://github.com/hannesm/conex/blob/github/analysis/maintainer.ml#L37-L91).
|
|
||||||
Results after filtering GitHub organisations are not yet satisfactory (but only
|
Results after filtering GitHub organisations are not yet satisfactory (but only
|
||||||
56 packages with no maintainer, [output repo](https://github.com/hannesm/opam-repository/tree/github). This approach
|
56 packages with no maintainer, <a href="https://github.com/hannesm/opam-repository/tree/github">output repo</a>. This approach
|
||||||
completely ignores the manually written maintainer field.
|
completely ignores the manually written maintainer field.</p>
|
||||||
|
<h3 id="conclusion">Conclusion</h3>
|
||||||
### Conclusion
|
<p>Manually maintained metadata is easily out of date, and not very useful. But
|
||||||
|
|
||||||
Manually maintained metadata is easily out of date, and not very useful. But
|
|
||||||
combining automatically created metadata with manually, and some manual tweaking
|
combining automatically created metadata with manually, and some manual tweaking
|
||||||
leads to reasonable data.
|
leads to reasonable data.</p>
|
||||||
|
<p>The resulting authorised inference is available <a href="https://github.com/hannesm/opam-repository/tree/auth">in this branch</a>.</p>
|
||||||
The resulting authorised inference is available [in this branch](https://github.com/hannesm/opam-repository/tree/auth).
|
</article></div></div></main></body></html>
|
||||||
180
Posts/Monitoring
180
Posts/Monitoring
|
|
@ -1,86 +1,57 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: All your metrics belong to influx
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>All your metrics belong to influx</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="All your metrics belong to influx" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>All your metrics belong to influx</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/monitoring" class="tag">monitoring</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2022-03-08 (last updated: 2023-05-16)</span><article><h1 id="introduction-to-monitoring">Introduction to monitoring</h1>
|
||||||
author: hannes
|
<p>At <a href="https://robur.coop">robur</a> we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our <a href="https://builds.robur.coop">reproducible builds infrastructure</a>. Another part is, once deployed we want to observe what is going on.</p>
|
||||||
tags: mirageos, monitoring, deployment
|
<p>I first got into touch with monitoring - collecting and graphing metrics - with <a href="https://oss.oetiker.ch/mrtg/">MRTG</a> and <a href="https://munin-monitoring.org/">munin</a> - and the simple network management protocol <a href="https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol">SNMP</a>. From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.</p>
|
||||||
abstract: How to monitor your MirageOS unikernel with albatross and monitoring-experiments
|
<p>The rise of monitoring where graphs are done dynamically (such as <a href="https://grafana.com/">Grafana</a>) and can be programmed (with a query language) by the operator are very neat, it allows to put metrics in relation after they have been recorded - thus if there's a thesis why something went berserk, you can graph the collected data from the past and prove or disprove the thesis.</p>
|
||||||
---
|
<h1 id="monitoring-a-mirageos-unikernel">Monitoring a MirageOS unikernel</h1>
|
||||||
|
<p>From the operational perspective, taking security into account - either the data should be authenticated and integrity-protected, or being transmitted on a private network. We chose the latter, there's a private network interface only for monitoring. Access to that network is only granted to the unikernels and metrics collector.</p>
|
||||||
# Introduction to monitoring
|
<p>For MirageOS unikernels, we use the <a href="https://github.com/mirage/metrics">metrics</a> library - which design shares the idea of <a href="https://erratique.ch/software/logs">logs</a> that only if there's a reporter registered, work is performed. We use the Influx line protocol via TCP to report via <a href="https://www.influxdata.com/time-series-platform/telegraf/">Telegraf</a> to <a href="https://www.influxdata.com/">InfluxDB</a>. But due to the design of <a href="https://github.com/mirage/metrics">metrics</a>, other reporters can be developed and used -- prometheus, SNMP, your-other-favourite are all possible.</p>
|
||||||
|
<p>Apart from monitoring metrics, we use the same network interface for logging via syslog. Since the logs library separates the log message generation (in the OCaml libraries) from the reporting, we developed <a href="https://github.com/hannesm/logs-syslog">logs-syslog</a>, which registers a log reporter sending each log message to a syslog sink.</p>
|
||||||
At [robur](https://robur.coop) we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our [reproducible builds infrastructure](https://builds.robur.coop). Another part is, once deployed we want to observe what is going on.
|
<p>We developed a small library for metrics reporting of a MirageOS unikernel into the <a href="https://github.com/roburio/monitoring-experiments">monitoring-experiments</a> package - which also allows to dynamically adjust log level and disable or enable metrics sources.</p>
|
||||||
|
<h2 id="required-components">Required components</h2>
|
||||||
I first got into touch with monitoring - collecting and graphing metrics - with [MRTG](https://oss.oetiker.ch/mrtg/) and [munin](https://munin-monitoring.org/) - and the simple network management protocol [SNMP](https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol). From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.
|
<p>Install from your operating system the packages providing telegraf, influxdb, and grafana.</p>
|
||||||
|
<p>Setup telegraf to contain a socket listener:</p>
|
||||||
The rise of monitoring where graphs are done dynamically (such as [Grafana](https://grafana.com/)) and can be programmed (with a query language) by the operator are very neat, it allows to put metrics in relation after they have been recorded - thus if there's a thesis why something went berserk, you can graph the collected data from the past and prove or disprove the thesis.
|
<pre><code>[[inputs.socket_listener]]
|
||||||
|
service_address = "tcp://192.168.42.14:8094"
|
||||||
# Monitoring a MirageOS unikernel
|
keep_alive_period = "5m"
|
||||||
|
data_format = "influx"
|
||||||
From the operational perspective, taking security into account - either the data should be authenticated and integrity-protected, or being transmitted on a private network. We chose the latter, there's a private network interface only for monitoring. Access to that network is only granted to the unikernels and metrics collector.
|
</code></pre>
|
||||||
|
<p>Use a unikernel that reports to Influx (below the heading "Unikernels (with metrics reported to Influx)" on <a href="https://builds.robur.coop">builds.robur.coop</a>) and provide <code>--monitor=192.168.42.14</code> as boot parameter. Conventionally, these unikernels expect a second network interface (on the "management" bridge) where telegraf (and a syslog sink) are running. You'll need to pass <code>--net=management</code> and <code>--arg='--management-ipv4=192.168.42.x/24'</code> to albatross-client.</p>
|
||||||
For MirageOS unikernels, we use the [metrics](https://github.com/mirage/metrics) library - which design shares the idea of [logs](https://erratique.ch/software/logs) that only if there's a reporter registered, work is performed. We use the Influx line protocol via TCP to report via [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) to [InfluxDB](https://www.influxdata.com/). But due to the design of [metrics](https://github.com/mirage/metrics), other reporters can be developed and used -- prometheus, SNMP, your-other-favourite are all possible.
|
<p>Albatross provides a <code>albatross-influx</code> daemon that reports information from the host system about the unikernels to influx. Start it with <code>--influx=192.168.42.14</code>.</p>
|
||||||
|
<h2 id="adding-monitoring-to-your-unikernel">Adding monitoring to your unikernel</h2>
|
||||||
Apart from monitoring metrics, we use the same network interface for logging via syslog. Since the logs library separates the log message generation (in the OCaml libraries) from the reporting, we developed [logs-syslog](https://github.com/hannesm/logs-syslog), which registers a log reporter sending each log message to a syslog sink.
|
<p>If you want to extend your own unikernel with metrics, follow along these lines.</p>
|
||||||
|
<p>An example is the <a href="https://github.com/roburio/dns-primary-git">dns-primary-git</a> unikernel, where on the branch <code>future</code> we have a single commit ahead of main that adds monitoring. The difference is in the unikernel configuration and the main entry point. See the <a href="https://builds.robur.coop/job/dns-primary-git-monitoring/build/latest/">binary builts</a> in contrast to the <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/">non-monitoring builts</a>.</p>
|
||||||
We developed a small library for metrics reporting of a MirageOS unikernel into the [monitoring-experiments](https://github.com/roburio/monitoring-experiments) package - which also allows to dynamically adjust log level and disable or enable metrics sources.
|
<p>In config, three new command line arguments are added: <code>--monitor=IP</code>, <code>--monitor-adjust=PORT</code> <code>--syslog=IP</code> and <code>--name=STRING</code>. In addition, the package <code>monitoring-experiments</code> is required. And a second network interface <code>management_stack</code> using the prefix <code>management</code> is required and passed to the unikernel. Since the syslog reporter requires a console (to report when logging fails), also a console is passed to the unikernel. Each reported metrics includes a tag <code>vm=<name></code> that can be used to distinguish several unikernels reporting to the same InfluxDB.</p>
|
||||||
|
<p>Command line arguments:</p>
|
||||||
## Required components
|
<pre><code class="language-patch"> let doc = Key.Arg.info ~doc:"The fingerprint of the TLS certificate." [ "tls-cert-fingerprint" ] in
|
||||||
|
Key.(create "tls_cert_fingerprint" Arg.(opt (some string) None doc))
|
||||||
Install from your operating system the packages providing telegraf, influxdb, and grafana.
|
|
||||||
|
|
||||||
Setup telegraf to contain a socket listener:
|
|
||||||
|
|
||||||
```
|
|
||||||
[[inputs.socket_listener]]
|
|
||||||
service_address = "tcp://192.168.42.14:8094"
|
|
||||||
keep_alive_period = "5m"
|
|
||||||
data_format = "influx"
|
|
||||||
```
|
|
||||||
|
|
||||||
Use a unikernel that reports to Influx (below the heading "Unikernels (with metrics reported to Influx)" on [builds.robur.coop](https://builds.robur.coop)) and provide `--monitor=192.168.42.14` as boot parameter. Conventionally, these unikernels expect a second network interface (on the "management" bridge) where telegraf (and a syslog sink) are running. You'll need to pass `--net=management` and `--arg='--management-ipv4=192.168.42.x/24'` to albatross-client.
|
|
||||||
|
|
||||||
Albatross provides a `albatross-influx` daemon that reports information from the host system about the unikernels to influx. Start it with `--influx=192.168.42.14`.
|
|
||||||
|
|
||||||
## Adding monitoring to your unikernel
|
|
||||||
|
|
||||||
If you want to extend your own unikernel with metrics, follow along these lines.
|
|
||||||
|
|
||||||
An example is the [dns-primary-git](https://github.com/roburio/dns-primary-git) unikernel, where on the branch `future` we have a single commit ahead of main that adds monitoring. The difference is in the unikernel configuration and the main entry point. See the [binary builts](https://builds.robur.coop/job/dns-primary-git-monitoring/build/latest/) in contrast to the [non-monitoring builts](https://builds.robur.coop/job/dns-primary-git/build/latest/).
|
|
||||||
|
|
||||||
In config, three new command line arguments are added: `--monitor=IP`, `--monitor-adjust=PORT` `--syslog=IP` and `--name=STRING`. In addition, the package `monitoring-experiments` is required. And a second network interface `management_stack` using the prefix `management` is required and passed to the unikernel. Since the syslog reporter requires a console (to report when logging fails), also a console is passed to the unikernel. Each reported metrics includes a tag `vm=<name>` that can be used to distinguish several unikernels reporting to the same InfluxDB.
|
|
||||||
|
|
||||||
Command line arguments:
|
|
||||||
```patch
|
|
||||||
let doc = Key.Arg.info ~doc:"The fingerprint of the TLS certificate." [ "tls-cert-fingerprint" ] in
|
|
||||||
Key.(create "tls_cert_fingerprint" Arg.(opt (some string) None doc))
|
|
||||||
|
|
||||||
+let monitor =
|
+let monitor =
|
||||||
+ let doc = Key.Arg.info ~doc:"monitor host IP" ["monitor"] in
|
+ let doc = Key.Arg.info ~doc:"monitor host IP" ["monitor"] in
|
||||||
+ Key.(create "monitor" Arg.(opt (some ip_address) None doc))
|
+ Key.(create "monitor" Arg.(opt (some ip_address) None doc))
|
||||||
+
|
+
|
||||||
+let monitor_adjust =
|
+let monitor_adjust =
|
||||||
+ let doc = Key.Arg.info ~doc:"adjust monitoring (log level, ..)" ["monitor-adjust"] in
|
+ let doc = Key.Arg.info ~doc:"adjust monitoring (log level, ..)" ["monitor-adjust"] in
|
||||||
+ Key.(create "monitor_adjust" Arg.(opt (some int) None doc))
|
+ Key.(create "monitor_adjust" Arg.(opt (some int) None doc))
|
||||||
+
|
+
|
||||||
+let syslog =
|
+let syslog =
|
||||||
+ let doc = Key.Arg.info ~doc:"syslog host IP" ["syslog"] in
|
+ let doc = Key.Arg.info ~doc:"syslog host IP" ["syslog"] in
|
||||||
+ Key.(create "syslog" Arg.(opt (some ip_address) None doc))
|
+ Key.(create "syslog" Arg.(opt (some ip_address) None doc))
|
||||||
+
|
+
|
||||||
+let name =
|
+let name =
|
||||||
+ let doc = Key.Arg.info ~doc:"Name of the unikernel" ["name"] in
|
+ let doc = Key.Arg.info ~doc:"Name of the unikernel" ["name"] in
|
||||||
+ Key.(create "name" Arg.(opt string "ns.nqsb.io" doc))
|
+ Key.(create "name" Arg.(opt string "ns.nqsb.io" doc))
|
||||||
+
|
+
|
||||||
let mimic_impl random stackv4v6 mclock pclock time =
|
let mimic_impl random stackv4v6 mclock pclock time =
|
||||||
let tcpv4v6 = tcpv4v6_of_stackv4v6 $ stackv4v6 in
|
let tcpv4v6 = tcpv4v6_of_stackv4v6 $ stackv4v6 in
|
||||||
let mhappy_eyeballs = mimic_happy_eyeballs $ random $ time $ mclock $ pclock $ stackv4v6 in
|
let mhappy_eyeballs = mimic_happy_eyeballs $ random $ time $ mclock $ pclock $ stackv4v6 in
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Requiring <code>monitoring-experiments</code>, registering command line arguments:</p>
|
||||||
Requiring `monitoring-experiments`, registering command line arguments:
|
<pre><code class="language-patch"> package ~min:"3.7.0" ~max:"3.8.0" "git-mirage";
|
||||||
```patch
|
package ~min:"3.7.0" "git-paf";
|
||||||
package ~min:"3.7.0" ~max:"3.8.0" "git-mirage";
|
package ~min:"0.0.8" ~sublibs:["mirage"] "paf";
|
||||||
package ~min:"3.7.0" "git-paf";
|
+ package "monitoring-experiments";
|
||||||
package ~min:"0.0.8" ~sublibs:["mirage"] "paf";
|
+ package ~sublibs:["mirage"] ~min:"0.3.0" "logs-syslog";
|
||||||
+ package "monitoring-experiments";
|
|
||||||
+ package ~sublibs:["mirage"] ~min:"0.3.0" "logs-syslog";
|
|
||||||
] in
|
] in
|
||||||
foreign
|
foreign
|
||||||
- ~keys:[Key.abstract remote_k ; Key.abstract axfr]
|
- ~keys:[Key.abstract remote_k ; Key.abstract axfr]
|
||||||
|
|
@ -89,31 +60,25 @@ Requiring `monitoring-experiments`, registering command line arguments:
|
||||||
+ Key.abstract name ; Key.abstract monitor ; Key.abstract monitor_adjust ; Key.abstract syslog
|
+ Key.abstract name ; Key.abstract monitor ; Key.abstract monitor_adjust ; Key.abstract syslog
|
||||||
+ ]
|
+ ]
|
||||||
~packages
|
~packages
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Added console and a second network stack to <code>foreign</code>:</p>
|
||||||
Added console and a second network stack to `foreign`:
|
<pre><code class="language-patch"> "Unikernel.Main"
|
||||||
```patch
|
- (random @-> pclock @-> mclock @-> time @-> stackv4v6 @-> mimic @-> job)
|
||||||
"Unikernel.Main"
|
+ (console @-> random @-> pclock @-> mclock @-> time @-> stackv4v6 @-> mimic @-> stackv4v6 @-> job)
|
||||||
- (random @-> pclock @-> mclock @-> time @-> stackv4v6 @-> mimic @-> job)
|
|
||||||
+ (console @-> random @-> pclock @-> mclock @-> time @-> stackv4v6 @-> mimic @-> stackv4v6 @-> job)
|
|
||||||
+
|
+
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Passing a console implementation (<code>default_console</code>) and a second network stack (with <code>management</code> prefix) to <code>register</code>:</p>
|
||||||
Passing a console implementation (`default_console`) and a second network stack (with `management` prefix) to `register`:
|
<pre><code class="language-patch">+let management_stack = generic_stackv4v6 ~group:"management" (netif ~group:"management" "management")
|
||||||
```patch
|
|
||||||
+let management_stack = generic_stackv4v6 ~group:"management" (netif ~group:"management" "management")
|
|
||||||
|
|
||||||
let () =
|
let () =
|
||||||
register "primary-git"
|
register "primary-git"
|
||||||
- [dns_handler $ default_random $ default_posix_clock $ default_monotonic_clock $
|
- [dns_handler $ default_random $ default_posix_clock $ default_monotonic_clock $
|
||||||
- default_time $ net $ mimic_impl]
|
- default_time $ net $ mimic_impl]
|
||||||
+ [dns_handler $ default_console $ default_random $ default_posix_clock $ default_monotonic_clock $
|
+ [dns_handler $ default_console $ default_random $ default_posix_clock $ default_monotonic_clock $
|
||||||
+ default_time $ net $ mimic_impl $ management_stack]
|
+ default_time $ net $ mimic_impl $ management_stack]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Now, in the unikernel module the functor changes (console and second network stack added):</p>
|
||||||
Now, in the unikernel module the functor changes (console and second network stack added):
|
<pre><code class="language-patch">@@ -4,17 +4,48 @@
|
||||||
```patch
|
|
||||||
@@ -4,17 +4,48 @@
|
|
||||||
|
|
||||||
open Lwt.Infix
|
open Lwt.Infix
|
||||||
|
|
||||||
|
|
@ -124,11 +89,9 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
|
||||||
|
|
||||||
module Store = Irmin_mirage_git.Mem.KV(Irmin.Contents.String)
|
module Store = Irmin_mirage_git.Mem.KV(Irmin.Contents.String)
|
||||||
module Sync = Irmin.Sync(Store)
|
module Sync = Irmin.Sync(Store)
|
||||||
```
|
</code></pre>
|
||||||
|
<p>And in the <code>start</code> function, the command line arguments are processed and used to setup syslog and metrics monitoring to the specified addresses. Also, a TCP listener is waiting for monitoring and logging adjustments if <code>--monitor-adjust</code> was provided:</p>
|
||||||
And in the `start` function, the command line arguments are processed and used to setup syslog and metrics monitoring to the specified addresses. Also, a TCP listener is waiting for monitoring and logging adjustments if `--monitor-adjust` was provided:
|
<pre><code class="language-patch"> module D = Dns_server_mirage.Make(P)(M)(T)(S)
|
||||||
```patch
|
|
||||||
module D = Dns_server_mirage.Make(P)(M)(T)(S)
|
|
||||||
+ module Monitoring = Monitoring_experiments.Make(T)(Management)
|
+ module Monitoring = Monitoring_experiments.Make(T)(Management)
|
||||||
+ module Syslog = Logs_syslog_mirage.Udp(C)(P)(Management)
|
+ module Syslog = Logs_syslog_mirage.Udp(C)(P)(Management)
|
||||||
|
|
||||||
|
|
@ -136,20 +99,17 @@ And in the `start` function, the command line arguments are processed and used t
|
||||||
+ let start c _rng _pclock _mclock _time s ctx management =
|
+ let start c _rng _pclock _mclock _time s ctx management =
|
||||||
+ let hostname = Key_gen.name () in
|
+ let hostname = Key_gen.name () in
|
||||||
+ (match Key_gen.syslog () with
|
+ (match Key_gen.syslog () with
|
||||||
+ | None -> Logs.warn (fun m -> m "no syslog specified, dumping on stdout")
|
+ | None -> Logs.warn (fun m -> m "no syslog specified, dumping on stdout")
|
||||||
+ | Some ip -> Logs.set_reporter (Syslog.create c management ip ~hostname ()));
|
+ | Some ip -> Logs.set_reporter (Syslog.create c management ip ~hostname ()));
|
||||||
+ (match Key_gen.monitor () with
|
+ (match Key_gen.monitor () with
|
||||||
+ | None -> Logs.warn (fun m -> m "no monitor specified, not outputting statistics")
|
+ | None -> Logs.warn (fun m -> m "no monitor specified, not outputting statistics")
|
||||||
+ | Some ip -> Monitoring.create ~hostname ?listen_port:(Key_gen.monitor_adjust ()) ip management);
|
+ | Some ip -> Monitoring.create ~hostname ?listen_port:(Key_gen.monitor_adjust ()) ip management);
|
||||||
connect_store ctx >>= fun (store, upstream) ->
|
connect_store ctx >>= fun (store, upstream) ->
|
||||||
load_git None store upstream >>= function
|
load_git None store upstream >>= function
|
||||||
| Error (`Msg msg) ->
|
| Error (`Msg msg) ->
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Once you compiled the unikernel (or downloaded a binary with monitoring), and start that unikernel by passing <code>--net:service=tap0</code> and <code>--net:management=tap10</code> (or whichever your <code>tap</code> interfaces are), and as unikernel arguments <code>--ipv4=<my-ip-address></code> and <code>--management-ipv4=192.168.42.2/24</code> for IPv4 configuration, <code>--monitor=192.168.42.14</code>, <code>--syslog=192.168.42.10</code>, <code>--name=my.unikernel</code>, <code>--monitor-adjust=12345</code>.</p>
|
||||||
Once you compiled the unikernel (or downloaded a binary with monitoring), and start that unikernel by passing `--net:service=tap0` and `--net:management=tap10` (or whichever your `tap` interfaces are), and as unikernel arguments `--ipv4=<my-ip-address>` and `--management-ipv4=192.168.42.2/24` for IPv4 configuration, `--monitor=192.168.42.14`, `--syslog=192.168.42.10`, `--name=my.unikernel`, `--monitor-adjust=12345`.
|
<p>With this, your unikernel will report metrics using the influx protocol to 192.168.42.14 on port 8094 (every 10 seconds), and syslog messages via UDP to 192.168.0.10 (port 514). You should see your InfluxDB getting filled and syslog server receiving messages.</p>
|
||||||
|
<p>When you configure <a href="https://grafana.com/docs/grafana/latest/getting-started/getting-started-influxdb/">Grafana to use InfluxDB</a>, you'll be able to see the data in the data sources.</p>
|
||||||
With this, your unikernel will report metrics using the influx protocol to 192.168.42.14 on port 8094 (every 10 seconds), and syslog messages via UDP to 192.168.0.10 (port 514). You should see your InfluxDB getting filled and syslog server receiving messages.
|
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
When you configure [Grafana to use InfluxDB](https://grafana.com/docs/grafana/latest/getting-started/getting-started-influxdb/), you'll be able to see the data in the data sources.
|
|
||||||
|
|
||||||
Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.
|
|
||||||
171
Posts/NGI
171
Posts/NGI
|
|
@ -1,104 +1,67 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: The road ahead for MirageOS in 2021
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The road ahead for MirageOS in 2021</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The road ahead for MirageOS in 2021" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The road ahead for MirageOS in 2021</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a></div><span class="date">Published: 2021-01-25 (last updated: 2021-11-19)</span><article><h2 id="introduction">Introduction</h2>
|
||||||
author: hannes
|
<p>2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations, and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)</p>
|
||||||
tags: mirageos
|
<p>I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.</p>
|
||||||
abstract: Home office, MirageOS unikernels, 2020 recap, 2021 tbd
|
<p>For 2021, we (at <a href="https://robur.coop">robur</a>) got a grant from the EU (via <a href="https://pointer.ngi.eu">NGI pointer</a>) for "Deploying MirageOS" (more details below), and another grant from <a href="https://ocaml-sf.org">OCaml software foundation</a> for securing the opam supply chain (using <a href="https://github.com/hannesm/conex">conex</a>). Some long-awaited releases for MirageOS libraries, namely a <a href="https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh">ssh implementation</a> and a rewrite of our <a href="https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/">git implementation</a> have already been published.</p>
|
||||||
---
|
<p>With my MirageOS view, 2020 was a pretty successful year, where we managed to add more features, fixed lots of bugs, and paved the road ahead. I want to thank <a href="https://ocamllabs.io/">OCamlLabs</a> for funding work on MirageOS maintenance.</p>
|
||||||
|
<h2 id="recap-2020">Recap 2020</h2>
|
||||||
## Introduction
|
<p>Here is a very subjective random collection of accomplishments in 2020, where I was involved with some degree.</p>
|
||||||
|
<h3 id="nethsm">NetHSM</h3>
|
||||||
2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations, and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)
|
<p><a href="https://www.nitrokey.com/products/nethsm">NetHSM</a> is a hardware security module in software. It is a product that uses MirageOS for security, and is based on the <a href="https://muen.sk">muen</a> separation kernel. We at <a href="https://robur.coop">robur</a> were heavily involved in this product. It already has been security audited by an external team. You can pre-order it from Nitrokey.</p>
|
||||||
|
<h3 id="tls-1.3">TLS 1.3</h3>
|
||||||
I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.
|
<p>Dating back to 2016, at the <a href="https://www.ndss-symposium.org/ndss2016/tron-workshop-programme/">TRON</a> (TLS 1.3 Ready or NOt), we developed a first draft of a 1.3 implementation of <a href="https://github.com/mirleft/ocaml-tls">OCaml-TLS</a>. Finally in May 2020 we got our act together, including ECC (ECDH P256 from <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a>, X25519 from <a href="https://project-everest.github.io/">hacl</a>) and testing with <a href="https://github.com/tlsfuzzer/tlsfuzzer">tlsfuzzer</a>, and release tls 0.12.0 with TLS 1.3 support. Later we added <a href="https://github.com/mirleft/ocaml-tls/pull/414">ECC ciphersuites to TLS version 1.2</a>, implemented <a href="https://github.com/mirleft/ocaml-tls/pull/414">ChaCha20/Poly1305</a>, and fixed an <a href="https://github.com/mirleft/ocaml-tls/pull/424">interoperability issue with Go's implementation</a>.</p>
|
||||||
|
<p><a href="https://github.com/mirage/mirage-crypto">Mirage-crypto</a> provides the underlying cryptographic primitives, initially released in March 2020 as a fork of <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a> -- huge thanks to <a href="https://github.com/pqwy">pqwy</a> for his great work. Mirage-crypto detects <a href="https://github.com/mirage/mirage-crypto/pull/53">CPU features at runtime</a> (thanks to <a href="https://github.com/Julow">Julow</a>) (<a href="https://github.com/mirage/mirage-crypto/pull/96">bugfix for bswap</a>), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports <a href="https://github.com/mirage/mirage-crypto/pull/39">compilation on Windows</a> (thanks to <a href="https://github.com/avsm">avsm</a>), <a href="https://github.com/mirage/mirage-crypto/pull/90">async entropy harvesting</a> (thanks to <a href="https://github.com/seliopou">seliopou</a>), <a href="https://github.com/mirage/mirage-crypto/pull/65">32 bit support</a>, <a href="https://github.com/mirage/mirage-crypto/pull/72">chacha20/poly1305</a> (thanks to <a href="https://github.com/abeaumont">abeaumont</a>), <a href="https://github.com/mirage/mirage-crypto/pull/84">cross-compilation</a> (thanks to <a href="https://github.com/EduardoRFS">EduardoRFS</a>) and <a href="https://github.com/mirage/mirage-crypto/pull/78">various</a> <a href="https://github.com/mirage/mirage-crypto/pull/81">bug</a> <a href="https://github.com/mirage/mirage-crypto/pull/83">fixes</a>, even <a href="https://github.com/mirage/mirage-crypto/pull/95">memory leak</a> (thanks to <a href="https://github.com/talex5">talex5</a> for reporting several of these issues), and <a href="https://github.com/mirage/mirage-crypto/pull/99">RSA</a> <a href="https://github.com/mirage/mirage-crypto/pull/100">interoperability</a> (thanks to <a href="https://github.com/psafont">psafont</a> for investigation and <a href="https://github.com/mattjbray">mattjbray</a> for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.</p>
|
||||||
For 2021, we (at [robur](https://robur.coop)) got a grant from the EU (via [NGI pointer](https://pointer.ngi.eu)) for "Deploying MirageOS" (more details below), and another grant from [OCaml software foundation](https://ocaml-sf.org) for securing the opam supply chain (using [conex](https://github.com/hannesm/conex)). Some long-awaited releases for MirageOS libraries, namely a [ssh implementation](https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh) and a rewrite of our [git implementation](https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/) have already been published.
|
<h3 id="qubes-firewall">Qubes Firewall</h3>
|
||||||
|
<p>The <a href="https://github.com/mirage/qubes-mirage-firewall/">MirageOS based Qubes firewall</a> is the most widely used MirageOS unikernel. And it got major updates: in May <a href="https://github.com/linse">Steffi</a> <a href="https://groups.google.com/g/qubes-users/c/Xzplmkjwa5Y">announced</a> her and <a href="https://github.com/yomimono">Mindy's</a> work on improving it for Qubes 4.0 - including <a href="https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x">dynamic firewall rules via QubesDB</a>. Thanks to <a href="https://prototypefund.de/project/portable-firewall-fuer-qubesos/">prototypefund</a> for sponsoring.</p>
|
||||||
With my MirageOS view, 2020 was a pretty successful year, where we managed to add more features, fixed lots of bugs, and paved the road ahead. I want to thank [OCamlLabs](https://ocamllabs.io/) for funding work on MirageOS maintenance.
|
<p>In October 2020, we released <a href="https://mirage.io/blog/announcing-mirage-39-release">Mirage 3.9</a> with PVH virtualization mode (thanks to <a href="https://github.com/mato">mato</a>). There's still a <a href="https://github.com/mirage/qubes-mirage-firewall/issues/120">memory leak</a> to be investigated and fixed.</p>
|
||||||
|
<h3 id="ipv6">IPv6</h3>
|
||||||
## Recap 2020
|
<p>In December, with <a href="https://mirage.io/blog/announcing-mirage-310-release">Mirage 3.10</a> we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to <a href="https://github.com/nojb">nojb</a> for the initial code and <a href="https://github.com/MagnusS">MagnusS</a>.</p>
|
||||||
|
<p>Turns out this blog, but also robur services, are now available via IPv6 :)</p>
|
||||||
Here is a very subjective random collection of accomplishments in 2020, where I was involved with some degree.
|
<h3 id="albatross">Albatross</h3>
|
||||||
|
<p>Also in December, I pushed an initial release of <a href="https://github.com/roburio/albatross">albatross</a>, a unikernel orchestration system with remote access. <em>Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.</em></p>
|
||||||
### NetHSM
|
<p>Thanks to <a href="https://github.com/reynir">reynir</a> for statistics support on Linux and improvements of the systemd service scripts. Also thanks to <a href="https://github.com/cfcs">cfcs</a> for the initial Linux port.</p>
|
||||||
|
<h3 id="ca-certs">CA certs</h3>
|
||||||
[NetHSM](https://www.nitrokey.com/products/nethsm) is a hardware security module in software. It is a product that uses MirageOS for security, and is based on the [muen](https://muen.sk) separation kernel. We at [robur](https://robur.coop) were heavily involved in this product. It already has been security audited by an external team. You can pre-order it from Nitrokey.
|
<p>For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to <a href="https://github.com/emillon">emillon</a> for initial code, there are now <a href="https://github.com/mirage/ca-certs">ca-certs</a> and <a href="https://github.com/mirage/ca-certs-nss">ca-certs-nss</a> opam packages (see <a href="https://discuss.ocaml.org/t/ann-ca-certs-and-ca-certs-nss">release announcement</a>) which fills this gap.</p>
|
||||||
|
<h2 id="unikernels">Unikernels</h2>
|
||||||
### TLS 1.3
|
<p>I developed several useful unikernels in 2020, and also pushed <a href="https://mirage.io/wiki/gallery">a unikernel gallery</a> to the Mirage website:</p>
|
||||||
|
<h3 id="traceroute-in-mirageos">Traceroute in MirageOS</h3>
|
||||||
Dating back to 2016, at the [TRON](https://www.ndss-symposium.org/ndss2016/tron-workshop-programme/) (TLS 1.3 Ready or NOt), we developed a first draft of a 1.3 implementation of [OCaml-TLS](https://github.com/mirleft/ocaml-tls). Finally in May 2020 we got our act together, including ECC (ECDH P256 from [fiat-crypto](https://github.com/mit-plv/fiat-crypto/), X25519 from [hacl](https://project-everest.github.io/)) and testing with [tlsfuzzer](https://github.com/tlsfuzzer/tlsfuzzer), and release tls 0.12.0 with TLS 1.3 support. Later we added [ECC ciphersuites to TLS version 1.2](https://github.com/mirleft/ocaml-tls/pull/414), implemented [ChaCha20/Poly1305](https://github.com/mirleft/ocaml-tls/pull/414), and fixed an [interoperability issue with Go's implementation](https://github.com/mirleft/ocaml-tls/pull/424).
|
<p>I already wrote about <a href="/Posts/Traceroute">traceroute</a> which traces the routing to a given remote host.</p>
|
||||||
|
<h3 id="unipi---static-website-hosting">Unipi - static website hosting</h3>
|
||||||
[Mirage-crypto](https://github.com/mirage/mirage-crypto) provides the underlying cryptographic primitives, initially released in March 2020 as a fork of [nocrypto](https://github.com/mirleft/ocaml-nocrypto) -- huge thanks to [pqwy](https://github.com/pqwy) for his great work. Mirage-crypto detects [CPU features at runtime](https://github.com/mirage/mirage-crypto/pull/53) (thanks to [Julow](https://github.com/Julow)) ([bugfix for bswap](https://github.com/mirage/mirage-crypto/pull/96)), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports [compilation on Windows](https://github.com/mirage/mirage-crypto/pull/39) (thanks to [avsm](https://github.com/avsm)), [async entropy harvesting](https://github.com/mirage/mirage-crypto/pull/90) (thanks to [seliopou](https://github.com/seliopou)), [32 bit support](https://github.com/mirage/mirage-crypto/pull/65), [chacha20/poly1305](https://github.com/mirage/mirage-crypto/pull/72) (thanks to [abeaumont](https://github.com/abeaumont)), [cross-compilation](https://github.com/mirage/mirage-crypto/pull/84) (thanks to [EduardoRFS](https://github.com/EduardoRFS)) and [various](https://github.com/mirage/mirage-crypto/pull/78) [bug](https://github.com/mirage/mirage-crypto/pull/81) [fixes](https://github.com/mirage/mirage-crypto/pull/83), even [memory leak](https://github.com/mirage/mirage-crypto/pull/95) (thanks to [talex5](https://github.com/talex5) for reporting several of these issues), and [RSA](https://github.com/mirage/mirage-crypto/pull/99) [interoperability](https://github.com/mirage/mirage-crypto/pull/100) (thanks to [psafont](https://github.com/psafont) for investigation and [mattjbray](https://github.com/mattjbray) for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.
|
<p><a href="https://github.com/roburio/unipi">Unipi</a> is a static site webserver which retrieves the content from a remote git repository. Let's encrypt certificate provisioning and dynamic updates via a webhook to be executed for every push.</p>
|
||||||
|
<h4 id="tlstunnel---tls-demultiplexing">TLSTunnel - TLS demultiplexing</h4>
|
||||||
### Qubes Firewall
|
<p>The physical machine this blog and other robur infrastructure runs on has been relocated from Sweden to Germany mid-December. Thanks to UPS! Fewer IPv4 addresses are available in the new data center, which motivated me to develop <a href="https://github.com/roburio/tlstunnel">tlstunnel</a>.</p>
|
||||||
|
<p>The new behaviour is as follows (see the <code>monitoring</code> branch):</p>
|
||||||
The [MirageOS based Qubes firewall](https://github.com/mirage/qubes-mirage-firewall/) is the most widely used MirageOS unikernel. And it got major updates: in May [Steffi](https://github.com/linse) [announced](https://groups.google.com/g/qubes-users/c/Xzplmkjwa5Y) her and [Mindy's](https://github.com/yomimono) work on improving it for Qubes 4.0 - including [dynamic firewall rules via QubesDB](https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x). Thanks to [prototypefund](https://prototypefund.de/project/portable-firewall-fuer-qubesos/) for sponsoring.
|
<ul>
|
||||||
|
<li>listener on TCP port 80 which replies with a permanent redirect to <code>https</code>
|
||||||
In October 2020, we released [Mirage 3.9](https://mirage.io/blog/announcing-mirage-39-release) with PVH virtualization mode (thanks to [mato](https://github.com/mato)). There's still a [memory leak](https://github.com/mirage/qubes-mirage-firewall/issues/120) to be investigated and fixed.
|
</li>
|
||||||
|
<li>listener on TCP port 443 which forwards to a backend host if the requested server name is configured
|
||||||
### IPv6
|
</li>
|
||||||
|
<li>its configuration is stored on a block device, and can be dynamically changed (with a custom protocol authenticated with a HMAC)
|
||||||
In December, with [Mirage 3.10](https://mirage.io/blog/announcing-mirage-310-release) we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to [nojb](https://github.com/nojb) for the initial code and [MagnusS](https://github.com/MagnusS).
|
</li>
|
||||||
|
<li>it is setup to hold a wildcard TLS certificate and in DNS a wildcard entry is pointing to it
|
||||||
Turns out this blog, but also robur services, are now available via IPv6 :)
|
</li>
|
||||||
|
<li>setting up a new service is very straightforward: only the new name needs to be registered with tlstunnel together with the TCP backend, and everything will just work
|
||||||
### Albatross
|
</li>
|
||||||
|
</ul>
|
||||||
Also in December, I pushed an initial release of [albatross](https://github.com/roburio/albatross), a unikernel orchestration system with remote access. *Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.*
|
<h2 id="section">2021</h2>
|
||||||
|
<p>The year started with a release of <a href="https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh">awa</a>, a SSH implementation in OCaml (thanks to <a href="https://github.com/haesbaert">haesbaert</a> for initial code). This was followed by a <a href="https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/">git 3.0 release</a> (thanks to <a href="https://github.com/dinosaure">dinosaure</a>).</p>
|
||||||
Thanks to [reynir](https://github.com/reynir) for statistics support on Linux and improvements of the systemd service scripts. Also thanks to [cfcs](https://github.com/cfcs) for the initial Linux port.
|
<h3 id="deploying-mirageos---ngi-pointer">Deploying MirageOS - NGI Pointer</h3>
|
||||||
|
<p>For 2021 we at robur received funding from the EU (via <a href="https://pointer.ngi.eu/">NGI pointer</a>) for "Deploying MirageOS", which boils down into three parts:</p>
|
||||||
### CA certs
|
<ul>
|
||||||
|
<li>reproducible binary releases of MirageOS unikernels,
|
||||||
For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to [emillon](https://github.com/emillon) for initial code, there are now [ca-certs](https://github.com/mirage/ca-certs) and [ca-certs-nss](https://github.com/mirage/ca-certs-nss) opam packages (see [release announcement](https://discuss.ocaml.org/t/ann-ca-certs-and-ca-certs-nss)) which fills this gap.
|
</li>
|
||||||
|
<li>monitoring (and other devops features: profiling) and integration into existing infrastructure,
|
||||||
## Unikernels
|
</li>
|
||||||
|
<li>and further documentation and advertisement.
|
||||||
I developed several useful unikernels in 2020, and also pushed [a unikernel gallery](https://mirage.io/wiki/gallery) to the Mirage website:
|
</li>
|
||||||
|
</ul>
|
||||||
### Traceroute in MirageOS
|
<p>Of course this will all be available open source. Please get in touch via eMail (team aT robur dot coop) if you're eager to integrate MirageOS unikernels into your infrastructure.</p>
|
||||||
|
<p>We discovered at an initial meeting with an infrastructure provider that a DNS resolver is of interest - even more now that dnsmasq suffered from <a href="https://www.jsof-tech.com/wp-content/uploads/2021/01/DNSpooq_Technical-Whitepaper.pdf">dnspooq</a>. We are already working on an <a href="https://github.com/mirage/ocaml-dns/pull/251">implementation of DNSSec</a>.</p>
|
||||||
I already wrote about [traceroute](/Posts/Traceroute) which traces the routing to a given remote host.
|
<p>MirageOS unikernels are binary reproducible, and <a href="https://github.com/rjbou/orb/pull/1">infrastructure tools are available</a>. We are working hard on a web interface (and REST API - think of it as "Docker Hub for MirageOS unikernels"), and more tooling to verify reproducibility.</p>
|
||||||
|
<h3 id="conex---securing-the-supply-chain">Conex - securing the supply chain</h3>
|
||||||
### Unipi - static website hosting
|
<p>Another funding from the <a href="http://ocaml-sf.org/">OCSF</a> is to continue development and deploy <a href="https://github.com/hannesm/conex">conex</a> - to bring trust into opam-repository. This is a great combination with the reproducible build efforts, and will bring much more trust into retrieving OCaml packages and using MirageOS unikernels.</p>
|
||||||
|
<h3 id="mirageos-4.0">MirageOS 4.0</h3>
|
||||||
[Unipi](https://github.com/roburio/unipi) is a static site webserver which retrieves the content from a remote git repository. Let's encrypt certificate provisioning and dynamic updates via a webhook to be executed for every push.
|
<p>Mirage so far still uses ocamlbuild and ocamlfind for compiling the virtual machine binary. But the switch to dune is <a href="https://github.com/mirage/mirage/issues/1195">close</a>, a lot of effort has been done. This will make the developer experience of MirageOS much more smooth, with a per-unikernel monorepo workflow where you can push your changes to the individual libraries.</p>
|
||||||
|
<h2 id="footer">Footer</h2>
|
||||||
#### TLSTunnel - TLS demultiplexing
|
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
The physical machine this blog and other robur infrastructure runs on has been relocated from Sweden to Germany mid-December. Thanks to UPS! Fewer IPv4 addresses are available in the new data center, which motivated me to develop [tlstunnel](https://github.com/roburio/tlstunnel).
|
|
||||||
|
|
||||||
The new behaviour is as follows (see the `monitoring` branch):
|
|
||||||
- listener on TCP port 80 which replies with a permanent redirect to `https`
|
|
||||||
- listener on TCP port 443 which forwards to a backend host if the requested server name is configured
|
|
||||||
- its configuration is stored on a block device, and can be dynamically changed (with a custom protocol authenticated with a HMAC)
|
|
||||||
- it is setup to hold a wildcard TLS certificate and in DNS a wildcard entry is pointing to it
|
|
||||||
- setting up a new service is very straightforward: only the new name needs to be registered with tlstunnel together with the TCP backend, and everything will just work
|
|
||||||
|
|
||||||
## 2021
|
|
||||||
|
|
||||||
The year started with a release of [awa](https://discuss.ocaml.org/t/ann-first-release-of-awa-ssh), a SSH implementation in OCaml (thanks to [haesbaert](https://github.com/haesbaert) for initial code). This was followed by a [git 3.0 release](https://discuss.ocaml.org/t/ann-release-of-ocaml-git-v3-0-duff-encore-decompress-etc/) (thanks to [dinosaure](https://github.com/dinosaure)).
|
|
||||||
|
|
||||||
### Deploying MirageOS - NGI Pointer
|
|
||||||
|
|
||||||
For 2021 we at robur received funding from the EU (via [NGI pointer](https://pointer.ngi.eu/)) for "Deploying MirageOS", which boils down into three parts:
|
|
||||||
* reproducible binary releases of MirageOS unikernels,
|
|
||||||
* monitoring (and other devops features: profiling) and integration into existing infrastructure,
|
|
||||||
* and further documentation and advertisement.
|
|
||||||
|
|
||||||
Of course this will all be available open source. Please get in touch via eMail (team aT robur dot coop) if you're eager to integrate MirageOS unikernels into your infrastructure.
|
|
||||||
|
|
||||||
We discovered at an initial meeting with an infrastructure provider that a DNS resolver is of interest - even more now that dnsmasq suffered from [dnspooq](https://www.jsof-tech.com/wp-content/uploads/2021/01/DNSpooq_Technical-Whitepaper.pdf). We are already working on an [implementation of DNSSec](https://github.com/mirage/ocaml-dns/pull/251).
|
|
||||||
|
|
||||||
MirageOS unikernels are binary reproducible, and [infrastructure tools are available](https://github.com/rjbou/orb/pull/1). We are working hard on a web interface (and REST API - think of it as "Docker Hub for MirageOS unikernels"), and more tooling to verify reproducibility.
|
|
||||||
|
|
||||||
### Conex - securing the supply chain
|
|
||||||
|
|
||||||
Another funding from the [OCSF](http://ocaml-sf.org/) is to continue development and deploy [conex](https://github.com/hannesm/conex) - to bring trust into opam-repository. This is a great combination with the reproducible build efforts, and will bring much more trust into retrieving OCaml packages and using MirageOS unikernels.
|
|
||||||
|
|
||||||
### MirageOS 4.0
|
|
||||||
|
|
||||||
Mirage so far still uses ocamlbuild and ocamlfind for compiling the virtual machine binary. But the switch to dune is [close](https://github.com/mirage/mirage/issues/1195), a lot of effort has been done. This will make the developer experience of MirageOS much more smooth, with a per-unikernel monorepo workflow where you can push your changes to the individual libraries.
|
|
||||||
|
|
||||||
## Footer
|
|
||||||
|
|
||||||
If you want to support our work on MirageOS unikernels, please [donate to robur](https://robur.coop/Donate). I'm interested in feedback, either via [twitter](https://twitter.com/h4nnes), [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
201
Posts/OCaml
201
Posts/OCaml
|
|
@ -1,142 +1,121 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Why OCaml
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Why OCaml</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Why OCaml" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Why OCaml</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-17 (last updated: 2021-11-19)</span><article><h2 id="programming">Programming</h2>
|
||||||
author: hannes
|
<p>For me, programming is fun. I enjoy doing it, every single second. All the way
|
||||||
tags: overview, background
|
|
||||||
abstract: a gentle introduction into OCaml
|
|
||||||
---
|
|
||||||
|
|
||||||
## Programming
|
|
||||||
|
|
||||||
For me, programming is fun. I enjoy doing it, every single second. All the way
|
|
||||||
from designing over experimenting to debugging why it does not do what I want.
|
from designing over experimenting to debugging why it does not do what I want.
|
||||||
In the end, the computer is dumb and executes only what you (or code from
|
In the end, the computer is dumb and executes only what you (or code from
|
||||||
someone else which you rely on) tell it to do.
|
someone else which you rely on) tell it to do.</p>
|
||||||
|
<p>To abstract from assembly code, which is not portable, programming languages were
|
||||||
To abstract from assembly code, which is not portable, programming languages were
|
|
||||||
developed. Different flavoured languages vary in
|
developed. Different flavoured languages vary in
|
||||||
expressive power and static guarantees. Many claim to be general purpose or
|
expressive power and static guarantees. Many claim to be general purpose or
|
||||||
systems languages; depending on the choices of
|
systems languages; depending on the choices of
|
||||||
the language designer and tooling around the language, it is a language which lets you conveniently develop programs in.
|
the language designer and tooling around the language, it is a language which lets you conveniently develop programs in.</p>
|
||||||
|
<p>A language designer decides on the builtin abstraction mechanisms, each of which
|
||||||
A language designer decides on the builtin abstraction mechanisms, each of which
|
is both a burden and a blessing: it might be interfering (which to use? <code>for</code> or <code>while</code>, <code>trait</code> or <code>object</code>),
|
||||||
is both a burden and a blessing: it might be interfering (which to use? `for` or `while`, `trait` or `object`),
|
|
||||||
orthogonal (one way to do it), or even synergistic (higher order functions and anonymous functions). Another choice is whether the language includes a type
|
orthogonal (one way to do it), or even synergistic (higher order functions and anonymous functions). Another choice is whether the language includes a type
|
||||||
system, and if the developer can cheat on it (by allowing arbitrary type casts, a *weak* type system). A strong static type system
|
system, and if the developer can cheat on it (by allowing arbitrary type casts, a <em>weak</em> type system). A strong static type system
|
||||||
allows a developer to encode invariants, without the need to defer to runtime
|
allows a developer to encode invariants, without the need to defer to runtime
|
||||||
assertions. Type systems differ in their expressive power ([dependent typing](https://en.wikipedia.org/wiki/Dependent_type) are the hot research area at the moment). Tooling depends purely
|
assertions. Type systems differ in their expressive power (<a href="https://en.wikipedia.org/wiki/Dependent_type">dependent typing</a> are the hot research area at the moment). Tooling depends purely
|
||||||
on the community size, natural selection will prevail the useful tools
|
on the community size, natural selection will prevail the useful tools
|
||||||
(community size gives inertia to other factors: demand for libraries, package manager, activity on stack overflow, etc.).
|
(community size gives inertia to other factors: demand for libraries, package manager, activity on stack overflow, etc.).</p>
|
||||||
|
<h2 id="why-ocaml">Why OCaml?</h2>
|
||||||
|
<p>As already mentioned in <a href="/Posts/About">other</a>
|
||||||
|
<a href="/Posts/OperatingSystem">articles</a> here, it is a
|
||||||
## Why OCaml?
|
|
||||||
|
|
||||||
As already mentioned in [other](/Posts/About)
|
|
||||||
[articles](/Posts/OperatingSystem) here, it is a
|
|
||||||
combination of sufficiently large community, runtime stability and performance, modularity,
|
combination of sufficiently large community, runtime stability and performance, modularity,
|
||||||
carefully thought out abstraction mechanisms, maturity (OCaml recently turned 20), and functional features.
|
carefully thought out abstraction mechanisms, maturity (OCaml recently turned 20), and functional features.</p>
|
||||||
|
<p>The latter is squishy, I'll try to explain it a bit: you define your concrete
|
||||||
The latter is squishy, I'll try to explain it a bit: you define your concrete
|
<em>data types</em> as <em>products</em> (<code>int * int</code>, a tuple of integers), <em>records</em> (<code>{ foo : int ; bar : int }</code> to name fields), sums (<code>type state = Initial | WaitingForKEX | Established</code>, or variants, or tagged union in C).
|
||||||
*data types* as *products* (`int * int`, a tuple of integers), *records* (`{
|
These are called <a href="https://en.wikipedia.org/wiki/Algebraic_data_type"><em>algebraic data types</em></a>. Whenever you have a
|
||||||
foo : int ; bar : int }` to name fields), sums (`type state = Initial | WaitingForKEX | Established`, or variants, or tagged union in C).
|
|
||||||
These are called [*algebraic data types*](https://en.wikipedia.org/wiki/Algebraic_data_type). Whenever you have a
|
|
||||||
state machine, you can encode the state as a variant and use a
|
state machine, you can encode the state as a variant and use a
|
||||||
pattern match to handle the different cases. The compiler checks whether your pattern match is complete
|
pattern match to handle the different cases. The compiler checks whether your pattern match is complete
|
||||||
(contains a line for each member of the variant). Another important aspect of
|
(contains a line for each member of the variant). Another important aspect of
|
||||||
functional programming is that you can pass functions to other functions
|
functional programming is that you can pass functions to other functions
|
||||||
(*higher-order functions*). Also, *recursion* is fundamental for functional
|
(<em>higher-order functions</em>). Also, <em>recursion</em> is fundamental for functional
|
||||||
programming: a function calls itself -- combined with a variant type (such as
|
programming: a function calls itself -- combined with a variant type (such as
|
||||||
`type 'a list = Nil | Cons of 'a * 'a list`) it is trivial to show termination.
|
<code>type 'a list = Nil | Cons of 'a * 'a list</code>) it is trivial to show termination.</p>
|
||||||
|
<p><em>Side effects</em> make the program interesting, because they
|
||||||
*Side effects* make the program interesting, because they
|
|
||||||
communicate with other systems or humans. Side effects should be isolated and
|
communicate with other systems or humans. Side effects should be isolated and
|
||||||
explicitly stated (in the type!). Algorithm and protocol
|
explicitly stated (in the type!). Algorithm and protocol
|
||||||
implementations should not deal with side effects internally, but leave this to an
|
implementations should not deal with side effects internally, but leave this to an
|
||||||
effectful layer on top of it. The internal pure functions
|
effectful layer on top of it. The internal pure functions
|
||||||
(which receive arguments and return values, no other way of communication) inside
|
(which receive arguments and return values, no other way of communication) inside
|
||||||
preserve [*referential
|
preserve <a href="https://en.wikipedia.org/wiki/Referential_transparency_%28computer_science%29"><em>referential
|
||||||
transparency*](https://en.wikipedia.org/wiki/Referential_transparency_%28computer_science%29).
|
transparency</em></a>.
|
||||||
Modularity helps to separate the concerns.
|
Modularity helps to separate the concerns.</p>
|
||||||
|
<p>The holy grail is <a href="https://en.wikipedia.org/wiki/Declarative_programming">declarative programing</a>, write <em>what</em>
|
||||||
The holy grail is [declarative programing](https://en.wikipedia.org/wiki/Declarative_programming), write *what*
|
a program should achieve, not <em>how to</em> achieve it (like often done in an imperative language).</p>
|
||||||
a program should achieve, not *how to* achieve it (like often done in an imperative language).
|
<p>OCaml has a object and class system, which I do not use. OCaml also contains
|
||||||
|
exceptions (and annoyingly the standard library (e.g. <code>List.find</code>) is full of
|
||||||
OCaml has a object and class system, which I do not use. OCaml also contains
|
|
||||||
exceptions (and annoyingly the standard library (e.g. `List.find`) is full of
|
|
||||||
them), which I avoid as well. Libraries should not expose any exception (apart from out of memory, a really exceptional situation). If your
|
them), which I avoid as well. Libraries should not expose any exception (apart from out of memory, a really exceptional situation). If your
|
||||||
code might end up in an error state (common for parsers which process input
|
code might end up in an error state (common for parsers which process input
|
||||||
from the network), return a variant type as value (`type ('a, 'b) result = Ok of 'a | Error of 'b`).
|
from the network), return a variant type as value (<code>type ('a, 'b) result = Ok of 'a | Error of 'b</code>).
|
||||||
That way, the caller has to handle
|
That way, the caller has to handle
|
||||||
both the success and failure case explicitly.
|
both the success and failure case explicitly.</p>
|
||||||
|
<h2 id="where-to-start">Where to start?</h2>
|
||||||
## Where to start?
|
<p>The <a href="https://ocaml.org">OCaml website</a> contains a <a href="https://ocaml.org/learn/tutorials/">variety of
|
||||||
|
tutorials</a> and examples, including
|
||||||
The [OCaml website](https://ocaml.org) contains a [variety of
|
<a href="https://ocaml.org/learn/tutorials/get_up_and_running.html">introductionary
|
||||||
tutorials](https://ocaml.org/learn/tutorials/) and examples, including
|
material</a> how to get
|
||||||
[introductionary
|
|
||||||
material](https://ocaml.org/learn/tutorials/get_up_and_running.html) how to get
|
|
||||||
started with a new library. Editor integration (at least for emacs, vim, and
|
started with a new library. Editor integration (at least for emacs, vim, and
|
||||||
atom) is via [merlin](https://github.com/the-lambda-church/merlin/wiki)
|
atom) is via <a href="https://github.com/the-lambda-church/merlin/wiki">merlin</a>
|
||||||
available.
|
available.</p>
|
||||||
|
<p>A very good starting book is <a href="http://ocaml-book.com/">OCaml from the very
|
||||||
A very good starting book is [OCaml from the very
|
beginning</a> to learn the functional ideas in OCaml (also
|
||||||
beginning](http://ocaml-book.com/) to learn the functional ideas in OCaml (also
|
its successor <a href="http://ocaml-book.com/more-ocaml-algorithms-methods-diversions/">More
|
||||||
its successor [More
|
OCaml</a>).
|
||||||
OCaml](http://ocaml-book.com/more-ocaml-algorithms-methods-diversions/)).
|
Another good book is <a href="https://realworldocaml.org">real world OCaml</a>, though it
|
||||||
Another good book is [real world OCaml](https://realworldocaml.org), though it
|
is focussed around the "core" library (which I do not recommend due to its
|
||||||
is focussed around the "core" library (which I do not recommend due to its
|
huge size).</p>
|
||||||
huge size).
|
<p>There are <a href="https://ocaml.org/learn/tutorials/guidelines.html">programming
|
||||||
|
guidelines</a>, best to re-read
|
||||||
There are [programming
|
on a regular schedule. Daniel wrote <a href="http://erratique.ch/software/rresult/doc/Rresult.html#usage">guidelines</a> how to handle with errors and results.</p>
|
||||||
guidelines](https://ocaml.org/learn/tutorials/guidelines.html), best to re-read
|
<p><a href="https://opam.ocaml.org">Opam</a> is the OCaml package manager.
|
||||||
on a regular schedule. Daniel wrote [guidelines](http://erratique.ch/software/rresult/doc/Rresult.html#usage) how to handle with errors and results.
|
The <a href="https://opam.ocaml.org/packages/">opam repository</a> contains over 1000
|
||||||
|
|
||||||
[Opam](https://opam.ocaml.org) is the OCaml package manager.
|
|
||||||
The [opam repository](https://opam.ocaml.org/packages/) contains over 1000
|
|
||||||
libraries. The quality varies, I personally like the small libraries done by
|
libraries. The quality varies, I personally like the small libraries done by
|
||||||
[Daniel Bünzli](http://erratique.ch/software), as well as our
|
<a href="http://erratique.ch/software">Daniel Bünzli</a>, as well as our
|
||||||
[nqsb](https://nqsb.io) libraries (see [mirleft org](https://github.com/mirleft)),
|
<a href="https://nqsb.io">nqsb</a> libraries (see <a href="https://github.com/mirleft">mirleft org</a>),
|
||||||
[notty](https://github.com/pqwy/notty).
|
<a href="https://github.com/pqwy/notty">notty</a>.
|
||||||
A concise library (not much code),
|
A concise library (not much code),
|
||||||
including tests, documentation, etc. is
|
including tests, documentation, etc. is
|
||||||
[hkdf](https://github.com/hannesm/ocaml-hkdf). For testing I currently prefer
|
<a href="https://github.com/hannesm/ocaml-hkdf">hkdf</a>. For testing I currently prefer
|
||||||
[alcotest](https://github.com/mirage/alcotest). For cooperative tasks,
|
<a href="https://github.com/mirage/alcotest">alcotest</a>. For cooperative tasks,
|
||||||
[lwt](https://github.com/ocsigen/lwt) is decent (though it is a bit convoluted by
|
<a href="https://github.com/ocsigen/lwt">lwt</a> is decent (though it is a bit convoluted by
|
||||||
integrating too many features).
|
integrating too many features).</p>
|
||||||
|
<p>I try to stay away from big libraries such as ocamlnet, core, extlib, batteries.
|
||||||
I try to stay away from big libraries such as ocamlnet, core, extlib, batteries.
|
|
||||||
When I develop a library I do not want to force anyone into using such large
|
When I develop a library I do not want to force anyone into using such large
|
||||||
code bases. Since opam is widely used, distributing libraries became easier,
|
code bases. Since opam is widely used, distributing libraries became easier,
|
||||||
thus the trend is towards small libraries (such as
|
thus the trend is towards small libraries (such as
|
||||||
[astring](http://erratique.ch/software/astring),
|
<a href="http://erratique.ch/software/astring">astring</a>,
|
||||||
[ptime](http://erratique.ch/software/ptime),
|
<a href="http://erratique.ch/software/ptime">ptime</a>,
|
||||||
[PBKDF](https://github.com/abeaumont/ocaml-pbkdf), [scrypt](https://github.com/abeaumont/ocaml-scrypt-kdf)).
|
<a href="https://github.com/abeaumont/ocaml-pbkdf">PBKDF</a>, <a href="https://github.com/abeaumont/ocaml-scrypt-kdf">scrypt</a>).</p>
|
||||||
|
<p>What is needed? This depends on your concrete goal. There are lots of
|
||||||
What is needed? This depends on your concrete goal. There are lots of
|
issues in lots of libraries, the MirageOS project also has a <a href="https://github.com/mirage/mirage-www/wiki/Pioneer-Projects">list of
|
||||||
issues in lots of libraries, the MirageOS project also has a [list of
|
Pioneer projects</a> which
|
||||||
Pioneer projects](https://github.com/mirage/mirage-www/wiki/Pioneer-Projects) which
|
would be useful to have. I personally would like to have a native <a href="https://tools.ietf.org/html/rfc4422">simple
|
||||||
would be useful to have. I personally would like to have a native [simple
|
authentication and security layer (SASL)</a>
|
||||||
authentication and security layer (SASL)](https://tools.ietf.org/html/rfc4422)
|
implementation in OCaml soon (amongst other things, such as using an <a href="https://github.com/mirage/mirage/issues/489">ELF section for
|
||||||
implementation in OCaml soon (amongst other things, such as using an [ELF section for
|
data</a>,
|
||||||
data](https://github.com/mirage/mirage/issues/489),
|
<a href="https://github.com/mirage/mirage-platform/issues/118">strtod</a>).</p>
|
||||||
[strtod](https://github.com/mirage/mirage-platform/issues/118)).
|
<p>A <a href="https://github.com/rudenoise/mirage-dashboard">dashboard</a> for MirageOS is
|
||||||
|
|
||||||
A [dashboard](https://github.com/rudenoise/mirage-dashboard) for MirageOS is
|
|
||||||
under development, which will hopefully ease tracking of what is being actively
|
under development, which will hopefully ease tracking of what is being actively
|
||||||
developed within MirageOS. Because I'm impatient, I setup an [atom
|
developed within MirageOS. Because I'm impatient, I setup an <a href="https://github.com/miragebot.private.atom?token=ARh4hnusZ1kC_bQ_Q6_HUzQteEEGTqy8ks61Fm2LwA==">atom
|
||||||
feed](https://github.com/miragebot.private.atom?token=ARh4hnusZ1kC_bQ_Q6_HUzQteEEGTqy8ks61Fm2LwA==)
|
feed</a>
|
||||||
which watches lots of MirageOS-related repositories.
|
which watches lots of MirageOS-related repositories.</p>
|
||||||
|
<p>I hope I gave some insight into OCaml, and why I currently enjoy it. A longer read on applicability of OCaml is our Usenix 2015 paper
|
||||||
I hope I gave some insight into OCaml, and why I currently enjoy it. A longer read on applicability of OCaml is our Usenix 2015 paper
|
<a href="https://nqsb.io/nqsbtls-usenix-security15.pdf">Not-quite-so-broken TLS: lessons in re-engineering a security protocol
|
||||||
[Not-quite-so-broken TLS: lessons in re-engineering a security protocol
|
|
||||||
specification and
|
specification and
|
||||||
implementation](https://nqsb.io/nqsbtls-usenix-security15.pdf). I'm interested in feedback, either via
|
implementation</a>. I'm interested in feedback, either via
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
|
<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
|
||||||
## Other updates in the MirageOS ecosystem
|
<ul>
|
||||||
|
<li>Canopy now sends out appropriate <a href="https://github.com/Engil/Canopy/pull/23">content type</a> HTTP headers
|
||||||
- Canopy now sends out appropriate [content type](https://github.com/Engil/Canopy/pull/23) HTTP headers
|
</li>
|
||||||
- [mirage-http 2.5.2](https://github.com/mirage/mirage-http/releases/tag/v2.5.2) was released to [opam](https://opam.ocaml.org/packages/mirage-http/mirage-http.2.5.2/) which fixes the resource leak
|
<li><a href="https://github.com/mirage/mirage-http/releases/tag/v2.5.2">mirage-http 2.5.2</a> was released to <a href="https://opam.ocaml.org/packages/mirage-http/mirage-http.2.5.2/">opam</a> which fixes the resource leak
|
||||||
- regression in [mirage-net-xen 1.6.0](https://github.com/mirage/mirage-net-xen/issues/39), I'm back on 1.4.1
|
</li>
|
||||||
- I stumbled upon [too large crunch for MirageOS](https://github.com/mirage/mirage/issues/396), no solution apart from using a FAT image ([putting the data into an ELF section](https://github.com/mirage/mirage/issues/489) would solve the issue, if anyone is interested in MirageOS, that'd be a great project to start with)
|
<li>regression in <a href="https://github.com/mirage/mirage-net-xen/issues/39">mirage-net-xen 1.6.0</a>, I'm back on 1.4.1
|
||||||
- unrelated, [X.509 0.5.2](https://opam.ocaml.org/packages/x509/x509.0.5.2/) fixes [this bug](https://github.com/mirleft/ocaml-x509/commit/1a1476308d24bdcc49d45c4cd9ef539ca57461d2) in certificate chain construction
|
</li>
|
||||||
|
<li>I stumbled upon <a href="https://github.com/mirage/mirage/issues/396">too large crunch for MirageOS</a>, no solution apart from using a FAT image (<a href="https://github.com/mirage/mirage/issues/489">putting the data into an ELF section</a> would solve the issue, if anyone is interested in MirageOS, that'd be a great project to start with)
|
||||||
|
</li>
|
||||||
|
<li>unrelated, <a href="https://opam.ocaml.org/packages/x509/x509.0.5.2/">X.509 0.5.2</a> fixes <a href="https://github.com/mirleft/ocaml-x509/commit/1a1476308d24bdcc49d45c4cd9ef539ca57461d2">this bug</a> in certificate chain construction
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
|
|
@ -1,66 +1,32 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Mirroring the opam repository and all tarballs
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Mirroring the opam repository and all tarballs</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Mirroring the opam repository and all tarballs" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a><a href="/tags/opam" class="tag">opam</a></div><span class="date">Published: 2022-09-29 (last updated: 2023-11-20)</span><article><p>We at <a href="https://robur.coop">robur</a> developed <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).</p>
|
||||||
author: hannes
|
<h1 id="what-is-opam-and-why-should-i-care">What is opam and why should I care?</h1>
|
||||||
tags: mirageos, deployment, opam
|
<p><a href="https://opam.ocaml.org">Opam</a> is the OCaml package manager (also used by other projects such as <a href="https://coq.inria.fr">coq</a>). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as <a href="https://github.com/ocaml/opam-repository">ocaml/opam-repository</a>, where authors of OCaml software can contribute (as pull request) their latest releases.</p>
|
||||||
abstract: Re-developing an opam cache from scratch, as a MirageOS unikernel
|
<p>When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state "my package will not work with <a href="https://erratique.ch/software/cmdliner">cmdliner</a> in version 1.2.0").</p>
|
||||||
---
|
<p>So, the opam-repository holds the metadata of lots of OCaml packages (around 4000 at the moment this article was written) with lots of versions (in total 25000) that have been released. It is used by the opam client to figure out which packages to install or upgrade (using a solver that takes the version bounds into consideration).</p>
|
||||||
|
<p>Of course, opam can use other repositories (overlays) or forks thereof. So nothing stops you from using any other opam repository. The url to the source code of each package may be a tarball, or a git repository or other version control systems.</p>
|
||||||
We at [robur](https://robur.coop) developed [opam-mirror](https://git.robur.coop/robur/opam-mirror) in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).
|
<p>The vast majority of opam packages released to the opam-repository include a link to the source tarball and a cryptographic hash of the tarball. This is crucial for security (under the assumption the opam-repository has been downloaded from a trustworthy source - check back later this year for updates on <a href="/Posts/Conex">conex</a>). At the moment, there are some weak spots in respect to security: md5 is still allowed, and the hash and the tarball are downloaded from the same server: anyone who is in control of that server can inject arbitrary malicious data. As outlined above, we're working on infrastructure which fixes the latter issue.</p>
|
||||||
|
<h1 id="how-does-the-opam-client-work">How does the opam client work?</h1>
|
||||||
# What is opam and why should I care?
|
<p>Opam, after initialisation, downloads the <code>index.tar.gz</code> from <code>https://opam.ocaml.org/index.tar.gz</code>, and uses this as the local opam universe. An <code>opam install cmdliner</code> will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: <code>https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e</code>.</p>
|
||||||
|
<h1 id="how-does-the-opam-repository-work">How does the opam repository work?</h1>
|
||||||
[Opam](https://opam.ocaml.org) is the OCaml package manager (also used by other projects such as [coq](https://coq.inria.fr)). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as [ocaml/opam-repository](https://github.com/ocaml/opam-repository), where authors of OCaml software can contribute (as pull request) their latest releases.
|
<p>According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses <code>opam admin index</code> periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was <a href="https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/">a reported downtime</a>.</p>
|
||||||
|
<p>Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS <a href="http://retreat.mirage.io">hack retreats</a> in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).</p>
|
||||||
When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state "my package will not work with [cmdliner](https://erratique.ch/software/cmdliner) in version 1.2.0").
|
<h1 id="re-developing-opam-admin-create-as-mirageos-unikernel">Re-developing "opam admin create" as MirageOS unikernel</h1>
|
||||||
|
<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
|
||||||
So, the opam-repository holds the metadata of lots of OCaml packages (around 4000 at the moment this article was written) with lots of versions (in total 25000) that have been released. It is used by the opam client to figure out which packages to install or upgrade (using a solver that takes the version bounds into consideration).
|
<p>Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.</p>
|
||||||
|
<p>In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with <a href="https://github.com/mirage/ocaml-crunch">crunch</a>, or using a remote git repository), but recent development, e.g. of <a href="https://somerandomidiot.com/blog/2022/03/04/chamelon/">chamelon</a> sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.</p>
|
||||||
Of course, opam can use other repositories (overlays) or forks thereof. So nothing stops you from using any other opam repository. The url to the source code of each package may be a tarball, or a git repository or other version control systems.
|
<p>Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.</p>
|
||||||
|
<h1 id="code-development-and-improvements">Code development and improvements</h1>
|
||||||
The vast majority of opam packages released to the opam-repository include a link to the source tarball and a cryptographic hash of the tarball. This is crucial for security (under the assumption the opam-repository has been downloaded from a trustworthy source - check back later this year for updates on [conex](/Posts/Conex)). At the moment, there are some weak spots in respect to security: md5 is still allowed, and the hash and the tarball are downloaded from the same server: anyone who is in control of that server can inject arbitrary malicious data. As outlined above, we're working on infrastructure which fixes the latter issue.
|
<p>Initially, our plan was to use <a href="https://github.com/mirage/ocaml-git">ocaml-git</a> for pulling the repository, <a href="https://github.com/yomimono/chamelon">chamelon</a> for persistent storage, and <a href="https://github.com/inhabitedtype/httpaf">httpaf</a> as web server. With <a href="https://github.com/mirage/ocaml-tar">ocaml-tar</a> recent support of <a href="https://github.com/mirage/ocaml-tar/pull/88">gzip</a> we should be all set, and done within a few days.</p>
|
||||||
|
<p>There is already a gap in the above plan: which http client to use - in the best case something similar to our <a href="https://github.com/roburio/http-lwt-client">http-lwt-client</a> - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using <a href="https://github.com/roburio/happy-eyeballs">happy-eyeballs</a> to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.</p>
|
||||||
# How does the opam client work?
|
<p>On the path (over the last month), we fixed file descriptor leaks (memory leaks) in <a href="https://github.com/dinosaure/paf-le-chien">paf</a> -- which is used as a runtime for httpaf and h2.</p>
|
||||||
|
<p>Then we ran into some trouble with chamelon (<a href="https://github.com/yomimono/chamelon/issues/11">out of memory</a>, some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive.</p>
|
||||||
Opam, after initialisation, downloads the `index.tar.gz` from `https://opam.ocaml.org/index.tar.gz`, and uses this as the local opam universe. An `opam install cmdliner` will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: `https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e`.
|
<p>We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed <a href="https://git.robur.coop/robur/git-kv">git-kv</a> which can dump and restore the git state.</p>
|
||||||
|
<p>Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well.</p>
|
||||||
# How does the opam repository work?
|
<p>Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as <a href="https://github.com/mirage/ocaml-fat">fat</a> may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed <a href="https://github.com/reynir/mirage-block-partition">mirage-block-partition</a> to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed <a href="https://git.robur.coop/reynir/oneffs">oneffs</a> which stores a single value on the block device.</p>
|
||||||
|
<p>We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains <a href="https://github.com/mirage/mirage-kv/pull/28">recent enhancements</a> of the mirage-kv API, there is a <code>get_partial</code>, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).</p>
|
||||||
According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses `opam admin index` periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was [a reported downtime](https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/).
|
<p>What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy <a href="https://github.com/roburio/tlstunnel">tlstunnel</a>) similar to how <a href="https://github.com/roburio/unipi">unipi</a> does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).</p>
|
||||||
|
<h1 id="conclusion">Conclusion</h1>
|
||||||
Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS [hack retreats](http://retreat.mirage.io) in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).
|
<p>To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide <a href="https://builds.robur.coop/job/opam-mirror">reproducible binaries</a> for solo5-hvt. You can use our opam mirror with <code>opam repository set-url default https://opam.robur.coop</code> (revert to the other with <code>opam repository set-url default https://opam.ocaml.org</code>) or use it as a backup with <code>opam repository add robur --rank 2 https://opam.robur.coop</code>.</p>
|
||||||
|
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
|
||||||
# Re-developing "opam admin create" as MirageOS unikernel
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.coop/robur/opam-mirror) as a [MirageOS unikernel](https://mirage.io). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.
|
|
||||||
|
|
||||||
Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.
|
|
||||||
|
|
||||||
In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with [crunch](https://github.com/mirage/ocaml-crunch), or using a remote git repository), but recent development, e.g. of [chamelon](https://somerandomidiot.com/blog/2022/03/04/chamelon/) sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.
|
|
||||||
|
|
||||||
Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.
|
|
||||||
|
|
||||||
# Code development and improvements
|
|
||||||
|
|
||||||
Initially, our plan was to use [ocaml-git](https://github.com/mirage/ocaml-git) for pulling the repository, [chamelon](https://github.com/yomimono/chamelon) for persistent storage, and [httpaf](https://github.com/inhabitedtype/httpaf) as web server. With [ocaml-tar](https://github.com/mirage/ocaml-tar) recent support of [gzip](https://github.com/mirage/ocaml-tar/pull/88) we should be all set, and done within a few days.
|
|
||||||
|
|
||||||
There is already a gap in the above plan: which http client to use - in the best case something similar to our [http-lwt-client](https://github.com/roburio/http-lwt-client) - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using [happy-eyeballs](https://github.com/roburio/happy-eyeballs) to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.
|
|
||||||
|
|
||||||
On the path (over the last month), we fixed file descriptor leaks (memory leaks) in [paf](https://github.com/dinosaure/paf-le-chien) -- which is used as a runtime for httpaf and h2.
|
|
||||||
|
|
||||||
Then we ran into some trouble with chamelon ([out of memory](https://github.com/yomimono/chamelon/issues/11), some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive.
|
|
||||||
|
|
||||||
We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed [git-kv](https://git.robur.coop/robur/git-kv) which can dump and restore the git state.
|
|
||||||
|
|
||||||
Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well.
|
|
||||||
|
|
||||||
Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as [fat](https://github.com/mirage/ocaml-fat) may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed [mirage-block-partition](https://github.com/reynir/mirage-block-partition) to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed [oneffs](https://git.robur.coop/reynir/oneffs) which stores a single value on the block device.
|
|
||||||
|
|
||||||
We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains [recent enhancements](https://github.com/mirage/mirage-kv/pull/28) of the mirage-kv API, there is a `get_partial`, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).
|
|
||||||
|
|
||||||
What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy [tlstunnel](https://github.com/roburio/tlstunnel)) similar to how [unipi](https://github.com/roburio/unipi) does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).
|
|
||||||
|
|
||||||
# Conclusion
|
|
||||||
|
|
||||||
To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide [reproducible binaries](https://builds.robur.coop/job/opam-mirror) for solo5-hvt. You can use our opam mirror with `opam repository set-url default https://opam.robur.coop` (revert to the other with `opam repository set-url default https://opam.ocaml.org`) or use it as a backup with `opam repository add robur --rank 2 https://opam.robur.coop`.
|
|
||||||
|
|
||||||
Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on [donations](https://robur.coop/Donate) for doing our work - everyone can contribute.
|
|
||||||
|
|
@ -1,96 +1,74 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Operating systems
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Operating systems</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Operating systems" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Operating systems</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/operating system" class="tag">operating system</a><a href="/tags/mirageos" class="tag">mirageos</a></div><span class="date">Published: 2016-04-09 (last updated: 2021-11-19)</span><article><p>Sorry to be late with this entry, but I had to fix some issues.</p>
|
||||||
author: hannes
|
<h2 id="what-is-an-operating-system">What is an operating system?</h2>
|
||||||
tags: overview, operating system, mirageos
|
<p>Wikipedia says: "An operating system (OS) is system software that manages
|
||||||
abstract: Operating systems and MirageOS
|
|
||||||
---
|
|
||||||
|
|
||||||
Sorry to be late with this entry, but I had to fix some issues.
|
|
||||||
|
|
||||||
## What is an operating system?
|
|
||||||
|
|
||||||
Wikipedia says: "An operating system (OS) is system software that manages
|
|
||||||
computer hardware and software resources and provides common services for
|
computer hardware and software resources and provides common services for
|
||||||
computer programs." Great. In other terms, it is an abstraction layer.
|
computer programs." Great. In other terms, it is an abstraction layer.
|
||||||
Applications don't need to deal with the low-level bits (device drivers) of the
|
Applications don't need to deal with the low-level bits (device drivers) of the
|
||||||
computer.
|
computer.</p>
|
||||||
|
<p>But if we look at the landscape of deployed operating systems, there is a lot
|
||||||
But if we look at the landscape of deployed operating systems, there is a lot
|
|
||||||
more going on than abstracting devices: usually this includes process management (scheduler),
|
more going on than abstracting devices: usually this includes process management (scheduler),
|
||||||
memory management (virtual memory), [C
|
memory management (virtual memory), <a href="https://en.wikipedia.org/wiki/C_standard_library">C
|
||||||
library](https://en.wikipedia.org/wiki/C_standard_library), user management
|
library</a>, user management
|
||||||
(including access control), persistent storage (file system), network stack,
|
(including access control), persistent storage (file system), network stack,
|
||||||
etc. all being part of the kernel, and executed in kernel space. A
|
etc. all being part of the kernel, and executed in kernel space. A
|
||||||
counterexample is [Minix](http://www.minix3.org/), which consists of a tiny
|
counterexample is <a href="http://www.minix3.org/">Minix</a>, which consists of a tiny
|
||||||
microkernel, and executes the above mentioned services as user-space processes.
|
microkernel, and executes the above mentioned services as user-space processes.</p>
|
||||||
|
<p>We are (or at least I am) interested in robust systems. Development is done
|
||||||
We are (or at least I am) interested in robust systems. Development is done
|
|
||||||
by humans, thus will always be error-prone. Even a proof of its functional
|
by humans, thus will always be error-prone. Even a proof of its functional
|
||||||
correctness can be flawed if the proof system is inconsistent or the
|
correctness can be flawed if the proof system is inconsistent or the
|
||||||
specification is wrong. We need to have damage control in place by striving
|
specification is wrong. We need to have damage control in place by striving
|
||||||
for the [principle of least authority](https://en.wikipedia.org/wiki/Principle_of_least_privilege).
|
for the <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">principle of least authority</a>.
|
||||||
The goods to guard is the user data (passwords, personal information, private
|
The goods to guard is the user data (passwords, personal information, private
|
||||||
mails, ...), which lives in memory.
|
mails, ...), which lives in memory.</p>
|
||||||
|
<p>A CPU contains <a href="https://en.wikipedia.org/wiki/Protection_ring">protection rings</a>,
|
||||||
A CPU contains [protection rings](https://en.wikipedia.org/wiki/Protection_ring),
|
|
||||||
where the kernel runs in ring 0 and thus has full access to the hardware,
|
where the kernel runs in ring 0 and thus has full access to the hardware,
|
||||||
including memory. A flaw in the kernel is devastating for the security of the
|
including memory. A flaw in the kernel is devastating for the security of the
|
||||||
entire system, it is part of the [trusted computing base](https://en.wikipedia.org/wiki/Trusted_computing_base)).
|
entire system, it is part of the <a href="https://en.wikipedia.org/wiki/Trusted_computing_base">trusted computing base</a>).
|
||||||
Every byte of kernel code should be carefully developed and audited. If we
|
Every byte of kernel code should be carefully developed and audited. If we
|
||||||
can contain code into areas with less authority, we should do so. Obviously,
|
can contain code into areas with less authority, we should do so. Obviously,
|
||||||
the mechanism to contain code needs to be carefully audited as well, since
|
the mechanism to contain code needs to be carefully audited as well, since
|
||||||
it will likely need to run in privileged mode.
|
it will likely need to run in privileged mode.</p>
|
||||||
|
<p>In a virtualised world, we run a
|
||||||
In a virtualised world, we run a
|
<a href="https://en.wikipedia.org/wiki/Hypervisor">hypervisor</a> in ring -1, on top of
|
||||||
[hypervisor](https://en.wikipedia.org/wiki/Hypervisor) in ring -1, on top of
|
|
||||||
which we run an operating system kernel. The hypervisor gives access to memory
|
which we run an operating system kernel. The hypervisor gives access to memory
|
||||||
and hardware to virtual machines, schedules those virtual machines on
|
and hardware to virtual machines, schedules those virtual machines on
|
||||||
processors, and should isolate the virtual machines from each other (by using
|
processors, and should isolate the virtual machines from each other (by using
|
||||||
the MMU).
|
the MMU).</p>
|
||||||
|
<p><img src="https://fsfe.org/contribute/promopics/thereisnocloud-v2-preview.png" alt="there's no cloud, just other people's computers" /></p>
|
||||||
|
<p>This ominous "cloud" uses hypervisors on huge amount of physical machines, and
|
||||||
|
|
||||||
This ominous "cloud" uses hypervisors on huge amount of physical machines, and
|
|
||||||
executes off-the-shelf operating systems as virtual machines on top. Accounting
|
executes off-the-shelf operating systems as virtual machines on top. Accounting
|
||||||
is done by resource usage (time, bandwidth, storage).
|
is done by resource usage (time, bandwidth, storage).</p>
|
||||||
|
<h2 id="from-scratch">From scratch</h2>
|
||||||
## From scratch
|
<p>Ok, now we have hypervisors which already deals with memory and scheduling. Why
|
||||||
|
|
||||||
Ok, now we have hypervisors which already deals with memory and scheduling. Why
|
|
||||||
should we have the very same functionality again in the (general purpose) operating
|
should we have the very same functionality again in the (general purpose) operating
|
||||||
system running as virtual machine?
|
system running as virtual machine?</p>
|
||||||
|
<p>Additionally, earlier in my life (back in 2005 at the Dutch hacker camp "What
|
||||||
Additionally, earlier in my life (back in 2005 at the Dutch hacker camp "What
|
the hack") I proposed (together with Andreas Bogk) to <a href="https://berlin.ccc.de/~hannes/wth.pdf">phase out UNIX before
|
||||||
the hack") I proposed (together with Andreas Bogk) to [phase out UNIX before
|
2038-01-19</a> (this is when <code>time_t</code>
|
||||||
2038-01-19](https://berlin.ccc.de/~hannes/wth.pdf) (this is when `time_t`
|
overflows, unless promoted to 64 bit), and replace it with Dylan. A <a href="http://www.citizen428.net/blog/2005/08/03/what-the-hack-recap/">random
|
||||||
overflows, unless promoted to 64 bit), and replace it with Dylan. A [random
|
comment</a> about
|
||||||
comment](http://www.citizen428.net/blog/2005/08/03/what-the-hack-recap/) about
|
our talk on the Internet is "the proposal that rewriting an entire OS in a
|
||||||
our talk on the Internet is "the proposal that rewriting an entire OS in a
|
|
||||||
language with obscure syntax was somewhat original. However, I now somewhat feel
|
language with obscure syntax was somewhat original. However, I now somewhat feel
|
||||||
a strange urge to spend some time on Dylan, which is really weird..."
|
a strange urge to spend some time on Dylan, which is really weird..."</p>
|
||||||
|
<p>Being without funding back then, we didn't get far (hugest success was a
|
||||||
Being without funding back then, we didn't get far (hugest success was a
|
<a href="https://github.com/dylan-hackers/network-night-vision/">TCP/IP</a> stack in
|
||||||
[TCP/IP](https://github.com/dylan-hackers/network-night-vision/) stack in
|
|
||||||
Dylan), and as mentioned earlier I went into formal methods and mechanised
|
Dylan), and as mentioned earlier I went into formal methods and mechanised
|
||||||
proofs of full functional correctness properties.
|
proofs of full functional correctness properties.</p>
|
||||||
|
<h3 id="mirageos">MirageOS</h3>
|
||||||
### MirageOS
|
<p>At the end of 2013, David pointed me to
|
||||||
|
<a href="https://mirage.io">MirageOS</a>, an operating system developed from scratch in the
|
||||||
At the end of 2013, David pointed me to
|
functional and statically typed language <a href="https://ocaml.org">OCaml</a>. I've not
|
||||||
[MirageOS](https://mirage.io), an operating system developed from scratch in the
|
|
||||||
functional and statically typed language [OCaml](https://ocaml.org). I've not
|
|
||||||
used much OCaml before, but some other functional programming languages.
|
used much OCaml before, but some other functional programming languages.
|
||||||
Since then, I spend nearly every day on developing OCaml libraries (with varying success on being happy
|
Since then, I spend nearly every day on developing OCaml libraries (with varying success on being happy
|
||||||
with my code). In contrast to Dylan, there are more than two people developing MirageOS.
|
with my code). In contrast to Dylan, there are more than two people developing MirageOS.</p>
|
||||||
|
<p>The idea is straightforward: use a hypervisor, and its hardware
|
||||||
The idea is straightforward: use a hypervisor, and its hardware
|
|
||||||
abstractions (virtualised input/output and network device), and execute the
|
abstractions (virtualised input/output and network device), and execute the
|
||||||
OCaml runtime directly on it. No C library included (since May 2015, see [this
|
OCaml runtime directly on it. No C library included (since May 2015, see <a href="http://lists.xenproject.org/archives/html/mirageos-devel/2014-05/msg00070.html">this
|
||||||
thread](http://lists.xenproject.org/archives/html/mirageos-devel/2014-05/msg00070.html)).
|
thread</a>).
|
||||||
The virtual machine, based on the OCaml runtime and composed of OCaml libraries,
|
The virtual machine, based on the OCaml runtime and composed of OCaml libraries,
|
||||||
uses a single address space and runs in ring 0.
|
uses a single address space and runs in ring 0.</p>
|
||||||
|
<p>As mentioned above, all code which runs in ring 0 needs to be carefully
|
||||||
As mentioned above, all code which runs in ring 0 needs to be carefully
|
|
||||||
developed and checked since a flaw in it can jeopardise the security properties
|
developed and checked since a flaw in it can jeopardise the security properties
|
||||||
of the entire system: the TCP/IP library should not have access to the private
|
of the entire system: the TCP/IP library should not have access to the private
|
||||||
key used for the TLS handshake. If we trust the OCaml runtime, especially its
|
key used for the TLS handshake. If we trust the OCaml runtime, especially its
|
||||||
|
|
@ -99,82 +77,85 @@ of the TLS subsystem: the TLS API does not expose the private key via an API
|
||||||
call, and being in a memory safe language, a library cannot read arbitrary
|
call, and being in a memory safe language, a library cannot read arbitrary
|
||||||
memory. There is no real need to isolate each library into a separate address
|
memory. There is no real need to isolate each library into a separate address
|
||||||
spaces. In my opinion, using capabilities for memory access would be a great
|
spaces. In my opinion, using capabilities for memory access would be a great
|
||||||
improvement, similar to [barrelfish](http://www.barrelfish.org). OCaml has a C
|
improvement, similar to <a href="http://www.barrelfish.org">barrelfish</a>. OCaml has a C
|
||||||
foreign function call interface which can be used to read arbitrary memory --
|
foreign function call interface which can be used to read arbitrary memory --
|
||||||
you have to take care that all C bits of the system are not malicious (it is
|
you have to take care that all C bits of the system are not malicious (it is
|
||||||
fortunately difficult to embed C code into MirageOS, thus only few bits written
|
fortunately difficult to embed C code into MirageOS, thus only few bits written
|
||||||
in C are in MirageOS (such as (loop and allocation free) [crypto
|
in C are in MirageOS (such as (loop and allocation free) <a href="https://github.com/mirleft/ocaml-nocrypto/tree/f076d4e75c56054d79b876e00b6bded06d90df86/src/native">crypto
|
||||||
primitives](https://github.com/mirleft/ocaml-nocrypto/tree/f076d4e75c56054d79b876e00b6bded06d90df86/src/native)).
|
primitives</a>).
|
||||||
To further read up on the topic, there is a [nice article about the
|
To further read up on the topic, there is a <a href="https://matildah.github.io/posts/2016-01-30-unikernel-security.html">nice article about the
|
||||||
security](https://matildah.github.io/posts/2016-01-30-unikernel-security.html).
|
security</a>.</p>
|
||||||
|
<p>This website is 12MB in size (and I didn't even bother to strip yet), which
|
||||||
This website is 12MB in size (and I didn't even bother to strip yet), which
|
includes the static CSS and JavaScript (bootstrap, jquery, fonts), <a href="https://github.com/mirage/ocaml-cohttp">HTTP</a>, <a href="https://github.com/mirleft/ocaml-tls">TLS</a> (also <a href="https://github.com/mirleft/ocaml-x509">X.509</a>, <a href="https://github.com/mirleft/ocaml-asn1-combinators">ASN.1</a>, <a href="https://github.com/mirleft/ocaml-nocrypto">crypto</a>), <a href="https://github.com/mirage/ocaml-git/">git</a> (and <a href="https://github.com/mirage/irmin">irmin</a>), <a href="https://github.com/mirage/mirage-tcpip">TCP/IP</a> libraries.
|
||||||
includes the static CSS and JavaScript (bootstrap, jquery, fonts), [HTTP](https://github.com/mirage/ocaml-cohttp), [TLS](https://github.com/mirleft/ocaml-tls) (also [X.509](https://github.com/mirleft/ocaml-x509), [ASN.1](https://github.com/mirleft/ocaml-asn1-combinators), [crypto](https://github.com/mirleft/ocaml-nocrypto)), [git](https://github.com/mirage/ocaml-git/) (and [irmin](https://github.com/mirage/irmin)), [TCP/IP](https://github.com/mirage/mirage-tcpip) libraries.
|
|
||||||
The memory management in MirageOS is
|
The memory management in MirageOS is
|
||||||
straightforward: the hypervisor provides the OCaml runtime with a chunk of memory, which
|
straightforward: the hypervisor provides the OCaml runtime with a chunk of memory, which
|
||||||
immediately takes all of it.
|
immediately takes all of it.</p>
|
||||||
|
<p>This is much simpler to configure and deploy than a UNIX operating system:
|
||||||
This is much simpler to configure and deploy than a UNIX operating system:
|
|
||||||
There is no virtual memory, no process management, no file
|
There is no virtual memory, no process management, no file
|
||||||
system (the markdown content is held in memory with irmin!), no user management in the image.
|
system (the markdown content is held in memory with irmin!), no user management in the image.</p>
|
||||||
|
<p>At compile (configuration) time, the TLS keys are baked into the image, in addition to the url of the remote
|
||||||
At compile (configuration) time, the TLS keys are baked into the image, in addition to the url of the remote
|
|
||||||
git repository, the IPv4 address and ports the image should use:
|
git repository, the IPv4 address and ports the image should use:
|
||||||
The full command line for configuring this website is: `mirage configure --no-opam --xen -i Posts -n "full stack engineer" -r git://git.robur.io/hannes/hannes.robur.coop.git --dhcp false --network 0 --ip 198.167.222.205 --netmask 255.255.255.0 --gateways 198.167.222.1 --tls 443 --port 80`.
|
The full command line for configuring this website is: <code>mirage configure --no-opam --xen -i Posts -n "full stack engineer" -r git://git.robur.io/hannes/hannes.robur.coop.git --dhcp false --network 0 --ip 198.167.222.205 --netmask 255.255.255.0 --gateways 198.167.222.1 --tls 443 --port 80</code>.
|
||||||
It relies on the fact that the TLS certificate chain and private key are in the `tls/` subdirectory, which is transformed to code and included in the image (using [crunch](https://github.com/mirage/ocaml-crunch)). An improvement would be to [use an ELF section](https://github.com/mirage/mirage/issues/489), but there is no code yet.
|
It relies on the fact that the TLS certificate chain and private key are in the <code>tls/</code> subdirectory, which is transformed to code and included in the image (using <a href="https://github.com/mirage/ocaml-crunch">crunch</a>). An improvement would be to <a href="https://github.com/mirage/mirage/issues/489">use an ELF section</a>, but there is no code yet.
|
||||||
After configuring and installing the required dependencies, a `make` builds the statically linked image.
|
After configuring and installing the required dependencies, a <code>make</code> builds the statically linked image.</p>
|
||||||
|
<p>Deployment is done via <code>xl create canopy.xl</code>. The file <code>canopy.xl</code> is automatically generated by <code>mirage --configure</code> (but might need modifications). It contains the full path to the image, the name of the bridge
|
||||||
Deployment is done via `xl create canopy.xl`. The file `canopy.xl` is automatically generated by `mirage --configure` (but might need modifications). It contains the full path to the image, the name of the bridge
|
interface, and how much memory the image can use:</p>
|
||||||
interface, and how much memory the image can use:
|
<pre><code>name = 'canopy'
|
||||||
```
|
|
||||||
name = 'canopy'
|
|
||||||
kernel = 'mir-canopy.xen'
|
kernel = 'mir-canopy.xen'
|
||||||
builder = 'linux'
|
builder = 'linux'
|
||||||
memory = 256
|
memory = 256
|
||||||
on_crash = 'preserve'
|
on_crash = 'preserve'
|
||||||
vif = [ 'bridge=br0' ]
|
vif = [ 'bridge=br0' ]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>To rephrase: instead of running on a multi-purpose operating system including processes, file system, etc., this website uses a
|
||||||
To rephrase: instead of running on a multi-purpose operating system including processes, file system, etc., this website uses a
|
|
||||||
set of libraries, which are compiled and statically
|
set of libraries, which are compiled and statically
|
||||||
linked into the virtual machine image.
|
linked into the virtual machine image.</p>
|
||||||
|
<p>MirageOS uses the module system of OCaml to define how interfaces should be, thus an
|
||||||
MirageOS uses the module system of OCaml to define how interfaces should be, thus an
|
|
||||||
application developer does not need to care whether they are using the TCP/IP
|
application developer does not need to care whether they are using the TCP/IP
|
||||||
stack written in OCaml, or the sockets API of a UNIX operating system. This
|
stack written in OCaml, or the sockets API of a UNIX operating system. This
|
||||||
also allows to compile and debug your library on UNIX using off-the-shelf tools
|
also allows to compile and debug your library on UNIX using off-the-shelf tools
|
||||||
before deploying it as a virtual machine (NB: this is a lie, since there is code
|
before deploying it as a virtual machine (NB: this is a lie, since there is code
|
||||||
which is only executed when running on Xen, and this code can be buggy) ;).
|
which is only executed when running on Xen, and this code can be buggy) ;).</p>
|
||||||
|
<p>Most of the MirageOS ecosystem is developed under MIT/ISC/BSD license, which
|
||||||
Most of the MirageOS ecosystem is developed under MIT/ISC/BSD license, which
|
allows everybody to use it for whichever project they want.</p>
|
||||||
allows everybody to use it for whichever project they want.
|
<p>Did I mention that by using less code the attack vector shrinks? In
|
||||||
|
|
||||||
Did I mention that by using less code the attack vector shrinks? In
|
|
||||||
addition to that, using a memory safe programming language, where the developer
|
addition to that, using a memory safe programming language, where the developer
|
||||||
does not need to care about memory management and bounds checks, immediately removes
|
does not need to care about memory management and bounds checks, immediately removes
|
||||||
several classes of security problems (namely spatial and temporal memory
|
several classes of security problems (namely spatial and temporal memory
|
||||||
issues), once the runtime is trusted.
|
issues), once the runtime is trusted.
|
||||||
The OCaml runtime was reviewed by the French [Agence nationale de la sécurité des systèmes d’information](http://www.ssi.gouv.fr/agence/publication/lafosec-securite-et-langages-fonctionnels/) in 2013,
|
The OCaml runtime was reviewed by the French <a href="http://www.ssi.gouv.fr/agence/publication/lafosec-securite-et-langages-fonctionnels/">Agence nationale de la sécurité des systèmes d’information</a> in 2013,
|
||||||
leading to some changes, such as separation of immutable strings (`String`) from mutable byte vectors (`Bytes`).
|
leading to some changes, such as separation of immutable strings (<code>String</code>) from mutable byte vectors (<code>Bytes</code>).</p>
|
||||||
|
<p>The attack surface is still big enough: logical issues, resource management, and there is no access
|
||||||
The attack surface is still big enough: logical issues, resource management, and there is no access
|
|
||||||
control. This website does not need access control, publishing of content is protected by relying on GitHub's
|
control. This website does not need access control, publishing of content is protected by relying on GitHub's
|
||||||
access control.
|
access control.</p>
|
||||||
|
<p>I hope I gave some insight into what the purpose of an operating systems is, and
|
||||||
I hope I gave some insight into what the purpose of an operating systems is, and
|
|
||||||
how MirageOS fits into the picture. I'm interested in feedback, either via
|
how MirageOS fits into the picture. I'm interested in feedback, either via
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
|
<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
|
||||||
## Other updates in the MirageOS ecosystem
|
<ul>
|
||||||
|
<li>this website is based on <a href="https://github.com/Engil/Canopy">Canopy</a>, the content is stored as markdown in a <a href="https://git.robur.io/hannes/hannes.robur.coop">git repository</a>
|
||||||
- this website is based on [Canopy](https://github.com/Engil/Canopy), the content is stored as markdown in a [git repository](https://git.robur.io/hannes/hannes.robur.coop)
|
</li>
|
||||||
- it was running in a [FreeBSD](https://FreeBSD.org) jail, but when I compiled too much the underlying [zfs file system](https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs.html) wasn't happy (and is now hanging in kernel space in a read)
|
<li>it was running in a <a href="https://FreeBSD.org">FreeBSD</a> jail, but when I compiled too much the underlying <a href="https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs.html">zfs file system</a> wasn't happy (and is now hanging in kernel space in a read)
|
||||||
- no remote power switch (borrowed to a friend 3 weeks ago), nobody was willing to go to the data centre and reboot
|
</li>
|
||||||
- I wanted to move it anyways to a host where I can deploy [Xen](http://www.xenproject.org/) guest VMs
|
<li>no remote power switch (borrowed to a friend 3 weeks ago), nobody was willing to go to the data centre and reboot
|
||||||
- turns out the Xen compilation and deployment mode needed some love:
|
</li>
|
||||||
- I ported a newer [bin_prot](https://github.com/hannesm/bin_prot/tree/113.33.00+xen) to xen
|
<li>I wanted to move it anyways to a host where I can deploy <a href="http://www.xenproject.org/">Xen</a> guest VMs
|
||||||
- I wrote a clean patch to [serve via TLS](https://github.com/Engil/Canopy/pull/15) (including [HSTS header](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security) and redirecting HTTP (moved permanently) to HTTPS)
|
</li>
|
||||||
- I found a memory leak in the [mirage-http](https://github.com/mirage/mirage-http/pull/23) library
|
<li>turns out the Xen compilation and deployment mode needed some love:
|
||||||
- I was travelling
|
<ul>
|
||||||
- good news: it now works on Xen, and there is [an atom feed](https://hannes.nqsb.io/atom)
|
<li>I ported a newer <a href="https://github.com/hannesm/bin_prot/tree/113.33.00+xen">bin_prot</a> to xen
|
||||||
- life of an "eat your own dogfood" full stack engineer ;)
|
</li>
|
||||||
|
<li>I wrote a clean patch to <a href="https://github.com/Engil/Canopy/pull/15">serve via TLS</a> (including <a href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HSTS header</a> and redirecting HTTP (moved permanently) to HTTPS)
|
||||||
|
</li>
|
||||||
|
<li>I found a memory leak in the <a href="https://github.com/mirage/mirage-http/pull/23">mirage-http</a> library
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
</li>
|
||||||
|
<li>I was travelling
|
||||||
|
</li>
|
||||||
|
<li>good news: it now works on Xen, and there is <a href="https://hannes.nqsb.io/atom">an atom feed</a>
|
||||||
|
</li>
|
||||||
|
<li>life of an "eat your own dogfood" full stack engineer ;)
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
100
Posts/Pinata
100
Posts/Pinata
|
|
@ -1,69 +1,45 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: The Bitcoin Piñata - no candy for you
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The Bitcoin Piñata - no candy for you</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The Bitcoin Piñata - no candy for you" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/bitcoin" class="tag">bitcoin</a></div><span class="date">Published: 2018-04-18 (last updated: 2021-11-19)</span><article><h2 id="history">History</h2>
|
||||||
author: hannes
|
<p>On February 10th 2015 David Kaloper-Meršinjak and Hannes Mehnert
|
||||||
tags: mirageos, security, bitcoin
|
<a href="https://mirage.io/announcing-bitcoin-pinata">launched</a> (read also <a href="http://amirchaudhry.com/bitcoin-pinata">Amir's
|
||||||
abstract: More than three years ago we launched our Bitcoin Piñata as a transparent security bait. It is still up and running!
|
description</a>) our <a href="https://en.wikipedia.org/wiki/Bug_bounty_program">bug bounty
|
||||||
---
|
program</a> in the form of our
|
||||||
|
<a href="http://ownme.ipredator.se">Bitcoin Piñata</a> MirageOS unikernel. Thanks again to
|
||||||
## History
|
<a href="https://ipredator.se">IPredator</a> for both hosting our services and lending us
|
||||||
|
the 10 Bitcoins! We <a href="https://mirage.io/blog/bitcoin-pinata-results">analysed</a> a
|
||||||
On February 10th 2015 David Kaloper-Meršinjak and Hannes Mehnert
|
|
||||||
[launched](https://mirage.io/announcing-bitcoin-pinata) (read also [Amir's
|
|
||||||
description](http://amirchaudhry.com/bitcoin-pinata)) our [bug bounty
|
|
||||||
program](https://en.wikipedia.org/wiki/Bug_bounty_program) in the form of our
|
|
||||||
[Bitcoin Piñata](http://ownme.ipredator.se) MirageOS unikernel. Thanks again to
|
|
||||||
[IPredator](https://ipredator.se) for both hosting our services and lending us
|
|
||||||
the 10 Bitcoins! We [analysed](https://mirage.io/blog/bitcoin-pinata-results) a
|
|
||||||
bit more in depth after running it for five months. Mindy recently wrote about
|
bit more in depth after running it for five months. Mindy recently wrote about
|
||||||
[whacking the Bitcoin
|
<a href="https://somerandomidiot.com/blog/2018/04/17/whacking-the-bitcoin-pinata/">whacking the Bitcoin
|
||||||
Piñata](https://somerandomidiot.com/blog/2018/04/17/whacking-the-bitcoin-pinata/).
|
Piñata</a>.</p>
|
||||||
|
<p>On March 18th 2018, after more than three years, IPredator, the lender of the Bitcoins, repurposed the 10 Bitcoins for other projects. Initially, we thought that the Piñata would maybe run for a month or two, but IPredator, David, and I decided to keep it running. The update of the Piñata's bounty is a good opportunity to reflect on the project.</p>
|
||||||
On March 18th 2018, after more than three years, IPredator, the lender of the Bitcoins, repurposed the 10 Bitcoins for other projects. Initially, we thought that the Piñata would maybe run for a month or two, but IPredator, David, and I decided to keep it running. The update of the Piñata's bounty is a good opportunity to reflect on the project.
|
<p>The 10 Bitcoin in the Piñata were fluctuating in price over time, at peak worth 165000€.</p>
|
||||||
|
<p>From the start of the Piñata project, we published the <a href="https://github.com/mirleft/btc-pinata">source code</a>, the virtual machine image, and the versions of the used libraries in a git repository. Everybody could develop their exploits locally before launching them against our Piñata. The Piñata provides TLS endpoints, which require private keys and certificates. These are generated by the Piñata at startup, and the secret for the Bitcoin wallet is provided as a command line argument.</p>
|
||||||
The 10 Bitcoin in the Piñata were fluctuating in price over time, at peak worth 165000€.
|
<p>Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with <a href="https://github.com/solo5/solo5">solo5</a>, and in December 2017 it was migrated to native BHyve (<a href="/Posts/Solo5">using <code>ukvm-bin</code> and solo5</a>). We also changed the Piñata code to accomodate for updates, such as the <a href="https://mirage.io/blog/announcing-mirage-30-release">MirageOS 3.0 release</a>, and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).</p>
|
||||||
|
<h2 id="motivation">Motivation</h2>
|
||||||
From the start of the Piñata project, we published the [source code](https://github.com/mirleft/btc-pinata), the virtual machine image, and the versions of the used libraries in a git repository. Everybody could develop their exploits locally before launching them against our Piñata. The Piñata provides TLS endpoints, which require private keys and certificates. These are generated by the Piñata at startup, and the secret for the Bitcoin wallet is provided as a command line argument.
|
<p>We built the Piñata for many purposes: to attract security professionals to evaluate our <a href="https://mirage.io/blog/introducing-ocaml-tls">from-scratch developed TLS stack</a>, to gather empirical data for our <a href="https://usenix15.nqsb.io">Usenix Security 15 paper</a>, and as an improvement to current bug bounty programs.</p>
|
||||||
|
<p>Most bug bounty programs require communication via forms and long wait times for
|
||||||
Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with [solo5](https://github.com/solo5/solo5), and in December 2017 it was migrated to native BHyve ([using `ukvm-bin` and solo5](/Posts/Solo5)). We also changed the Piñata code to accomodate for updates, such as the [MirageOS 3.0 release](https://mirage.io/blog/announcing-mirage-30-release), and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).
|
|
||||||
|
|
||||||
## Motivation
|
|
||||||
|
|
||||||
We built the Piñata for many purposes: to attract security professionals to evaluate our [from-scratch developed TLS stack](https://mirage.io/blog/introducing-ocaml-tls), to gather empirical data for our [Usenix Security 15 paper](https://usenix15.nqsb.io), and as an improvement to current bug bounty programs.
|
|
||||||
|
|
||||||
Most bug bounty programs require communication via forms and long wait times for
|
|
||||||
human experts to evaluate the potential bug. This evaluation is subjective,
|
human experts to evaluate the potential bug. This evaluation is subjective,
|
||||||
intransparent, and often requires signing of non-disclosure agreements (NDA),
|
intransparent, and often requires signing of non-disclosure agreements (NDA),
|
||||||
even before the evaluation starts.
|
even before the evaluation starts.</p>
|
||||||
|
<p>Our Piñata <em>automates</em> these parts, getting rid of wait times and NDAs. To get
|
||||||
Our Piñata *automates* these parts, getting rid of wait times and NDAs. To get
|
|
||||||
the private wallet key that holds the bounty, you need to successfully establish
|
the private wallet key that holds the bounty, you need to successfully establish
|
||||||
an authenticated TLS session or find a flaw elsewhere in the stack, which allows
|
an authenticated TLS session or find a flaw elsewhere in the stack, which allows
|
||||||
to read arbitrary memory. Everyone can track transactions of the blockchain and
|
to read arbitrary memory. Everyone can track transactions of the blockchain and
|
||||||
see if the wallet still contains the bounty.
|
see if the wallet still contains the bounty.</p>
|
||||||
|
<p>Of course, the Piñata can't prove that our stack is secure, and it can't prove
|
||||||
Of course, the Piñata can't prove that our stack is secure, and it can't prove
|
that the access to the wallet is actually inside. But trust us, it is!</p>
|
||||||
that the access to the wallet is actually inside. But trust us, it is!
|
<h2 id="observations">Observations</h2>
|
||||||
|
<p>I still remember vividly the first nights in February 2015, being so nervous that I woke up every two hours and checked the blockchain. Did the Piñata still have the Bitcoins? I was familiar with the code of the Piñata and was afraid there might be a bug which allows to bypass authentication or leak the private key. So far, this doesn't seem to be the case.</p>
|
||||||
## Observations
|
<p>In April 2016 we stumbled upon an <a href="/Posts/BadRecordMac">information disclosure in the virtual network
|
||||||
|
device driver for Xen in MirageOS</a>. Given enough
|
||||||
I still remember vividly the first nights in February 2015, being so nervous that I woke up every two hours and checked the blockchain. Did the Piñata still have the Bitcoins? I was familiar with the code of the Piñata and was afraid there might be a bug which allows to bypass authentication or leak the private key. So far, this doesn't seem to be the case.
|
|
||||||
|
|
||||||
In April 2016 we stumbled upon an [information disclosure in the virtual network
|
|
||||||
device driver for Xen in MirageOS](/Posts/BadRecordMac). Given enough
|
|
||||||
bandwidth, this could have been used to access the private wallet key. We
|
bandwidth, this could have been used to access the private wallet key. We
|
||||||
upgraded the Piñata and released the [MirageOS Security Advisory
|
upgraded the Piñata and released the <a href="https://mirage.io/blog/MSA00">MirageOS Security Advisory
|
||||||
00](https://mirage.io/blog/MSA00).
|
00</a>.</p>
|
||||||
|
<p>We analysed the Piñata's access logs to the and bucketed them into website traffic and bounty connections. We are still wondering what happened in July 2015 and July 2017 where the graph shows spikes. Could it be a presentation mentioning the Piñata, or a new automated tool which tests for TLS vulnerabilities, or an increase in market price for Bitcoins?</p>
|
||||||
We analysed the Piñata's access logs to the and bucketed them into website traffic and bounty connections. We are still wondering what happened in July 2015 and July 2017 where the graph shows spikes. Could it be a presentation mentioning the Piñata, or a new automated tool which tests for TLS vulnerabilities, or an increase in market price for Bitcoins?
|
<p><img src="/static/img/pinata_access_20180403.png" alt="Piñata access" /> <img src="/static/img/pinata_access_cumulative_20180403.png" alt="Piñata access cumulative" /></p>
|
||||||
|
<p>The cumulative graph shows that more than 500,000 accesses to the Piñata website, and more than 150,000 attempts at connecting to the Piñata bounty.</p>
|
||||||
 
|
<p>You can short-circuit the client and server Piñata endpoint and observe the private wallet key being transferred on your computer, TLS encrypted with the secret exchanged by client and server, using <code>socat -x TCP:ownme.ipredator.se:10000 TCP:ownme.ipredator.se:10002</code>.</p>
|
||||||
|
<p>If you attempted to exploit the Piñata, please let us know what you tried! Via
|
||||||
The cumulative graph shows that more than 500,000 accesses to the Piñata website, and more than 150,000 attempts at connecting to the Piñata bounty.
|
<strike><a href="https://twitter.com/h4nnes">twitter</a></strike>
|
||||||
|
<a href="http://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
You can short-circuit the client and server Piñata endpoint and observe the private wallet key being transferred on your computer, TLS encrypted with the secret exchanged by client and server, using `socat -x TCP:ownme.ipredator.se:10000 TCP:ownme.ipredator.se:10002`.
|
<p>Since <a href="/Posts/DNS">the start of 2018</a> we are developing robust software and systems at <a href="http://robur.io">robur</a>. If you like our work and want to support us with donations or development contracts, please get in touch with <code>team@robur.io</code>. Robur is a project of the German non-profit <a href="https://techcultivation.org">Center for the cultivation of technology</a>. Donations to robur are tax-deductible in Europe.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
If you attempted to exploit the Piñata, please let us know what you tried! Via
|
|
||||||
<strike>[twitter](https://twitter.com/h4nnes)</strike>
|
|
||||||
[hannesm@mastodon.social](http://mastodon.social/@hannesm) or via eMail.
|
|
||||||
|
|
||||||
Since [the start of 2018](/Posts/DNS) we are developing robust software and systems at [robur](http://robur.io). If you like our work and want to support us with donations or development contracts, please get in touch with `team@robur.io`. Robur is a project of the German non-profit [Center for the cultivation of technology](https://techcultivation.org). Donations to robur are tax-deductible in Europe.
|
|
||||||
|
|
@ -1,87 +1,43 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Reproducible MirageOS unikernel builds
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Reproducible MirageOS unikernel builds</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Reproducible MirageOS unikernel builds" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/package signing" class="tag">package signing</a></div><span class="date">Published: 2019-12-16 (last updated: 2021-11-19)</span><article><h2 id="reproducible-builds-summit">Reproducible builds summit</h2>
|
||||||
author: hannes
|
<p>I'm just back from the <a href="https://reproducible-builds.org/events/Marrakesh2019/">Reproducible builds summit 2019</a>. In 2018, several people developing <a href="https://ocaml.org">OCaml</a> and <a href="https://opam.ocaml.org">opam</a> and <a href="https://mirage.io">MirageOS</a>, attended <a href="https://reproducible-builds.org/events/paris2018/">the Reproducible builds summit in Paris</a>. The notes from last year on <a href="https://reproducible-builds.org/events/paris2018/report/#Toc11410_331763073">opam reproducibility</a> and <a href="https://reproducible-builds.org/events/paris2018/report/#Toc11681_331763073">MirageOS reproducibility</a> are online. After last years workshop, Raja started developing the opam reproducibilty builder <a href="https://github.com/rjbou/orb">orb</a>, which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.</p>
|
||||||
tags: mirageos, security, package signing
|
<h2 id="goal">Goal</h2>
|
||||||
abstract: MirageOS unikernels are reproducible :)
|
<p>Stepping back a bit, first look on the <a href="https://reproducible-builds.org/">goal of reproducible builds</a>: when compiling source code multiple times, the produced binaries should be identical. It should be sufficient if the binaries are behaviourally equal, but this is pretty hard to check. It is much easier to check <strong>bit-wise identity of binaries</strong>, and relaxes the burden on the checker -- checking for reproducibility is reduced to computing the hash of the binaries. Let's stick to the bit-wise identical binary definition, which also means software developers have to avoid non-determinism during compilation in their toolchains, dependent libraries, and developed code.</p>
|
||||||
---
|
<p>A <a href="https://reproducible-builds.org/docs/test-bench/">checklist</a> of potential things leading to non-determinism has been written up by the reproducible builds project. Examples include recording the build timestamp into the binary, ordering of code and embedded data. The reproducible builds project also developed <a href="https://packages.debian.org/sid/disorderfs">disorderfs</a> for testing reproducibility and <a href="https://diffoscope.org/">diffoscope</a> for comparing binaries with file-dependent readers, falling back to <code>objdump</code> and <code>hexdump</code>. A giant <a href="https://tests.reproducible-builds.org/">test infrastructure</a> with <a href="https://tests.reproducible-builds.org/debian/index_variations.html">lots of variations</a> between the builds, mostly using Debian, has been setup over the years.</p>
|
||||||
|
<p>Reproducibility is a precondition for trustworthy binaries. See <a href="https://reproducible-builds.org/#why-does-it-matter">why does it matter</a>. If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. <strong>I will only ever publish binaries if they are reproducible</strong>.</p>
|
||||||
## Reproducible builds summit
|
<p>My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of <a href="https://mirage.io">MirageOS unikernels</a> -- a unikernel is a statically linked ELF binary to be run as Unix process or <a href="https://github.com/solo5/solo5">virtual machine</a>. MirageOS heavily uses <a href="https://ocaml.org">OCaml</a> and <a href="https://opam.ocaml.org">opam</a>, the OCaml package manager, and is an opam package itself. Thus, <em>checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package</em>.</p>
|
||||||
|
<h2 id="reproducible-builds-with-opam">Reproducible builds with opam</h2>
|
||||||
I'm just back from the [Reproducible builds summit 2019](https://reproducible-builds.org/events/Marrakesh2019/). In 2018, several people developing [OCaml](https://ocaml.org) and [opam](https://opam.ocaml.org) and [MirageOS](https://mirage.io), attended [the Reproducible builds summit in Paris](https://reproducible-builds.org/events/paris2018/). The notes from last year on [opam reproducibility](https://reproducible-builds.org/events/paris2018/report/#Toc11410_331763073) and [MirageOS reproducibility](https://reproducible-builds.org/events/paris2018/report/#Toc11681_331763073) are online. After last years workshop, Raja started developing the opam reproducibilty builder [orb](https://github.com/rjbou/orb), which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.
|
<p>Testing for reproducibility is achieved by taking the sources and compile them twice independently. Afterwards the equality of the resulting binaries can be checked. In trivial projects, the sources is just a single file, or originate from a single tarball. In OCaml, opam uses <a href="https://github.com/ocaml/opam-repository">a community repository</a> where OCaml developers publish their package releases to, but can also use custom repositores, and in addition pin packages to git remotes (url including branch or commit), or a directory on the local filesystem. Manually tracking and updating all dependent packages of a MirageOS unikernel is not feasible: our hello-world compiled for hvt (kvm/BHyve) already has 79 opam dependencies, including the OCaml compiler which is distribued as opam package. The unikernel serving this website depends on 175 opam packages.</p>
|
||||||
|
<p>Conceptually there should be two tools, the <em>initial builder</em>, which takes the latest opam packages which do not conflict, and exports exact package versions used during the build, as well as hashes of binaries. The other tool is a <em>rebuilder</em>, which imports the export, conducts a build, and outputs the hashes of the produced binaries.</p>
|
||||||
## Goal
|
<p>Opam has the concept of a <code>switch</code>, which is an environment where a package set is installed. Switches are independent of each other, and can already be exported and imported. Unfortunately the export is incomplete: if a package includes additional patches as part of the repository -- sometimes needed for fixing releases where the actual author or maintainer of a package responds slowly -- these package neither the patches end up in the export. Also, if a package is pinned to a git branch, the branch appears in the export, but this may change over time by pushing more commits or even force-pushing to that branch. In <a href="https://github.com/ocaml/opam/pull/4040">PR #4040</a> (under discussion and review), also developed during the summit, I propose to embed the additional files as base64 encoded values in the opam file. To solve the latter issue, I modified the export mechanism to <a href="https://github.com/ocaml/opam/pull/4055">embed the git commit hash (PR #4055)</a>, and avoid sources from a local directory and which do not have a checksum.</p>
|
||||||
|
<p>So the opam export contains the information required to gather the exact same sources and build instructions of the opam packages. If the opam repository would be self-contained (i.e. not depend on any other tools), this would be sufficient. But opam does not run in thin air, it requires some system utilities such as <code>/bin/sh</code>, <code>sed</code>, a GNU make, commonly <code>git</code>, a C compiler, a linker, an assembler. Since opam is available on various operating systems, the plugin <code>depext</code> handles host system dependencies, e.g. if your opam package requires <code>gmp</code> to be installed, this requires slightly different names depending on host system or distribution, take a look at <a href="https://github.com/ocaml/opam-repository/blob/master/packages/conf-gmp/conf-gmp.1/opam">conf-gmp</a>. This also means, opam has rather good information about both the opam dependencies and the host system dependencies for each package. Please note that the host system packages used during compilation are not yet recorded (i.e. which <code>gmp</code> package was installed and used during the build, only that a <code>gmp</code> package has to be installed). The base utilities mentioned above (C compiler, linker, shell) are also not recorded yet.</p>
|
||||||
Stepping back a bit, first look on the [goal of reproducible builds](https://reproducible-builds.org/): when compiling source code multiple times, the produced binaries should be identical. It should be sufficient if the binaries are behaviourally equal, but this is pretty hard to check. It is much easier to check **bit-wise identity of binaries**, and relaxes the burden on the checker -- checking for reproducibility is reduced to computing the hash of the binaries. Let's stick to the bit-wise identical binary definition, which also means software developers have to avoid non-determinism during compilation in their toolchains, dependent libraries, and developed code.
|
<p>Operating system information available in opam (such as architecture, distribution, version), which in some cases maps to exact base utilities, is recorded in the build-environment, a separate artifact. The environment variable <a href="https://reproducible-builds.org/specs/source-date-epoch/"><code>SOURCE_DATE_EPOCH</code></a>, used for communicating the same timestamp when software is required to record a timestamp into the resulting binary, is also captured in the build environment.</p>
|
||||||
|
<p>Additional environment variables may be captured or used by opam packages to produce different output. To avoid this, both the initial builder and the rebuilder are run with minimal environment variables: only <code>PATH</code> (normalised to a whitelist of <code>/bin</code>, <code>/usr/bin</code>, <code>/usr/local/bin</code> and <code>/opt/bin</code>) and <code>HOME</code> are defined. Missing information at the moment includes CPU features: some libraries (gmp?, nocrypto) emit different code depending on the CPU feature.</p>
|
||||||
A [checklist](https://reproducible-builds.org/docs/test-bench/) of potential things leading to non-determinism has been written up by the reproducible builds project. Examples include recording the build timestamp into the binary, ordering of code and embedded data. The reproducible builds project also developed [disorderfs](https://packages.debian.org/sid/disorderfs) for testing reproducibility and [diffoscope](https://diffoscope.org/) for comparing binaries with file-dependent readers, falling back to `objdump` and `hexdump`. A giant [test infrastructure](https://tests.reproducible-builds.org/) with [lots of variations](https://tests.reproducible-builds.org/debian/index_variations.html) between the builds, mostly using Debian, has been setup over the years.
|
<h2 id="tooling">Tooling</h2>
|
||||||
|
<p><em>TL;DR: A <strong>build</strong> builds an opam package, and outputs <code>.opam-switch</code>, <code>.build-hashes.N</code>, and <code>.build-environment.N</code>. A <strong>rebuild</strong> uses these artifacts as input, builds the package and outputs another <code>.build-hashes.M</code> and <code>.build-environment.M</code>.</em></p>
|
||||||
Reproducibility is a precondition for trustworthy binaries. See [why does it matter](https://reproducible-builds.org/#why-does-it-matter). If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. **I will only ever publish binaries if they are reproducible**.
|
<p>The command-line utility <code>orb</code> can be installed and used:</p>
|
||||||
|
<pre><code class="language-sh">$ opam pin add orb git+https://github.com/hannesm/orb.git#active
|
||||||
My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of [MirageOS unikernels](https://mirage.io) -- a unikernel is a statically linked ELF binary to be run as Unix process or [virtual machine](https://github.com/solo5/solo5). MirageOS heavily uses [OCaml](https://ocaml.org) and [opam](https://opam.ocaml.org), the OCaml package manager, and is an opam package itself. Thus, *checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package*.
|
$ orb build --twice --keep-build-dir --diffoscope <your-favourite-opam-package>
|
||||||
|
</code></pre>
|
||||||
## Reproducible builds with opam
|
<p>It provides two subcommands <code>build</code> and <code>rebuild</code>. The <code>build</code> command takes a list of local opam <code>--repos</code> where to take opam packages from (defaults to <code>default</code>), a compiler (either a variant <code>--compiler=4.09.0+flambda</code>, a version <code>--compiler=4.06.0</code>, or a pin to a local development version <code>--compiler-pin=~/ocaml</code>), and optionally an existing switch <code>--use-switch</code>. It creates a switch, builds the packages, and emits the opam export, hashes of all files installed by these packages, and the build environment. The flags <code>--keep-build</code> retains the build products, opam's <code>--keep-build-dir</code> in addition temporary build products and generated source code. If <code>--twice</code> is provided, a rebuild (described next) is executed after the initial build.</p>
|
||||||
|
<p>The <code>rebuild</code> command takes a directory with the opam export and build environment to build the opam package. It first compares the build-environment with the host system, sets the <code>SOURCE_DATE_EPOCH</code> and switch location accordingly and executes the import. Once the build is finished, it compares the hashes of the resulting files with the previous run. On divergence, if build directories were kept in the previous build, and if diffoscope is available and <code>--diffoscope</code> was provided, diffoscope is run on the diverging files. If <code>--keep-build-dir</code> was provided as well, <code>diff -ur</code> can be used to compare the temporary build and sources, including build logs.</p>
|
||||||
Testing for reproducibility is achieved by taking the sources and compile them twice independently. Afterwards the equality of the resulting binaries can be checked. In trivial projects, the sources is just a single file, or originate from a single tarball. In OCaml, opam uses [a community repository](https://github.com/ocaml/opam-repository) where OCaml developers publish their package releases to, but can also use custom repositores, and in addition pin packages to git remotes (url including branch or commit), or a directory on the local filesystem. Manually tracking and updating all dependent packages of a MirageOS unikernel is not feasible: our hello-world compiled for hvt (kvm/BHyve) already has 79 opam dependencies, including the OCaml compiler which is distribued as opam package. The unikernel serving this website depends on 175 opam packages.
|
<p>The builds are run in parallel, as opam does, this parallelism does not lead to different binaries in my experiments.</p>
|
||||||
|
<h2 id="results-and-discussion">Results and discussion</h2>
|
||||||
Conceptually there should be two tools, the *initial builder*, which takes the latest opam packages which do not conflict, and exports exact package versions used during the build, as well as hashes of binaries. The other tool is a *rebuilder*, which imports the export, conducts a build, and outputs the hashes of the produced binaries.
|
<p><strong>All MirageOS unikernels I have deployed are reproducible \o/</strong>. Also, several binaries such as <code>orb</code> itself, <code>opam</code>, <code>solo5-hvt</code>, and all <code>albatross</code> utilities are reproducible.</p>
|
||||||
|
<p>The unikernel range from hello world, web servers (e.g. this blog, getting its data on startup via a git clone to memory), authoritative DNS servers, CalDAV server. They vary in size between 79 and 200 opam packages, resulting in 2MB - 16MB big ELF binaries (including debug symbols). The <a href="https://github.com/roburio/reproducible-unikernel-repo">unikernel opam repository</a> contains some reproducible unikernels used for testing. Some work-in-progress enhancements are needed to achieve this:</p>
|
||||||
Opam has the concept of a `switch`, which is an environment where a package set is installed. Switches are independent of each other, and can already be exported and imported. Unfortunately the export is incomplete: if a package includes additional patches as part of the repository -- sometimes needed for fixing releases where the actual author or maintainer of a package responds slowly -- these package neither the patches end up in the export. Also, if a package is pinned to a git branch, the branch appears in the export, but this may change over time by pushing more commits or even force-pushing to that branch. In [PR #4040](https://github.com/ocaml/opam/pull/4040) (under discussion and review), also developed during the summit, I propose to embed the additional files as base64 encoded values in the opam file. To solve the latter issue, I modified the export mechanism to [embed the git commit hash (PR #4055)](https://github.com/ocaml/opam/pull/4055), and avoid sources from a local directory and which do not have a checksum.
|
<p>At the moment, the opam package of a MirageOS unikernel is automatically generated by <code>mirage configure</code>, but only used for tracking opam dependencies. I worked on <a href="https://github.com/mirage/mirage/pull/1022">mirage PR #1022</a> to extend the generated opam package with build and install instructions.</p>
|
||||||
|
<p>As mentioned above, if locale is set, ocamlgraph needs to be patched to emit a (locale-dependent) timestamp.</p>
|
||||||
So the opam export contains the information required to gather the exact same sources and build instructions of the opam packages. If the opam repository would be self-contained (i.e. not depend on any other tools), this would be sufficient. But opam does not run in thin air, it requires some system utilities such as `/bin/sh`, `sed`, a GNU make, commonly `git`, a C compiler, a linker, an assembler. Since opam is available on various operating systems, the plugin `depext` handles host system dependencies, e.g. if your opam package requires `gmp` to be installed, this requires slightly different names depending on host system or distribution, take a look at [conf-gmp](https://github.com/ocaml/opam-repository/blob/master/packages/conf-gmp/conf-gmp.1/opam). This also means, opam has rather good information about both the opam dependencies and the host system dependencies for each package. Please note that the host system packages used during compilation are not yet recorded (i.e. which `gmp` package was installed and used during the build, only that a `gmp` package has to be installed). The base utilities mentioned above (C compiler, linker, shell) are also not recorded yet.
|
<p>The OCaml program <a href="https://github.com/mirage/ocaml-crunch"><code>crunch</code></a> embeds a subdirectory as OCaml code into a binary, which we use in MirageOS quite regularly for static assets, etc. This plays in several ways into reproducibility: on the one hand, it needs a timestamp for its <code>last_modified</code> functionality (and adheres since <a href="https://github.com/mirage/ocaml-crunch/pull/45">June 2018</a> to the <code>SOURCE_DATE_EPOCH</code> spec, thanks to Xavier Clerc). On the other hand, it used before version 3.2.0 (released Dec 14th) hashtables for storing the file contents, where iteration is not deterministic (the insertion is not sorted), <a href="https://github.com/mirage/ocaml-crunch/pull/51">fixed in PR #51</a> by using a Map instead.</p>
|
||||||
|
<p>In functoria, a tool used to configure MirageOS devices and their dependencies, can emit a list of opam packages which were required to build the unikernel. This uses <code>opam list --required-by --installed --rec <pkgs></code>, which uses the cudf graph (<a href="https://github.com/mirage/functoria/pull/189#issuecomment-566696426">thanks to Raja for explanation</a>), that is during the rebuild dropping some packages. The <a href="https://github.com/mirage/functoria/pull/189">PR #189</a> avoids by not using the <code>--rec</code> argument, but manually computing the fixpoint.</p>
|
||||||
Operating system information available in opam (such as architecture, distribution, version), which in some cases maps to exact base utilities, is recorded in the build-environment, a separate artifact. The environment variable [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/specs/source-date-epoch/), used for communicating the same timestamp when software is required to record a timestamp into the resulting binary, is also captured in the build environment.
|
<p>Certainly, the choice of environment variables, and whether to vary them (as <a href="https://tests.reproducible-builds.org/debian/index_variations.html">debian does</a>) or to not define them (or normalise) while building, is arguably. Since MirageOS does neither support time zone nor internationalisation, there is no need to prematurely solving this issue. On related note, even with different locale settings, MirageOS unikernels are reproducible apart from an <a href="https://github.com/backtracking/ocamlgraph/pull/90">issue in ocamlgraph #90</a> embedding the output of <a href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html"><code>date</code></a>, which is different depending on <code>LANG</code> and locale (<code>LC_*</code>) settings.</p>
|
||||||
|
<p>Prior art in reproducible MirageOS unikernels is the <a href="https://github.com/mirage/qubes-mirage-firewall/">mirage-qubes-firewall</a>. Since <a href="https://github.com/mirage/qubes-mirage-firewall/commit/07ff3d61477383860216c69869a1ffee59145e45">early 2017</a> it is reproducible. Their approach is different by building in a docker container with the opam repository pinned to an exact git commit.</p>
|
||||||
Additional environment variables may be captured or used by opam packages to produce different output. To avoid this, both the initial builder and the rebuilder are run with minimal environment variables: only `PATH` (normalised to a whitelist of `/bin`, `/usr/bin`, `/usr/local/bin` and `/opt/bin`) and `HOME` are defined. Missing information at the moment includes CPU features: some libraries (gmp?, nocrypto) emit different code depending on the CPU feature.
|
<h2 id="further-work">Further work</h2>
|
||||||
|
<p>I only tested a certain subset of opam packages and MirageOS unikernels, mainly on a single machine (my laptop) running FreeBSD, and am happy if others will test reproducibility of their OCaml programs with the tools provided. There could as well be CI machines rebuilding opam packages and reporting results to a central repository. I'm pretty sure there are more reproducibility issues in the opam ecosystem. I developed an <a href="https://github.com/roburio/reproducible-testing-repo">reproducible testing opam repository</a> with opam packages that do not depend on OCaml, mainly for further tooling development. Some tests were also conducted on a Debian system with the same result. The variations, apart from build time, were using a different user, and different locale settings.</p>
|
||||||
## Tooling
|
<p>As mentioned above, more environment, such as the CPU features, and external system packages, should be captured in the build environment.</p>
|
||||||
|
<p>When comparing OCaml libraries, some output files (cmt / cmti / cma / cmxa) are not deterministic, but contain minimal diverge where I was not able to spot the root cause. It would be great to fix this, likely in the OCaml compiler distribution. Since the final result, the binary I'm interested in, is not affected by non-identical intermediate build products, I hope someone (you?) is interested in improving on this side. OCaml bytecode output also seems to be non-deterministic. There is <a href="https://github.com/coq/coq/issues/11229">a discussion on the coq issue tracker</a> which may be related.</p>
|
||||||
*TL;DR: A **build** builds an opam package, and outputs `.opam-switch`, `.build-hashes.N`, and `.build-environment.N`. A **rebuild** uses these artifacts as input, builds the package and outputs another `.build-hashes.M` and `.build-environment.M`.*
|
<p>In contrast to initial plans, I did not used the <a href="https://reproducible-builds.org/specs/build-path-prefix-map/"><code>BUILD_PATH_PREFIX_MAP</code></a> environment variable, which is implemented in OCaml by <a href="https://github.com/ocaml/ocaml/pull/1515">PR #1515</a> (and followups). The main reasons are that something in the OCaml toolchain (I suspect the bytecode interpreter) needed absolute paths to find libraries, thus I'd need a symlink from the left-hand side to the current build directory, which was tedious. Also, my installed assembler does not respect the build path prefix map, and BUILD_PATH_PREFIX_MAP is not widely supported. See e.g. the Debian <a href="https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ocaml-zarith.html">zarith</a> package with different build paths and its effects on the binary.</p>
|
||||||
|
<p>I'm fine with recording the build path (switch location) in the build environment for now - it turns out to end up only once in MirageOS unikernels, likely by the last linking step, which <a href="http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html">hopefully soon be solved by llvm 9.0</a>.</p>
|
||||||
The command-line utility `orb` can be installed and used:
|
<p>What was fun was to compare the unikernel when built on Linux with gcc against a built on FreeBSD with clang and lld - spoiler: they emit debug sections with different dwarf versions, it is pretty big. Other fun differences were between OCaml compiler versions: the difference between minor versions (4.08.0 vs 4.08.1) is pretty small (~100kB as human-readable output), while the difference between major version (4.08.1 vs 4.09.0) is rather big (~900kB as human-readable diff).</p>
|
||||||
|
<p>An item on my list for the future is to distribute the opam export, build hashes and build environment artifacts in a authenticated way. I want to integrate this as <a href="https://in-toto.io/">in-toto</a> style into <a href="https://github.com/hannesm/conex">conex</a>, my not-yet-deployed implementation of <a href="https://theupdateframework.github.io/">tuf</a> for opam that needs further development and a test installation, hopefully in 2020.</p>
|
||||||
```sh
|
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
$ opam pin add orb git+https://github.com/hannesm/orb.git#active
|
</article></div></div></main></body></html>
|
||||||
$ orb build --twice --keep-build-dir --diffoscope <your-favourite-opam-package>
|
|
||||||
```
|
|
||||||
|
|
||||||
It provides two subcommands `build` and `rebuild`. The `build` command takes a list of local opam `--repos` where to take opam packages from (defaults to `default`), a compiler (either a variant `--compiler=4.09.0+flambda`, a version `--compiler=4.06.0`, or a pin to a local development version `--compiler-pin=~/ocaml`), and optionally an existing switch `--use-switch`. It creates a switch, builds the packages, and emits the opam export, hashes of all files installed by these packages, and the build environment. The flags `--keep-build` retains the build products, opam's `--keep-build-dir` in addition temporary build products and generated source code. If `--twice` is provided, a rebuild (described next) is executed after the initial build.
|
|
||||||
|
|
||||||
The `rebuild` command takes a directory with the opam export and build environment to build the opam package. It first compares the build-environment with the host system, sets the `SOURCE_DATE_EPOCH` and switch location accordingly and executes the import. Once the build is finished, it compares the hashes of the resulting files with the previous run. On divergence, if build directories were kept in the previous build, and if diffoscope is available and `--diffoscope` was provided, diffoscope is run on the diverging files. If `--keep-build-dir` was provided as well, `diff -ur` can be used to compare the temporary build and sources, including build logs.
|
|
||||||
|
|
||||||
The builds are run in parallel, as opam does, this parallelism does not lead to different binaries in my experiments.
|
|
||||||
|
|
||||||
## Results and discussion
|
|
||||||
|
|
||||||
**All MirageOS unikernels I have deployed are reproducible \o/**. Also, several binaries such as `orb` itself, `opam`, `solo5-hvt`, and all `albatross` utilities are reproducible.
|
|
||||||
|
|
||||||
The unikernel range from hello world, web servers (e.g. this blog, getting its data on startup via a git clone to memory), authoritative DNS servers, CalDAV server. They vary in size between 79 and 200 opam packages, resulting in 2MB - 16MB big ELF binaries (including debug symbols). The [unikernel opam repository](https://github.com/roburio/reproducible-unikernel-repo) contains some reproducible unikernels used for testing. Some work-in-progress enhancements are needed to achieve this:
|
|
||||||
|
|
||||||
At the moment, the opam package of a MirageOS unikernel is automatically generated by `mirage configure`, but only used for tracking opam dependencies. I worked on [mirage PR #1022](https://github.com/mirage/mirage/pull/1022) to extend the generated opam package with build and install instructions.
|
|
||||||
|
|
||||||
As mentioned above, if locale is set, ocamlgraph needs to be patched to emit a (locale-dependent) timestamp.
|
|
||||||
|
|
||||||
The OCaml program [`crunch`](https://github.com/mirage/ocaml-crunch) embeds a subdirectory as OCaml code into a binary, which we use in MirageOS quite regularly for static assets, etc. This plays in several ways into reproducibility: on the one hand, it needs a timestamp for its `last_modified` functionality (and adheres since [June 2018](https://github.com/mirage/ocaml-crunch/pull/45) to the `SOURCE_DATE_EPOCH` spec, thanks to Xavier Clerc). On the other hand, it used before version 3.2.0 (released Dec 14th) hashtables for storing the file contents, where iteration is not deterministic (the insertion is not sorted), [fixed in PR #51](https://github.com/mirage/ocaml-crunch/pull/51) by using a Map instead.
|
|
||||||
|
|
||||||
In functoria, a tool used to configure MirageOS devices and their dependencies, can emit a list of opam packages which were required to build the unikernel. This uses `opam list --required-by --installed --rec <pkgs>`, which uses the cudf graph ([thanks to Raja for explanation](https://github.com/mirage/functoria/pull/189#issuecomment-566696426)), that is during the rebuild dropping some packages. The [PR #189](https://github.com/mirage/functoria/pull/189) avoids by not using the `--rec` argument, but manually computing the fixpoint.
|
|
||||||
|
|
||||||
Certainly, the choice of environment variables, and whether to vary them (as [debian does](https://tests.reproducible-builds.org/debian/index_variations.html)) or to not define them (or normalise) while building, is arguably. Since MirageOS does neither support time zone nor internationalisation, there is no need to prematurely solving this issue. On related note, even with different locale settings, MirageOS unikernels are reproducible apart from an [issue in ocamlgraph #90](https://github.com/backtracking/ocamlgraph/pull/90) embedding the output of [`date`](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html), which is different depending on `LANG` and locale (`LC_*`) settings.
|
|
||||||
|
|
||||||
Prior art in reproducible MirageOS unikernels is the [mirage-qubes-firewall](https://github.com/mirage/qubes-mirage-firewall/). Since [early 2017](https://github.com/mirage/qubes-mirage-firewall/commit/07ff3d61477383860216c69869a1ffee59145e45) it is reproducible. Their approach is different by building in a docker container with the opam repository pinned to an exact git commit.
|
|
||||||
|
|
||||||
## Further work
|
|
||||||
|
|
||||||
I only tested a certain subset of opam packages and MirageOS unikernels, mainly on a single machine (my laptop) running FreeBSD, and am happy if others will test reproducibility of their OCaml programs with the tools provided. There could as well be CI machines rebuilding opam packages and reporting results to a central repository. I'm pretty sure there are more reproducibility issues in the opam ecosystem. I developed an [reproducible testing opam repository](https://github.com/roburio/reproducible-testing-repo) with opam packages that do not depend on OCaml, mainly for further tooling development. Some tests were also conducted on a Debian system with the same result. The variations, apart from build time, were using a different user, and different locale settings.
|
|
||||||
|
|
||||||
As mentioned above, more environment, such as the CPU features, and external system packages, should be captured in the build environment.
|
|
||||||
|
|
||||||
When comparing OCaml libraries, some output files (cmt / cmti / cma / cmxa) are not deterministic, but contain minimal diverge where I was not able to spot the root cause. It would be great to fix this, likely in the OCaml compiler distribution. Since the final result, the binary I'm interested in, is not affected by non-identical intermediate build products, I hope someone (you?) is interested in improving on this side. OCaml bytecode output also seems to be non-deterministic. There is [a discussion on the coq issue tracker](https://github.com/coq/coq/issues/11229) which may be related.
|
|
||||||
|
|
||||||
In contrast to initial plans, I did not used the [`BUILD_PATH_PREFIX_MAP`](https://reproducible-builds.org/specs/build-path-prefix-map/) environment variable, which is implemented in OCaml by [PR #1515](https://github.com/ocaml/ocaml/pull/1515) (and followups). The main reasons are that something in the OCaml toolchain (I suspect the bytecode interpreter) needed absolute paths to find libraries, thus I'd need a symlink from the left-hand side to the current build directory, which was tedious. Also, my installed assembler does not respect the build path prefix map, and BUILD_PATH_PREFIX_MAP is not widely supported. See e.g. the Debian [zarith](https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/ocaml-zarith.html) package with different build paths and its effects on the binary.
|
|
||||||
|
|
||||||
I'm fine with recording the build path (switch location) in the build environment for now - it turns out to end up only once in MirageOS unikernels, likely by the last linking step, which [hopefully soon be solved by llvm 9.0](http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html).
|
|
||||||
|
|
||||||
What was fun was to compare the unikernel when built on Linux with gcc against a built on FreeBSD with clang and lld - spoiler: they emit debug sections with different dwarf versions, it is pretty big. Other fun differences were between OCaml compiler versions: the difference between minor versions (4.08.0 vs 4.08.1) is pretty small (~100kB as human-readable output), while the difference between major version (4.08.1 vs 4.09.0) is rather big (~900kB as human-readable diff).
|
|
||||||
|
|
||||||
An item on my list for the future is to distribute the opam export, build hashes and build environment artifacts in a authenticated way. I want to integrate this as [in-toto](https://in-toto.io/) style into [conex](https://github.com/hannesm/conex), my not-yet-deployed implementation of [tuf](https://theupdateframework.github.io/) for opam that needs further development and a test installation, hopefully in 2020.
|
|
||||||
|
|
||||||
If you want to support our work on MirageOS unikernels, please [donate to robur](https://robur.coop/Donate). I'm interested in feedback, either via [twitter](https://twitter.com/h4nnes), [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
|
|
@ -1,50 +1,24 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: The MirageOS retreat 2024
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>The MirageOS retreat 2024</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="The MirageOS retreat 2024" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>The MirageOS retreat 2024</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/community" class="tag">community</a></div><span class="date">Published: 2024-05-15 (last updated: 2024-05-16)</span><article><p>End of April I spent a week in Marrakech to work on MirageOS at our annual <a href="https://retreat.mirage.io">retreat</a>. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.</p>
|
||||||
author: hannes
|
<p>Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.</p>
|
||||||
tags: mirageos, community
|
<p>Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our <a href="https://mirage.io">website</a> only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).</p>
|
||||||
abstract: My involvement and experience of the MirageOS retreat 2024
|
<p>I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.</p>
|
||||||
---
|
<h2 id="operator-handbook">Operator handbook</h2>
|
||||||
|
<p>I sat down with some people to start off a <a href="https://mirage.github.io/operator-handbook/index.html">MirageOS operator handbook</a> - we talked about the scope and the early chapters (and what is not in scope). This is still very much work in progress, <a href="https://github.com/mirage/operator-handbook">issues</a> and pull requests are welcome. The target audience is someone who wants to execute a MirageOS unikernel with hvt on a laptop or server. It will not be necessary to install OCaml, but the focus is really on the operational aspects with binaries that we provide from our <a href="https://builds.robur.coop">reproducible build infrastructure</a>.</p>
|
||||||
End of April I spent a week in Marrakech to work on MirageOS at our annual [retreat](https://retreat.mirage.io). This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
|
<p>In contrast to earlier attempts, this time we started with the focus and the pain points that we encountered at the retreat. We didn't plan all the chapters upfront or tried to start by developing and compiling a MirageOS "hello world". I have confidence that our approach is a great start for a handbook that has been missing in the last decade.</p>
|
||||||
|
<h2 id="mollymawk">Mollymawk</h2>
|
||||||
Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.
|
<p>Also related to the deployment discussion, I re-started work from last year's retreat on a Web UI for <a href="/Posts/Albatross">Albatross</a>. Welcome <a href="https://github.com/robur-coop/mollymawk">mollymawk</a>. The idea is to have both a UI but also a HTTP API for deploying MirageOS unikernels via e.g. GitHub actions by a simple HTTP POST request. Observation and management of unikernels is also part of mollymawk. <a href="https://github.com/PizieDust">Pixie</a> worked a lot on the JavaScript and web side of it, while I put some effort into the MirageOS and Albatross interaction. Within days we got an initial prototype up and running and could show it to others. This already is able to gather information of running unikernels, the console output, and create and destroy unikernels. We will continue on mollymawk in the future.</p>
|
||||||
|
<p>From the design point, mollymawk communicates via TLS to albatross. It contains the TLS certificate and private key to access albatross (and generates temporary certificates for all the actions required). This also means that mollymawk needs to protect its secrets, otherwise anyone can mess around with albatross. Mollymawk itself is a unikernel that runs on top of albatross. What could possibly go wrong? We plan to add authentication soon to mollymawk, and can then use it for our own purposes. This will greatly improve the state of our running unikernels, since updating our unikernels from our reproducible build infrastructure will then be a click on a button, with a revert button if there are issues.</p>
|
||||||
Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our [website](https://mirage.io) only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
|
<h2 id="defunctorising-mirageos">Defunctorising MirageOS</h2>
|
||||||
|
<p>Other discussions were about porting existing OCaml code to MirageOS, and what are the hurdles. The excessive use of functors are usually cumbersome for existing projects that need to revise their internal code structure. I have some issues with the use of functors as well, especially at places where they don't seem to be necessary. A rule of thumb I learned is that a functor is great if you want to have something in the same application instantiated with different modules -- i.e. a set of string and a set of integers. Now, in MirageOS we use even for time (providing the function <code>sleep</code>), clocks (monotonic and posix time), random (random data), network interface functors. There's some history about it: mirage before 4.0 used opam packages for "cross-compilation". Since mirage 4, a monorepo is used to compile the unikernel. This allows to reconsider the options we have.</p>
|
||||||
I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.
|
<p>Talking about time - I don't think anyone wants to have two different sleep functions in their application, similar for monotonic clock and posix time. Also the existing open source unikernels only use a single time and clock. The random interface has as well some history connected, since our cryptographic library used to bundle everything in a single opam package, and required arbitrary precision integers (via zarith using gmp), and we didn't want to impose a dependency on gmp for every unikernel. Nowadays, the CSPRNG code is separated from gmp, and we can just use it. The network interface: indeed if you're using solo5 (hvt or virtio or whatnot), you will never need a unix (tap-based) network implementation, neither xen netfront/netback. Thus, the functorisation is not necessary.</p>
|
||||||
|
<p>At the retreat we discussed and thought about replacing the functors in MirageOS with other mechanisms - conditional compilation (depending on the target) or the linking trick (dune variants) or a ppx are candidates, apart from other things (maybe first-class modules fit the requirement as well).</p>
|
||||||
## Operator handbook
|
<p>Another benefit of conditional compilation and dune variants is editor support -- jumping to the definition of time (clock, etc.) now simply works because the compiler knows which code is used at runtime (in contrast to when a functor is used).</p>
|
||||||
|
<p>I got into coding a bit, and first looked into conditional compilation with the <a href="https://dune.readthedocs.io/en/stable/reference/library-dependencies.html#alternative-dependencies"><code>select</code></a> clauses from dune. They turned out to be not very flexible (you've to decide in the package what are all the options, and which one to take). I switched to <a href="https://dune.readthedocs.io/en/stable/variants.html#dune-variants">dune variants</a>, where the idea is that you provide a virtual interface, and only when compiling the executable you specify the implementation. This is great, since the mirage tool (at mirage configure time) can emit the dependencies. I went ahead and defunctorised the whole network stack - it was nice to see it is possible, but this is not easily upstreamable. There's as well at least <a href="https://github.com/ocaml/dune/issues/10460">one issue</a> in the dune variant code, I'm sure it will get debugged and fixed once we decide that dune variants are the solution we want to try out.</p>
|
||||||
I sat down with some people to start off a [MirageOS operator handbook](https://mirage.github.io/operator-handbook/index.html) - we talked about the scope and the early chapters (and what is not in scope). This is still very much work in progress, [issues](https://github.com/mirage/operator-handbook) and pull requests are welcome. The target audience is someone who wants to execute a MirageOS unikernel with hvt on a laptop or server. It will not be necessary to install OCaml, but the focus is really on the operational aspects with binaries that we provide from our [reproducible build infrastructure](https://builds.robur.coop).
|
<p>The current result is a <a href="https://lists.xenproject.org/archives/html/mirageos-devel/2024-05/msg00002.html">mail to the development list</a>, and a <a href="https://github.com/mirage/mirage/pull/1529">defunctorising time PR</a>, as well as a <a href="https://github.com/mirage/mirage/pull/1533">bootvar PR</a> (relying on other PRs).</p>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
In contrast to earlier attempts, this time we started with the focus and the pain points that we encountered at the retreat. We didn't plan all the chapters upfront or tried to start by developing and compiling a MirageOS "hello world". I have confidence that our approach is a great start for a handbook that has been missing in the last decade.
|
<p>I found the retreat very inspiring, there were lots of interest in various topics, and a lot of projects have been worked on. I'm also very happy that I managed to contribute and start some projects that will hopefully ease MirageOS adaption -- both on the deployment side, but as well on the developer experience side.</p>
|
||||||
|
<p>I'm looking forward to other writeups and future retreats, or other events where I can say hi to attendees.</p>
|
||||||
## Mollymawk
|
<p>We at <a href="https://robur.coop">Robur</a> are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by <a href="https://aenderwerk.de/donate/">donating</a> (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).</p>
|
||||||
|
<p>If you have any questions, reach us best via eMail to team AT robur DOT coop.</p>
|
||||||
Also related to the deployment discussion, I re-started work from last year's retreat on a Web UI for [Albatross](/Posts/Albatross). Welcome [mollymawk](https://github.com/robur-coop/mollymawk). The idea is to have both a UI but also a HTTP API for deploying MirageOS unikernels via e.g. GitHub actions by a simple HTTP POST request. Observation and management of unikernels is also part of mollymawk. [Pixie](https://github.com/PizieDust) worked a lot on the JavaScript and web side of it, while I put some effort into the MirageOS and Albatross interaction. Within days we got an initial prototype up and running and could show it to others. This already is able to gather information of running unikernels, the console output, and create and destroy unikernels. We will continue on mollymawk in the future.
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
From the design point, mollymawk communicates via TLS to albatross. It contains the TLS certificate and private key to access albatross (and generates temporary certificates for all the actions required). This also means that mollymawk needs to protect its secrets, otherwise anyone can mess around with albatross. Mollymawk itself is a unikernel that runs on top of albatross. What could possibly go wrong? We plan to add authentication soon to mollymawk, and can then use it for our own purposes. This will greatly improve the state of our running unikernels, since updating our unikernels from our reproducible build infrastructure will then be a click on a button, with a revert button if there are issues.
|
|
||||||
|
|
||||||
## Defunctorising MirageOS
|
|
||||||
|
|
||||||
Other discussions were about porting existing OCaml code to MirageOS, and what are the hurdles. The excessive use of functors are usually cumbersome for existing projects that need to revise their internal code structure. I have some issues with the use of functors as well, especially at places where they don't seem to be necessary. A rule of thumb I learned is that a functor is great if you want to have something in the same application instantiated with different modules -- i.e. a set of string and a set of integers. Now, in MirageOS we use even for time (providing the function `sleep`), clocks (monotonic and posix time), random (random data), network interface functors. There's some history about it: mirage before 4.0 used opam packages for "cross-compilation". Since mirage 4, a monorepo is used to compile the unikernel. This allows to reconsider the options we have.
|
|
||||||
|
|
||||||
Talking about time - I don't think anyone wants to have two different sleep functions in their application, similar for monotonic clock and posix time. Also the existing open source unikernels only use a single time and clock. The random interface has as well some history connected, since our cryptographic library used to bundle everything in a single opam package, and required arbitrary precision integers (via zarith using gmp), and we didn't want to impose a dependency on gmp for every unikernel. Nowadays, the CSPRNG code is separated from gmp, and we can just use it. The network interface: indeed if you're using solo5 (hvt or virtio or whatnot), you will never need a unix (tap-based) network implementation, neither xen netfront/netback. Thus, the functorisation is not necessary.
|
|
||||||
|
|
||||||
At the retreat we discussed and thought about replacing the functors in MirageOS with other mechanisms - conditional compilation (depending on the target) or the linking trick (dune variants) or a ppx are candidates, apart from other things (maybe first-class modules fit the requirement as well).
|
|
||||||
|
|
||||||
Another benefit of conditional compilation and dune variants is editor support -- jumping to the definition of time (clock, etc.) now simply works because the compiler knows which code is used at runtime (in contrast to when a functor is used).
|
|
||||||
|
|
||||||
I got into coding a bit, and first looked into conditional compilation with the [`select`](https://dune.readthedocs.io/en/stable/reference/library-dependencies.html#alternative-dependencies) clauses from dune. They turned out to be not very flexible (you've to decide in the package what are all the options, and which one to take). I switched to [dune variants](https://dune.readthedocs.io/en/stable/variants.html#dune-variants), where the idea is that you provide a virtual interface, and only when compiling the executable you specify the implementation. This is great, since the mirage tool (at mirage configure time) can emit the dependencies. I went ahead and defunctorised the whole network stack - it was nice to see it is possible, but this is not easily upstreamable. There's as well at least [one issue](https://github.com/ocaml/dune/issues/10460) in the dune variant code, I'm sure it will get debugged and fixed once we decide that dune variants are the solution we want to try out.
|
|
||||||
|
|
||||||
The current result is a [mail to the development list](https://lists.xenproject.org/archives/html/mirageos-devel/2024-05/msg00002.html), and a [defunctorising time PR](https://github.com/mirage/mirage/pull/1529), as well as a [bootvar PR](https://github.com/mirage/mirage/pull/1533) (relying on other PRs).
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
I found the retreat very inspiring, there were lots of interest in various topics, and a lot of projects have been worked on. I'm also very happy that I managed to contribute and start some projects that will hopefully ease MirageOS adaption -- both on the deployment side, but as well on the developer experience side.
|
|
||||||
|
|
||||||
I'm looking forward to other writeups and future retreats, or other events where I can say hi to attendees.
|
|
||||||
|
|
||||||
We at [Robur](https://robur.coop) are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by [donating](https://aenderwerk.de/donate/) (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).
|
|
||||||
|
|
||||||
If you have any questions, reach us best via eMail to team AT robur DOT coop.
|
|
||||||
112
Posts/Solo5
112
Posts/Solo5
|
|
@ -1,51 +1,33 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Minimising the virtual machine monitor
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Minimising the virtual machine monitor</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Minimising the virtual machine monitor" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Minimising the virtual machine monitor</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/future" class="tag">future</a><a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a></div><span class="date">Published: 2016-07-02 (last updated: 2021-11-19)</span><article><ul>
|
||||||
author: hannes
|
<li>Update (2016-10-19): all has been merged upstream now!
|
||||||
tags: future, mirageos, security
|
</li>
|
||||||
abstract: MirageOS solo5 multiboot native on bhyve
|
<li>Update (2016-10-30): <code>static_website_tls</code> works (TLS,HTTP,network via tap device)!
|
||||||
---
|
</li>
|
||||||
|
<li>Update (2017-02-23): no more extra remotes, Mirage3 is released!
|
||||||
- Update (2016-10-19): all has been merged upstream now!
|
</li>
|
||||||
- Update (2016-10-30): `static_website_tls` works (TLS,HTTP,network via tap device)!
|
</ul>
|
||||||
- Update (2017-02-23): no more extra remotes, Mirage3 is released!
|
<h2 id="what">What?</h2>
|
||||||
|
<p>As described <a href="/Posts/OperatingSystem">earlier</a>, MirageOS is a library operating system developed in <a href="/Posts/OCaml">OCaml</a>. The code size is already pretty small, deployments are so far either as a UNIX binary or as a Xen virtual machine.</p>
|
||||||
## What?
|
<p>Xen is a hypervisor, providing concrete device drivers for the actual hardware of a physical machine, memory management, scheduling, etc. The initial release of Xen was done in 2003, since then the code size and code complexity of Xen is growing. It also has various different mechanisms for virtualisation, hardware assisted ones or purely software based ones, some where the guest operating system needs to cooperate others where it does not need to cooperate.</p>
|
||||||
|
<p>Since 2005, Intel CPUs (as well as AMD CPUs) provide hardware assistance for virtualisation (the VT-x extension), since 2008 extended page tables (EPT) are around which allow a guest to safely access the MMU. Those features gave rise to much smaller hypervisors, such as KVM (mainly Linux), <a href="http://bhyve.org">bhyve</a> (FreeBSD), <a href="http://xhyve.org">xhyve</a> (MacOSX), <a href="http://undeadly.org/cgi?action=article&sid=20150831183826">vmm</a> (OpenBSD), which do not need to emulate the MMU and other things in software. The boot sequence in those hypervisors uses kexec or multiboot, instead of doing all the 16 bit, 32 bit, 64 bit mode changes manually.</p>
|
||||||
As described [earlier](/Posts/OperatingSystem), MirageOS is a library operating system developed in [OCaml](/Posts/OCaml). The code size is already pretty small, deployments are so far either as a UNIX binary or as a Xen virtual machine.
|
<p>MirageOS initially targeted only Xen, in 2015 there was a port to use rumpkernel (a modularised NetBSD), and 2016 <a href="https://github.com/Solo5/solo5">solo5</a> emerged where you can run MirageOS on. Solo5 comes in two shapes, either as <code>ukvm</code> on top of KVM, or as a multiboot image using <code>virtio</code> interfaces (block and network, plus a serial console). Solo5 is only ~1000 lines of code (plus dlmalloc), and ISC-licensed.</p>
|
||||||
|
<p>A recent <a href="https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_williams.pdf">paper</a> describes the advantages of a tiny virtual machine monitor in detail, namely no more <a href="http://venom.crowdstrike.com/">venom</a> like security issues since there is no legacy hardware emulated. Also, each virtual machine monitor can be customised to the unikernel running on top of it: if the unikernel does not need a block device, the monitor shouldn't contain code for it. The idea is to have one customised monitor for each unikernel.</p>
|
||||||
Xen is a hypervisor, providing concrete device drivers for the actual hardware of a physical machine, memory management, scheduling, etc. The initial release of Xen was done in 2003, since then the code size and code complexity of Xen is growing. It also has various different mechanisms for virtualisation, hardware assisted ones or purely software based ones, some where the guest operating system needs to cooperate others where it does not need to cooperate.
|
<p>While lots of people seem to like KVM and Linux, I still prefer FreeBSD, their jails, and nowadays bhyve. I finally found some time, thanks to various cleanups to the solo5 code base, to finally look into porting solo5 to FreeBSD/bhyve. It runs and can output to console.</p>
|
||||||
|
<h2 id="how">How?</h2>
|
||||||
Since 2005, Intel CPUs (as well as AMD CPUs) provide hardware assistance for virtualisation (the VT-x extension), since 2008 extended page tables (EPT) are around which allow a guest to safely access the MMU. Those features gave rise to much smaller hypervisors, such as KVM (mainly Linux), [bhyve](http://bhyve.org) (FreeBSD), [xhyve](http://xhyve.org) (MacOSX), [vmm](http://undeadly.org/cgi?action=article&sid=20150831183826) (OpenBSD), which do not need to emulate the MMU and other things in software. The boot sequence in those hypervisors uses kexec or multiboot, instead of doing all the 16 bit, 32 bit, 64 bit mode changes manually.
|
<p>These instructions are still slightly bumpy. If you've a FreeBSD with bhyve (I use FreeBSD-CURRENT), and OCaml and opam (>=1.2.2) installed, it is pretty straightforward to get solo5 running. First, I'd suggest to use a fresh opam switch in case you work on other OCaml projects: <code>opam switch -A 4.04.0 solo5</code> (followed by <code>eval `opam config env` </code> to setup some environment variables).</p>
|
||||||
|
<p>You need some software from the ports: <code>devel/pkgconf</code>, <code>devel/gmake</code>, <code>devel/binutils</code>, and <code>sysutils/grub2-bhyve</code>.</p>
|
||||||
MirageOS initially targeted only Xen, in 2015 there was a port to use rumpkernel (a modularised NetBSD), and 2016 [solo5](https://github.com/Solo5/solo5) emerged where you can run MirageOS on. Solo5 comes in two shapes, either as `ukvm` on top of KVM, or as a multiboot image using `virtio` interfaces (block and network, plus a serial console). Solo5 is only ~1000 lines of code (plus dlmalloc), and ISC-licensed.
|
<p>An <code>opam install mirage mirage-logs solo5-kernel-virtio mirage-bootvar-solo5 mirage-solo5</code> should provide you with a basic set of libraries.</p>
|
||||||
|
<p>Now you can get the <a href="https://github.com/mirage/mirage-skeleton">mirage-skeleton</a> repository, and inside of <code>device-usage/console</code>, run <code>mirage configure --no-opam --virtio</code> followed by <code>make</code>. There should be a resulting <code>mir-console.virtio</code>.</p>
|
||||||
A recent [paper](https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_williams.pdf) describes the advantages of a tiny virtual machine monitor in detail, namely no more [venom](http://venom.crowdstrike.com/) like security issues since there is no legacy hardware emulated. Also, each virtual machine monitor can be customised to the unikernel running on top of it: if the unikernel does not need a block device, the monitor shouldn't contain code for it. The idea is to have one customised monitor for each unikernel.
|
<p>Once that is in place, start your VM:</p>
|
||||||
|
<pre><code class="language-bash">sudo grub-bhyve -M 128M console
|
||||||
While lots of people seem to like KVM and Linux, I still prefer FreeBSD, their jails, and nowadays bhyve. I finally found some time, thanks to various cleanups to the solo5 code base, to finally look into porting solo5 to FreeBSD/bhyve. It runs and can output to console.
|
> multiboot (host)/home/hannes/mirage-skeleton/console/mir-console.virtio
|
||||||
|
> boot
|
||||||
## How?
|
|
||||||
|
|
||||||
These instructions are still slightly bumpy. If you've a FreeBSD with bhyve (I use FreeBSD-CURRENT), and OCaml and opam (>=1.2.2) installed, it is pretty straightforward to get solo5 running. First, I'd suggest to use a fresh opam switch in case you work on other OCaml projects: `opam switch -A 4.04.0 solo5` (followed by ``eval `opam config env` `` to setup some environment variables).
|
|
||||||
|
|
||||||
You need some software from the ports: `devel/pkgconf`, `devel/gmake`, `devel/binutils`, and `sysutils/grub2-bhyve`.
|
|
||||||
|
|
||||||
An `opam install mirage mirage-logs solo5-kernel-virtio mirage-bootvar-solo5 mirage-solo5` should provide you with a basic set of libraries.
|
|
||||||
|
|
||||||
Now you can get the [mirage-skeleton](https://github.com/mirage/mirage-skeleton) repository, and inside of `device-usage/console`, run `mirage configure --no-opam --virtio` followed by `make`. There should be a resulting `mir-console.virtio`.
|
|
||||||
|
|
||||||
Once that is in place, start your VM:
|
|
||||||
```bash
|
|
||||||
sudo grub-bhyve -M 128M console
|
|
||||||
> multiboot (host)/home/hannes/mirage-skeleton/console/mir-console.virtio
|
|
||||||
> boot
|
|
||||||
|
|
||||||
sudo bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -l com1,stdio -m 128M console
|
sudo bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -l com1,stdio -m 128M console
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The following output will appear on your controlling terminal:</p>
|
||||||
The following output will appear on your controlling terminal:
|
<pre><code class="language-bash"> | ___|
|
||||||
|
|
||||||
```bash
|
|
||||||
| ___|
|
|
||||||
__| _ \ | _ \ __ \
|
__| _ \ | _ \ __ \
|
||||||
\__ \ ( | | ( | ) |
|
\__ \ ( | | ( | ) |
|
||||||
____/\___/ _|\___/____/
|
____/\___/ _|\___/____/
|
||||||
|
|
@ -64,23 +46,21 @@ world
|
||||||
solo5_app_main() returned with 0
|
solo5_app_main() returned with 0
|
||||||
Kernel done.
|
Kernel done.
|
||||||
Goodbye!
|
Goodbye!
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Network and TLS stack works as well (tested 30th October).</p>
|
||||||
Network and TLS stack works as well (tested 30th October).
|
<h2 id="open-issues">Open issues</h2>
|
||||||
|
<ul>
|
||||||
## Open issues
|
<li>I'm not happy to require <code>ld</code> from the ports (but the one in base does not produce sensible binaries with <code>-z max-page-size=0x1000</code> <a href="https://github.com/Solo5/solo5/pull/56">related</a>)
|
||||||
|
</li>
|
||||||
- I'm not happy to require `ld` from the ports (but the one in base does not produce sensible binaries with `-z max-page-size=0x1000` [related](https://github.com/Solo5/solo5/pull/56))
|
<li>Via <a href="https://twitter.com/bhyve_dev/status/748930600581492736">twitter</a>, bhyve devs suggested to port ukvm to ubhyve. This is a great idea, to no longer depend on virtio, and get more speed. Any takers?
|
||||||
- Via [twitter](https://twitter.com/bhyve_dev/status/748930600581492736), bhyve devs suggested to port ukvm to ubhyve. This is a great idea, to no longer depend on virtio, and get more speed. Any takers?
|
</li>
|
||||||
- Debugging via gdb should be doable somehow, bhyve has [some support for gdb](https://wiki.freebsd.org/bhyve/DebuggingWithGdb), but it is unclear to me what I need to do to enter the debugger (busy looping in the VM and a gdb remote to the port opened by bhyve does not work).
|
<li>Debugging via gdb should be doable somehow, bhyve has <a href="https://wiki.freebsd.org/bhyve/DebuggingWithGdb">some support for gdb</a>, but it is unclear to me what I need to do to enter the debugger (busy looping in the VM and a gdb remote to the port opened by bhyve does not work).
|
||||||
|
</li>
|
||||||
## Conclusion
|
</ul>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
I managed to get solo5 to work with bhyve. I even use clang instead of gcc and don't need to link `libgcc.a`. :) It is great to see further development in hypervisors and virtual machine monitors. Especially thanks to [Martin Lucina](https://lucina.net) for getting things sorted.
|
<p>I managed to get solo5 to work with bhyve. I even use clang instead of gcc and don't need to link <code>libgcc.a</code>. :) It is great to see further development in hypervisors and virtual machine monitors. Especially thanks to <a href="https://lucina.net">Martin Lucina</a> for getting things sorted.</p>
|
||||||
|
<p>I'm interested in feedback, either via
|
||||||
I'm interested in feedback, either via
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
|
||||||
|
<p>There were some busy times, several pull requests are still waiting to get merged (e.g. some cosmetics in <a href="https://github.com/mirage/mirage/pull/544">mirage</a> as preconditions for treemaps and dependency diagrams), I <a href="https://github.com/mirage/mirage/pull/547">proposed</a> to use <code>sleep_ns : int64 -> unit io</code> instead of the <code>sleep : float -> unit io</code> (nobody wants floating point numbers); also an RFC for <a href="https://github.com/mirage/mirage/pull/551">random</a>, Matt Gray <a href="https://github.com/mirage/mirage/pull/548">proposed</a> to get rid of <code>CLOCK</code> (and have a <code>PCLOCK</code> and a <code>MCLOCK</code> instead). Soon there will be a major MirageOS release which breaks all the previous unikernels! :)</p>
|
||||||
## Other updates in the MirageOS ecosystem
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
There were some busy times, several pull requests are still waiting to get merged (e.g. some cosmetics in [mirage](https://github.com/mirage/mirage/pull/544) as preconditions for treemaps and dependency diagrams), I [proposed](https://github.com/mirage/mirage/pull/547) to use `sleep_ns : int64 -> unit io` instead of the `sleep : float -> unit io` (nobody wants floating point numbers); also an RFC for [random](https://github.com/mirage/mirage/pull/551), Matt Gray [proposed](https://github.com/mirage/mirage/pull/548) to get rid of `CLOCK` (and have a `PCLOCK` and a `MCLOCK` instead). Soon there will be a major MirageOS release which breaks all the previous unikernels! :)
|
|
||||||
|
|
@ -1,65 +1,31 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Summer 2019
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Summer 2019</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Summer 2019" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Summer 2019</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/package signing" class="tag">package signing</a><a href="/tags/tls" class="tag">tls</a><a href="/tags/monitoring" class="tag">monitoring</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2019-07-08 (last updated: 2021-11-19)</span><article><h2 id="working-at-robur">Working at <a href="https://robur.io">robur</a></h2>
|
||||||
author: hannes
|
<p>As announced <a href="/Posts/DNS">previously</a>, I started to work at robur early 2018. We're a collective of five people, distributed around Europe and the US, with the goal to deploy MirageOS unikernels. We do this by developing bespoke MirageOS unikernels which provide useful services, and deploy them for ourselves. We also develop new libraries and enhance existing ones and other components of MirageOS. Example unikernels include <a href="https://robur.io">our website</a> which uses <a href="https://github.com/Engil/Canopy">Canopy</a>, a <a href="https://robur.io/Our%20Work/Projects#CalDAV-Server">CalDAV server that stores entries in a git remote</a>, and <a href="https://github.com/roburio/unikernels">DNS servers</a> (the latter two are further described below).</p>
|
||||||
tags: mirageos, security, package signing, tls, monitoring, deployment
|
<p>Robur is part of the non-profit company <a href="https://techcultivation.org">Center for the Cultivation of Technology</a>, who are managing the legal and administrative sides for us. We're ourselves responsible to acquire funding to pay ourselves reasonable salaries. We received funding for CalDAV from <a href="https://prototypefund.de">prototypefund</a> and further funding from <a href="https://tarides.com">Tarides</a>, for TLS 1.3 from <a href="http://ocamllabs.io/">OCaml Labs</a>; security-audited an OCaml codebase, and received <a href="https://robur.io/Donate">donations</a>, also in the form of Bitcoins. We're looking for further funded collaborations and also contracting, mail us at <code>team@robur.io</code>. Please <a href="https://robur.io/Donate">donate</a> (tax-deductible in EU), so we can accomplish our goal of putting robust and sustainable MirageOS unikernels into production, replacing insecure legacy system that emit tons of CO<span style="vertical-align: baseline; position: relative;bottom: -0.4em;">2</span>.</p>
|
||||||
abstract: Bringing MirageOS into production, take IV monitoring, CalDAV, DNS
|
<h2 id="deploying-mirageos-unikernels">Deploying MirageOS unikernels</h2>
|
||||||
---
|
<p>While several examples are running since years (the <a href="https://mirage.io">MirageOS website</a>, <a href="http://ownme.ipredator.se">Bitcoin Piñata</a>, <a href="https://tls.nqsb.io">TLS demo server</a>, etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.</p>
|
||||||
|
<p>Service deployment is complex: you have to consider its configuration, exfiltration of logs and metrics, provisioning with valid key material (TLS certificate, hmac shared secret) and authenticators (CA certificate, ssh key fingerprint). Instead of requiring millions lines of code during orchestration (such as Kubernetes), creating the images (docker), or provisioning (ansible), why not minimise the required configuration and dependencies?</p>
|
||||||
## Working at [robur](https://robur.io)
|
<p><a href="/Posts/VMM">Earlier in this blog I introduced Albatross</a>, which serves in an enhanced version as our deployment platform on a physical machine (running 15 unikernels at the moment), I won't discuss more detail thereof in this article.</p>
|
||||||
|
<h2 id="caldav">CalDAV</h2>
|
||||||
As announced [previously](/Posts/DNS), I started to work at robur early 2018. We're a collective of five people, distributed around Europe and the US, with the goal to deploy MirageOS unikernels. We do this by developing bespoke MirageOS unikernels which provide useful services, and deploy them for ourselves. We also develop new libraries and enhance existing ones and other components of MirageOS. Example unikernels include [our website](https://robur.io) which uses [Canopy](https://github.com/Engil/Canopy), a [CalDAV server that stores entries in a git remote](https://robur.io/Our%20Work/Projects#CalDAV-Server), and [DNS servers](https://github.com/roburio/unikernels) (the latter two are further described below).
|
<p><a href="https://linse.me/">Steffi</a> and I developed in 2018 a CalDAV server. Since November 2018 we have a test installation for robur, initially running as a Unix process on a virtual machine and persisting data to files on the disk. Mid-June 2019 we migrated it to a MirageOS unikernel, thanks to great efforts in <a href="https://github.com/mirage/ocaml-git">git</a> and <a href="https://github.com/mirage/irmin">irmin</a>, unikernels can push to a remote git repository. We <a href="https://github.com/haesbaert/awa-ssh/pull/8">extended the ssh library</a> with a ssh client and <a href="https://github.com/mirage/ocaml-git/pull/362">use this in git</a>. This also means our CalDAV server is completely immutable (does not carry state across reboots, apart from the data in the remote repository) and does not have persistent state in the form of a block device. Its configuration is mainly done at compile time by the selection of libraries (syslog, monitoring, ...), and boot arguments passed to the unikernel at startup.</p>
|
||||||
|
<p>We monitored the resource usage when migrating our CalDAV server from Unix process to a MirageOS unikernel. The unikernel size is just below 10MB. The workload is some clients communicating with the server on a regular basis. We use <a href="https://grafana.com/">Grafana</a> with a <a href="https://www.influxdata.com/">influx</a> time series database to monitor virtual machines. Data is collected on the host system (<code>rusage</code> sysctl, <code>kinfo_mem</code> sysctl, <code>ifdata</code> sysctl, <code>vm_get_stats</code> BHyve statistics), and our unikernels these days emit further metrics (mostly counters: gc statistics, malloc statistics, tcp sessions, http requests and status codes).</p>
|
||||||
Robur is part of the non-profit company [Center for the Cultivation of Technology](https://techcultivation.org), who are managing the legal and administrative sides for us. We're ourselves responsible to acquire funding to pay ourselves reasonable salaries. We received funding for CalDAV from [prototypefund](https://prototypefund.de) and further funding from [Tarides](https://tarides.com), for TLS 1.3 from [OCaml Labs](http://ocamllabs.io/); security-audited an OCaml codebase, and received [donations](https://robur.io/Donate), also in the form of Bitcoins. We're looking for further funded collaborations and also contracting, mail us at `team@robur.io`. Please [donate](https://robur.io/Donate) (tax-deductible in EU), so we can accomplish our goal of putting robust and sustainable MirageOS unikernels into production, replacing insecure legacy system that emit tons of CO<span style="vertical-align: baseline; position: relative;bottom: -0.4em;">2</span>.
|
<p><a href="/static/img/crobur-june-2019.png"><img src="/static/img/crobur-june-2019.png" width="700" /></a></p>
|
||||||
|
<p>Please note that memory usage (upper right) and vm exits (lower right) use logarithmic scale. The CPU usage reduced by more than a factor of 4. The memory usage dropped by a factor of 25, and the network traffic increased - previously we stored log messages on the virtual machine itself, now we send them to a dedicated log host.</p>
|
||||||
## Deploying MirageOS unikernels
|
<p>A MirageOS unikernel, apart from a smaller attack surface, indeed uses fewer resources and actually emits less CO<span style="vertical-align: baseline; position: relative;bottom: -0.4em;">2</span> than the same service on a Unix virtual machine. So we're doing something good for the environment! :)</p>
|
||||||
|
<p>Our calendar server contains at the moment 63 events, the git repository had around 500 commits in the past month: nearly all of them from the CalDAV server itself when a client modified data via CalDAV, and two manual commits: the initial data imported from the file system, and one commit for fixing a bug of the encoder in our <a href="https://github.com/roburio/icalendar/pull/2">icalendar library</a>.</p>
|
||||||
While several examples are running since years (the [MirageOS website](https://mirage.io), [Bitcoin Piñata](http://ownme.ipredator.se), [TLS demo server](https://tls.nqsb.io), etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.
|
<p>Our CalDAV implementation is very basic, scheduling, adding attendees (which requires sending out eMail), is not supported. But it works well for us, we have individual calendars and a shared one which everyone can write to. On the client side we use macOS and iOS iCalendar, Android DAVdroid, and Thunderbird. If you like to try our CalDAV server, have a look <a href="https://github.com/roburio/caldav/tree/future/README.md">at our installation instructions</a>. Please <a href="https://github.com/roburio/caldav/issues">report issues</a> if you find issues or struggle with the installation.</p>
|
||||||
|
<h2 id="dns">DNS</h2>
|
||||||
Service deployment is complex: you have to consider its configuration, exfiltration of logs and metrics, provisioning with valid key material (TLS certificate, hmac shared secret) and authenticators (CA certificate, ssh key fingerprint). Instead of requiring millions lines of code during orchestration (such as Kubernetes), creating the images (docker), or provisioning (ansible), why not minimise the required configuration and dependencies?
|
<p>There has been more work on our DNS implementation, now <a href="https://github.com/mirage/ocaml-dns">here</a>. We included a DNS client library, and some <a href="https://github.com/roburio/unikernels/tree/future">example unikernels</a> are available. They as well require our <a href="https://github.com/roburio/git-ssh-dns-mirage3-repo">opam repository overlay</a>. Please report issues if you run into trouble while experimenting with that.</p>
|
||||||
|
<p>Most prominently is <code>primary-git</code>, a unikernel which acts as a primary authoritative DNS server (UDP and TCP). On startup, it fetches a remote git repository that contains zone files and shared hmac secrets. The zones are served, and secondary servers are notified with the respective serial numbers of the zones, authenticated using TSIG with the shared secrets. The primary server provides dynamic in-protocol updates of DNS resource records (<code>nsupdate</code>), and after successful authentication pushes the change to the remote git. To change the zone, you can just edit the zonefile and push to the git remote - with the proper pre- and post-commit-hooks an authenticated notify is send to the primary server which then pulls the git remote.</p>
|
||||||
[Earlier in this blog I introduced Albatross](/Posts/VMM), which serves in an enhanced version as our deployment platform on a physical machine (running 15 unikernels at the moment), I won't discuss more detail thereof in this article.
|
<p>Another noteworthy unikernel is <code>letsencrypt</code>, which acts as a secondary server, and whenever a TLSA record with custom type (0xFF) and a DER-encoded certificate signing request is observed, it requests a signature from letsencrypt by solving the DNS challenge. The certificate is pushed to the DNS server as TLSA record as well. The DNS implementation provides <code>ocertify</code> and <code>dns-mirage-certify</code> which use the above mechanism to retrieve valid let's encrypt certificates. The caller (unikernel or Unix command-line utility) either takes a private key directly or generates one from a (provided) seed and generates a certificate signing request. It then looks in DNS for a certificate which is still valid and matches the public key and the hostname. If such a certificate is not present, the certificate signing request is pushed to DNS (via the nsupdate protocol), authenticated using TSIG with a given secret. This way our public facing unikernels (website, this blog, TLS demo server, ..) block until they got a certificate via DNS on startup - we avoid embedding of the certificate into the unikernel image.</p>
|
||||||
|
<h2 id="monitoring">Monitoring</h2>
|
||||||
## CalDAV
|
<p>We like to gather statistics about the resource usage of our unikernels to find potential bottlenecks and observe memory leaks ;) The base for the setup is the <a href="https://github.com/mirage/metrics">metrics</a> library, which is similarly in design to the <a href="https://erratique.ch/software/logs">logs</a> library: libraries use the core to gather metrics. A different aspect is the reporter, which is globally registered and responsible for exfiltrating the data via their favourite protocol. If no reporter is registered, the work overhead is negligible.</p>
|
||||||
|
<p><a href="/static/img/crobur-june-2019-unikernel.png"><img src="/static/img/crobur-june-2019-unikernel.png" width="700" /></a></p>
|
||||||
[Steffi](https://linse.me/) and I developed in 2018 a CalDAV server. Since November 2018 we have a test installation for robur, initially running as a Unix process on a virtual machine and persisting data to files on the disk. Mid-June 2019 we migrated it to a MirageOS unikernel, thanks to great efforts in [git](https://github.com/mirage/ocaml-git) and [irmin](https://github.com/mirage/irmin), unikernels can push to a remote git repository. We [extended the ssh library](https://github.com/haesbaert/awa-ssh/pull/8) with a ssh client and [use this in git](https://github.com/mirage/ocaml-git/pull/362). This also means our CalDAV server is completely immutable (does not carry state across reboots, apart from the data in the remote repository) and does not have persistent state in the form of a block device. Its configuration is mainly done at compile time by the selection of libraries (syslog, monitoring, ...), and boot arguments passed to the unikernel at startup.
|
<p>This is a dashboard which combines both statistics gathered from the host system and various metrics from the MirageOS unikernel. The <code>monitoring</code> branch of our opam repository overlay is used together with <a href="https://github.com/hannesm/monitoring-experiments">monitoring-experiments</a>. The logs errors counter (middle right) was the icalendar parser which tried to parse its badly emitted ics (the bug is now fixed, the dashboard is from last month).</p>
|
||||||
|
<h2 id="ocaml-libraries">OCaml libraries</h2>
|
||||||
We monitored the resource usage when migrating our CalDAV server from Unix process to a MirageOS unikernel. The unikernel size is just below 10MB. The workload is some clients communicating with the server on a regular basis. We use [Grafana](https://grafana.com/) with a [influx](https://www.influxdata.com/) time series database to monitor virtual machines. Data is collected on the host system (`rusage` sysctl, `kinfo_mem` sysctl, `ifdata` sysctl, `vm_get_stats` BHyve statistics), and our unikernels these days emit further metrics (mostly counters: gc statistics, malloc statistics, tcp sessions, http requests and status codes).
|
<p>The <a href="https://github.com/hannesm/domain-name">domain-name</a> library was developed to handle RFC 1035 domain names and host names. It initially was part of the DNS code, but is now freestanding to be used in other core libraries (such as ipaddr) with a small dependency footprint.</p>
|
||||||
|
<p>The <a href="https://github.com/hannesm/gmap">GADT map</a> is a normal OCaml Map structure, but takes key-dependent value types by using a GADT. This library also was part of DNS, but is more broadly useful, we already use it in our icalendar (the data format for calendar entries in CalDAV) library, our <a href="https://git.robur.io/?p=openvpn.git;a=summary">OpenVPN</a> configuration parser uses it as well, and also <a href="https://github.com/mirleft/ocaml-x509/pull/115">x509</a> - which got reworked quite a bit recently (release pending), and there's preliminary PKCS12 support (which deserves its own article). <a href="https://github.com/hannesm/ocaml-tls">TLS 1.3</a> is available on a branch, but is not yet merged. More work is underway, hopefully with sufficient time to write more articles about it.</p>
|
||||||
[<img src="/static/img/crobur-june-2019.png" width="700" />](/static/img/crobur-june-2019.png)
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>More projects are happening as we speak, it takes time to upstream all the changes, such as monitoring, new core libraries, getting our DNS implementation released, pushing Conex into production, more features such as DNSSec, ...</p>
|
||||||
Please note that memory usage (upper right) and vm exits (lower right) use logarithmic scale. The CPU usage reduced by more than a factor of 4. The memory usage dropped by a factor of 25, and the network traffic increased - previously we stored log messages on the virtual machine itself, now we send them to a dedicated log host.
|
<p>I'm interested in feedback, either via <strike><a href="https://twitter.com/h4nnes">twitter</a></strike> <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
A MirageOS unikernel, apart from a smaller attack surface, indeed uses fewer resources and actually emits less CO<span style="vertical-align: baseline; position: relative;bottom: -0.4em;">2</span> than the same service on a Unix virtual machine. So we're doing something good for the environment! :)
|
|
||||||
|
|
||||||
Our calendar server contains at the moment 63 events, the git repository had around 500 commits in the past month: nearly all of them from the CalDAV server itself when a client modified data via CalDAV, and two manual commits: the initial data imported from the file system, and one commit for fixing a bug of the encoder in our [icalendar library](https://github.com/roburio/icalendar/pull/2).
|
|
||||||
|
|
||||||
Our CalDAV implementation is very basic, scheduling, adding attendees (which requires sending out eMail), is not supported. But it works well for us, we have individual calendars and a shared one which everyone can write to. On the client side we use macOS and iOS iCalendar, Android DAVdroid, and Thunderbird. If you like to try our CalDAV server, have a look [at our installation instructions](https://github.com/roburio/caldav/tree/future/README.md). Please [report issues](https://github.com/roburio/caldav/issues) if you find issues or struggle with the installation.
|
|
||||||
|
|
||||||
## DNS
|
|
||||||
|
|
||||||
There has been more work on our DNS implementation, now [here](https://github.com/mirage/ocaml-dns). We included a DNS client library, and some [example unikernels](https://github.com/roburio/unikernels/tree/future) are available. They as well require our [opam repository overlay](https://github.com/roburio/git-ssh-dns-mirage3-repo). Please report issues if you run into trouble while experimenting with that.
|
|
||||||
|
|
||||||
Most prominently is `primary-git`, a unikernel which acts as a primary authoritative DNS server (UDP and TCP). On startup, it fetches a remote git repository that contains zone files and shared hmac secrets. The zones are served, and secondary servers are notified with the respective serial numbers of the zones, authenticated using TSIG with the shared secrets. The primary server provides dynamic in-protocol updates of DNS resource records (`nsupdate`), and after successful authentication pushes the change to the remote git. To change the zone, you can just edit the zonefile and push to the git remote - with the proper pre- and post-commit-hooks an authenticated notify is send to the primary server which then pulls the git remote.
|
|
||||||
|
|
||||||
Another noteworthy unikernel is `letsencrypt`, which acts as a secondary server, and whenever a TLSA record with custom type (0xFF) and a DER-encoded certificate signing request is observed, it requests a signature from letsencrypt by solving the DNS challenge. The certificate is pushed to the DNS server as TLSA record as well. The DNS implementation provides `ocertify` and `dns-mirage-certify` which use the above mechanism to retrieve valid let's encrypt certificates. The caller (unikernel or Unix command-line utility) either takes a private key directly or generates one from a (provided) seed and generates a certificate signing request. It then looks in DNS for a certificate which is still valid and matches the public key and the hostname. If such a certificate is not present, the certificate signing request is pushed to DNS (via the nsupdate protocol), authenticated using TSIG with a given secret. This way our public facing unikernels (website, this blog, TLS demo server, ..) block until they got a certificate via DNS on startup - we avoid embedding of the certificate into the unikernel image.
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
We like to gather statistics about the resource usage of our unikernels to find potential bottlenecks and observe memory leaks ;) The base for the setup is the [metrics](https://github.com/mirage/metrics) library, which is similarly in design to the [logs](https://erratique.ch/software/logs) library: libraries use the core to gather metrics. A different aspect is the reporter, which is globally registered and responsible for exfiltrating the data via their favourite protocol. If no reporter is registered, the work overhead is negligible.
|
|
||||||
|
|
||||||
[<img src="/static/img/crobur-june-2019-unikernel.png" width="700" />](/static/img/crobur-june-2019-unikernel.png)
|
|
||||||
|
|
||||||
This is a dashboard which combines both statistics gathered from the host system and various metrics from the MirageOS unikernel. The `monitoring` branch of our opam repository overlay is used together with [monitoring-experiments](https://github.com/hannesm/monitoring-experiments). The logs errors counter (middle right) was the icalendar parser which tried to parse its badly emitted ics (the bug is now fixed, the dashboard is from last month).
|
|
||||||
|
|
||||||
|
|
||||||
## OCaml libraries
|
|
||||||
|
|
||||||
The [domain-name](https://github.com/hannesm/domain-name) library was developed to handle RFC 1035 domain names and host names. It initially was part of the DNS code, but is now freestanding to be used in other core libraries (such as ipaddr) with a small dependency footprint.
|
|
||||||
|
|
||||||
The [GADT map](https://github.com/hannesm/gmap) is a normal OCaml Map structure, but takes key-dependent value types by using a GADT. This library also was part of DNS, but is more broadly useful, we already use it in our icalendar (the data format for calendar entries in CalDAV) library, our [OpenVPN](https://git.robur.io/?p=openvpn.git;a=summary) configuration parser uses it as well, and also [x509](https://github.com/mirleft/ocaml-x509/pull/115) - which got reworked quite a bit recently (release pending), and there's preliminary PKCS12 support (which deserves its own article). [TLS 1.3](https://github.com/hannesm/ocaml-tls) is available on a branch, but is not yet merged. More work is underway, hopefully with sufficient time to write more articles about it.
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
More projects are happening as we speak, it takes time to upstream all the changes, such as monitoring, new core libraries, getting our DNS implementation released, pushing Conex into production, more features such as DNSSec, ...
|
|
||||||
|
|
||||||
I'm interested in feedback, either via <strike>[twitter](https://twitter.com/h4nnes)</strike> [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
248
Posts/Syslog
248
Posts/Syslog
|
|
@ -1,192 +1,152 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Exfiltrating log data using syslog
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Exfiltrating log data using syslog</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Exfiltrating log data using syslog" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a><a href="/tags/logging" class="tag">logging</a></div><span class="date">Published: 2016-11-05 (last updated: 2021-11-19)</span><article><p>It has been a while since my last entry... I've been busy working on too many
|
||||||
author: hannes
|
|
||||||
tags: mirageos, protocol, logging
|
|
||||||
abstract: sometimes preservation of data is useful
|
|
||||||
---
|
|
||||||
|
|
||||||
It has been a while since my last entry... I've been busy working on too many
|
|
||||||
projects in parallel, and was also travelling on several continents. I hope to
|
projects in parallel, and was also travelling on several continents. I hope to
|
||||||
get back to a biweekly cycle.
|
get back to a biweekly cycle.</p>
|
||||||
|
<h2 id="what-is-syslog">What is syslog?</h2>
|
||||||
## What is syslog?
|
<p>According to <a href="https://en.wikipedia.org/wiki/Syslog">Wikipedia</a>, syslog is a
|
||||||
|
|
||||||
According to [Wikipedia](https://en.wikipedia.org/wiki/Syslog), syslog is a
|
|
||||||
standard for message logging. Syslog permits separation of the software which
|
standard for message logging. Syslog permits separation of the software which
|
||||||
generates, stores, reports, and analyses the message. A syslog message contains
|
generates, stores, reports, and analyses the message. A syslog message contains
|
||||||
at least a timestamp, a facility, and a severity. It was initially specified in
|
at least a timestamp, a facility, and a severity. It was initially specified in
|
||||||
[RFC 3164](https://tools.ietf.org/html/rfc3164), though usage predates this RFC.
|
<a href="https://tools.ietf.org/html/rfc3164">RFC 3164</a>, though usage predates this RFC.</p>
|
||||||
|
<p>For a unikernel, which likely won't have any persistent storage, syslog is a way
|
||||||
For a unikernel, which likely won't have any persistent storage, syslog is a way
|
|
||||||
to emit log messages (HTTP access log, debug messages, ...) via the network, and
|
to emit log messages (HTTP access log, debug messages, ...) via the network, and
|
||||||
defer the persistency problem to some other service.
|
defer the persistency problem to some other service.</p>
|
||||||
|
<p>Lots of programming languages have logger libraries, which reflect the different
|
||||||
Lots of programming languages have logger libraries, which reflect the different
|
|
||||||
severity of syslog roughly as log levels (debug, informational, warning, error,
|
severity of syslog roughly as log levels (debug, informational, warning, error,
|
||||||
fatal). So does OCaml since the beginning of 2016, there is the
|
fatal). So does OCaml since the beginning of 2016, there is the
|
||||||
[Logs](http://erratique.ch/software/logs) library which separates log message
|
<a href="http://erratique.ch/software/logs">Logs</a> library which separates log message
|
||||||
generation from reporting: the closure producing the log string is only
|
generation from reporting: the closure producing the log string is only
|
||||||
evaluated if there is a reporter which needs to send it out. Additionally, the
|
evaluated if there is a reporter which needs to send it out. Additionally, the
|
||||||
reporter can extend the message with the log source name, a timestamp, etc.
|
reporter can extend the message with the log source name, a timestamp, etc.</p>
|
||||||
|
<p>The Logs library is slowly getting adopted by the MirageOS community (you can
|
||||||
The Logs library is slowly getting adopted by the MirageOS community (you can
|
|
||||||
see an incomplete list
|
see an incomplete list
|
||||||
[here](https://opam.ocaml.org/packages/logs/logs.0.6.2/)), there are reporters
|
<a href="https://opam.ocaml.org/packages/logs/logs.0.6.2/">here</a>), there are reporters
|
||||||
available which integrate into [Apple System
|
available which integrate into <a href="https://github.com/mirage/ocaml-asl">Apple System
|
||||||
Log](https://github.com/mirage/ocaml-asl), [Windows event
|
Log</a>, <a href="https://github.com/djs55/ocaml-win-eventlog">Windows event
|
||||||
log](https://github.com/djs55/ocaml-win-eventlog), and also for [MirageOS
|
log</a>, and also for <a href="https://github.com/mirage/mirage-logs">MirageOS
|
||||||
console](https://github.com/mirage/mirage-logs). There is a command-line
|
console</a>. There is a command-line
|
||||||
argument interface to set the log levels of your individual sources, which is
|
argument interface to set the log levels of your individual sources, which is
|
||||||
pretty neat. For debugging and running on Unix, console output is usually
|
pretty neat. For debugging and running on Unix, console output is usually
|
||||||
sufficient, but for production usage having a console in some `screen` or `tmux`
|
sufficient, but for production usage having a console in some <code>screen</code> or <code>tmux</code>
|
||||||
or dumped to a file is usually annoying.
|
or dumped to a file is usually annoying.</p>
|
||||||
|
<p>Gladly there was already the
|
||||||
Gladly there was already the
|
<a href="https://github.com/verbosemode/syslog-message">syslog-message</a> library, which
|
||||||
[syslog-message](https://github.com/verbosemode/syslog-message) library, which
|
|
||||||
encodes and decodes syslog messages from the wire format to a typed
|
encodes and decodes syslog messages from the wire format to a typed
|
||||||
representation. I plugged those together and [implemented a
|
representation. I plugged those together and <a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog.html">implemented a
|
||||||
reporter](https://hannesm.github.io/logs-syslog/doc/Logs_syslog.html). The
|
reporter</a>. The
|
||||||
[simplest
|
<a href="https://github.com/hannesm/logs-syslog/blob/e35ffe704e998d9a6867f3f504c103861a4408ef/src/logs_syslog_unix.ml#L4-L32">simplest
|
||||||
one](https://github.com/hannesm/logs-syslog/blob/e35ffe704e998d9a6867f3f504c103861a4408ef/src/logs_syslog_unix.ml#L4-L32)
|
one</a>
|
||||||
emits each log message via UDP to a log collector. All reporters contain a
|
emits each log message via UDP to a log collector. All reporters contain a
|
||||||
socket and handle socket errors themselves (trying to recover) - your
|
socket and handle socket errors themselves (trying to recover) - your
|
||||||
application (or unikernel) shouldn't fail just because the log collector is
|
application (or unikernel) shouldn't fail just because the log collector is
|
||||||
currently offline.
|
currently offline.</p>
|
||||||
|
<p>The setup for Unix is straightforward:</p>
|
||||||
The setup for Unix is straightforward:
|
<pre><code class="language-OCaml">Logs.set_reporter (udp_reporter (Unix.inet_addr_of_string "127.0.0.1") ())
|
||||||
|
</code></pre>
|
||||||
```OCaml
|
<p>It will report all log messages (you have to set the Log level yourself,
|
||||||
Logs.set_reporter (udp_reporter (Unix.inet_addr_of_string "127.0.0.1") ())
|
|
||||||
```
|
|
||||||
|
|
||||||
It will report all log messages (you have to set the Log level yourself,
|
|
||||||
defaults to warning) to your local syslog. You might have already listening a
|
defaults to warning) to your local syslog. You might have already listening a
|
||||||
collector on your host, look in `netstat -an` for UDP port 514 (and in your
|
collector on your host, look in <code>netstat -an</code> for UDP port 514 (and in your
|
||||||
`/etc/syslog.conf` to see where log messages are routed to).
|
<code>/etc/syslog.conf</code> to see where log messages are routed to).</p>
|
||||||
|
<p>You can even do this from the OCaml toplevel (after <code>opam install logs-syslog</code>):</p>
|
||||||
You can even do this from the OCaml toplevel (after `opam install logs-syslog`):
|
<pre><code class="language-OCaml">$ utop
|
||||||
```OCaml
|
# #require "logs-syslog.unix";;
|
||||||
$ utop
|
# Logs.set_reporter (Logs_syslog_unix.udp_reporter (Unix.inet_addr_of_string "127.0.0.1") ());;
|
||||||
# #require "logs-syslog.unix";;
|
# Logs.app (fun m -> m "hello, syslog world");;
|
||||||
# Logs.set_reporter (Logs_syslog_unix.udp_reporter (Unix.inet_addr_of_string "127.0.0.1") ());;
|
</code></pre>
|
||||||
# Logs.app (fun m -> m "hello, syslog world");;
|
<p>I configured my syslog to have all <code>informational</code> messages routed to
|
||||||
```
|
<code>/var/log/info.log</code>, you can also try <code>Logs.err (fun m -> m "err");;</code> and look
|
||||||
|
into your <code>/var/log/messages</code>.</p>
|
||||||
I configured my syslog to have all `informational` messages routed to
|
<p>This is a good first step, but we want more: on the one side integration into
|
||||||
`/var/log/info.log`, you can also try `Logs.err (fun m -> m "err");;` and look
|
|
||||||
into your `/var/log/messages`.
|
|
||||||
|
|
||||||
This is a good first step, but we want more: on the one side integration into
|
|
||||||
MirageOS, and a more reliable log stream (what about authentication and
|
MirageOS, and a more reliable log stream (what about authentication and
|
||||||
encryption?). I'll cover both topics in the rest of this article.
|
encryption?). I'll cover both topics in the rest of this article.</p>
|
||||||
|
<h3 id="mirageos-integration">MirageOS integration</h3>
|
||||||
### MirageOS integration
|
<p>Since Mirage3, syslog is integrated (see
|
||||||
|
<a href="http://docs.mirage.io/mirage/Mirage/index.html#type-syslog_config">documentation</a>).
|
||||||
Since Mirage3, syslog is integrated (see
|
Some additions to your <code>config.ml</code> are needed, see <a href="https://github.com/hannesm/ns.nqsb.io/blob/master/config.ml">ns
|
||||||
[documentation](http://docs.mirage.io/mirage/Mirage/index.html#type-syslog_config)).
|
example</a> or
|
||||||
Some additions to your `config.ml` are needed, see [ns
|
<a href="https://github.com/mirage/marrakech2017/blob/master/config.ml">marrakech
|
||||||
example](https://github.com/hannesm/ns.nqsb.io/blob/master/config.ml) or
|
example</a>.</p>
|
||||||
[marrakech
|
<pre><code class="language-OCaml">let logger =
|
||||||
example](https://github.com/mirage/marrakech2017/blob/master/config.ml).
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let logger =
|
|
||||||
syslog_udp (* or _tcp or _tls *)
|
syslog_udp (* or _tcp or _tls *)
|
||||||
(syslog_config ~truncate:1484 "my_first_unikernel"
|
(syslog_config ~truncate:1484 "my_first_unikernel"
|
||||||
(Ipaddr.V4.of_string_exn "10.0.0.1")) (* your log host *)
|
(Ipaddr.V4.of_string_exn "10.0.0.1")) (* your log host *)
|
||||||
stack
|
stack
|
||||||
|
|
||||||
let () =
|
let () =
|
||||||
register "my_first_unikernel" [
|
register "my_first_unikernel" [
|
||||||
foreign ~deps:[abstract logger]
|
foreign ~deps:[abstract logger]
|
||||||
...
|
...
|
||||||
```
|
</code></pre>
|
||||||
|
<h3 id="reliable-syslog">Reliable syslog</h3>
|
||||||
### Reliable syslog
|
<p>The old BSD syslog RFC is obsoleted by <a href="https://tools.ietf.org/html/rfc5424">RFC
|
||||||
|
5424</a>, which describes a new wire format,
|
||||||
The old BSD syslog RFC is obsoleted by [RFC
|
and also a transport over TCP, and <a href="https://tools.ietf.org/html/rfc5425">TLS</a> in
|
||||||
5424](https://tools.ietf.org/html/rfc5424), which describes a new wire format,
|
a subsequent RFC. Unfortunately the <code>syslog-message</code> library does not yet
|
||||||
and also a transport over TCP, and [TLS](https://tools.ietf.org/html/rfc5425) in
|
|
||||||
a subsequent RFC. Unfortunately the `syslog-message` library does not yet
|
|
||||||
support the new format (which supports user-defined structured data (key/value
|
support the new format (which supports user-defined structured data (key/value
|
||||||
fields), and unicode encoding), but I'm sure one day it will.
|
fields), and unicode encoding), but I'm sure one day it will.</p>
|
||||||
|
<p>Another competing syslog <a href="https://tools.ietf.org/html/rfc3195">RFC 3195</a> uses
|
||||||
Another competing syslog [RFC 3195](https://tools.ietf.org/html/rfc3195) uses
|
XML encoding, but I have not bothered to look deeper into that one.</p>
|
||||||
XML encoding, but I have not bothered to look deeper into that one.
|
<p>I implemented both the transport via TCP and via TLS. There are various
|
||||||
|
solutions used for framing (as described in <a href="https://tools.ietf.org/html/rfc6587">RFC
|
||||||
I implemented both the transport via TCP and via TLS. There are various
|
6587</a>): either prepend a decimal encoded
|
||||||
solutions used for framing (as described in [RFC
|
|
||||||
6587](https://tools.ietf.org/html/rfc6587)): either prepend a decimal encoded
|
|
||||||
length (also specified in RFC6524, but obviously violates streaming
|
length (also specified in RFC6524, but obviously violates streaming
|
||||||
characteristics: the log source needs to have the full message in memory before
|
characteristics: the log source needs to have the full message in memory before
|
||||||
sending it out), or have a special delimiter between messages (0 byte, line
|
sending it out), or have a special delimiter between messages (0 byte, line
|
||||||
feed, CR LN, a custom byte sequence).
|
feed, CR LN, a custom byte sequence).</p>
|
||||||
|
<p>The <a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_lwt_tls.html">TLS
|
||||||
The [TLS
|
reporter</a>
|
||||||
reporter](https://hannesm.github.io/logs-syslog/doc/Logs_syslog_lwt_tls.html)
|
|
||||||
uses our TLS library written entirely in OCaml, and requires mutual
|
uses our TLS library written entirely in OCaml, and requires mutual
|
||||||
authentication, both the log reporter has a private key and certificate, and the
|
authentication, both the log reporter has a private key and certificate, and the
|
||||||
log collector needs to present a certificate chain rooted in a provided CA
|
log collector needs to present a certificate chain rooted in a provided CA
|
||||||
certificate.
|
certificate.</p>
|
||||||
|
<p>Logs supports synchronous and asynchronous logging (where the latter is the
|
||||||
Logs supports synchronous and asynchronous logging (where the latter is the
|
default, please read the <a href="http://erratique.ch/software/logs/doc/Logs.html#sync">note on synchronous
|
||||||
default, please read the [note on synchronous
|
logging</a>). In logs-syslog
|
||||||
logging](http://erratique.ch/software/logs/doc/Logs.html#sync)). In logs-syslog
|
|
||||||
this behaviour is not altered. There is no buffer or queue and single writer
|
this behaviour is not altered. There is no buffer or queue and single writer
|
||||||
task to emit log messages, but a mutex and error recovery which tries to
|
task to emit log messages, but a mutex and error recovery which tries to
|
||||||
reconnect once for each log message (of course only if there is not already a
|
reconnect once for each log message (of course only if there is not already a
|
||||||
working connection). It is still not clear to me what the desired behaviour
|
working connection). It is still not clear to me what the desired behaviour
|
||||||
should be, but when introducing buffers I'd loose the synchronous logging (or
|
should be, but when introducing buffers I'd loose the synchronous logging (or
|
||||||
will have to write rather intricate code).
|
will have to write rather intricate code).</p>
|
||||||
|
<p>To rewrap, <code>logs-syslog</code> implements the old BSD syslog protocol via UDP, TCP,
|
||||||
To rewrap, `logs-syslog` implements the old BSD syslog protocol via UDP, TCP,
|
|
||||||
and TLS. There are reporters available using only the Caml
|
and TLS. There are reporters available using only the Caml
|
||||||
[Unix](https://hannesm.github.io/logs-syslog/doc/Logs_syslog_unix.html) module
|
<a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_unix.html">Unix</a> module
|
||||||
(dependency-free!), using
|
(dependency-free!), using
|
||||||
[Lwt](https://hannesm.github.io/logs-syslog/doc/Logs_syslog_lwt.html) (also
|
<a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_lwt.html">Lwt</a> (also
|
||||||
[lwt-tls](https://hannesm.github.io/logs-syslog/doc/Logs_syslog_lwt_tls.html),
|
<a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_lwt_tls.html">lwt-tls</a>,
|
||||||
and using [MirageOS
|
and using <a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_mirage.html">MirageOS
|
||||||
interface](https://hannesm.github.io/logs-syslog/doc/Logs_syslog_mirage.html)
|
interface</a>
|
||||||
(also
|
(also
|
||||||
[TLS](https://hannesm.github.io/logs-syslog/doc/Logs_syslog_mirage_tls.html)).
|
<a href="https://hannesm.github.io/logs-syslog/doc/Logs_syslog_mirage_tls.html">TLS</a>).
|
||||||
The code size is below 500 lines in total.
|
The code size is below 500 lines in total.</p>
|
||||||
|
<h3 id="mirageos-syslog-in-production">MirageOS syslog in production</h3>
|
||||||
### MirageOS syslog in production
|
<p>As collector I use syslog-ng, which is capable of receiving both the new and the
|
||||||
|
|
||||||
As collector I use syslog-ng, which is capable of receiving both the new and the
|
|
||||||
old syslog messages on all three transports. The configuration snippet for a
|
old syslog messages on all three transports. The configuration snippet for a
|
||||||
BSD syslog TLS collector is as following:
|
BSD syslog TLS collector is as following:</p>
|
||||||
|
<pre><code>source s_tls {
|
||||||
```
|
|
||||||
source s_tls {
|
|
||||||
tcp(port(6514)
|
tcp(port(6514)
|
||||||
tls(peer-verify(require-trusted)
|
tls(peer-verify(require-trusted)
|
||||||
cert-file("/etc/log/server.pem")
|
cert-file("/etc/log/server.pem")
|
||||||
key-file("/etc/log/server.key")
|
key-file("/etc/log/server.key")
|
||||||
ca-dir("/etc/log/certs"))); };
|
ca-dir("/etc/log/certs"))); };
|
||||||
|
|
||||||
destination d_tls { file("/var/log/ng-tls.log"); };
|
destination d_tls { file("/var/log/ng-tls.log"); };
|
||||||
|
|
||||||
log { source(s_tls); destination(d_tls); };
|
log { source(s_tls); destination(d_tls); };
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The <code>"/etc/log/certs"</code> directory contains the CA certificates, together with
|
||||||
The `"/etc/log/certs"` directory contains the CA certificates, together with
|
links to their hashes (with a 0 appended: <code>ln -s cacert.pem `openssl x509 -noout -hash -in cacert.pem`.0</code>). I used
|
||||||
links to their hashes (with a 0 appended: ``ln -s cacert.pem `openssl x509
|
<a href="https://github.com/yomimono/ocaml-certify">certify</a> to generate the CA
|
||||||
-noout -hash -in cacert.pem`.0``). I used
|
|
||||||
[certify](https://github.com/yomimono/ocaml-certify) to generate the CA
|
|
||||||
infrastructure (CA cert, a server certificate for syslog-ng, and a client
|
infrastructure (CA cert, a server certificate for syslog-ng, and a client
|
||||||
certificate for my MirageOS unikernel).
|
certificate for my MirageOS unikernel).</p>
|
||||||
|
<p>It is running since a week like a
|
||||||
It is running since a week like a
|
|
||||||
charm (already collected 700KB of HTTP access log), and feels much better than
|
charm (already collected 700KB of HTTP access log), and feels much better than
|
||||||
previous ad-hoc solutions to exfiltrate log data.
|
previous ad-hoc solutions to exfiltrate log data.</p>
|
||||||
|
<p>The downside of syslog is obviously that it only works when the network is up --
|
||||||
The downside of syslog is obviously that it only works when the network is up --
|
|
||||||
thus it does not work while booting, or when a persistent network failure
|
thus it does not work while booting, or when a persistent network failure
|
||||||
occured.
|
occured.</p>
|
||||||
|
<p><a href="https://github.com/hannesm/logs-syslog">Code is on GitHub</a>, <a href="https://hannesm.github.io/logs-syslog/doc">documentation is
|
||||||
[Code is on GitHub](https://github.com/hannesm/logs-syslog), [documentation is
|
online</a>, released in opam.</p>
|
||||||
online](https://hannesm.github.io/logs-syslog/doc), released in opam.
|
<p>I'm interested in feedback, either via
|
||||||
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
I'm interested in feedback, either via
|
</article></div></div></main></body></html>
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
|
||||||
103
Posts/TCP-ns
103
Posts/TCP-ns
|
|
@ -1,65 +1,38 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Redeveloping TCP from the ground up
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Redeveloping TCP from the ground up</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Redeveloping TCP from the ground up" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Redeveloping TCP from the ground up</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a><a href="/tags/tcp" class="tag">tcp</a></div><span class="date">Published: 2023-11-28 (last updated: 2023-11-29)</span><article><p>The <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">Transmission Control Protocol (TCP)</a> is one of the main Internet protocols. Usually spoken on top of the Internet Protocol (legacy version 4 or version 6), it provides a reliable, ordered, and error-checked stream of octets. When an application uses TCP, they get these properties for free (in contrast to UDP).</p>
|
||||||
author: hannes
|
<p>As common for Internet protocols, TCP is specified in a series of so-called requests for comments (RFC). The latest revised version from August 2022 is <a href="https://datatracker.ietf.org/doc/html/rfc9293">RFC 9293</a>; the initial one was <a href="https://datatracker.ietf.org/doc/html/rfc793">RFC 793</a> from September 1981.</p>
|
||||||
tags: mirageos, protocol, tcp
|
<h1 id="my-brief-personal-tcp-story">My brief personal TCP story</h1>
|
||||||
abstract: Core Internet protocols require operational experiments, even if formally specified
|
<p>My interest in TCP started back in 2006 when we worked on a <a href="https://github.com/dylan-hackers/network-night-vision">network stack in Dylan</a> (these days abandoned). Ever since, I wanted to understand the implementation tradeoffs in more detail, including attacks and how to prevent a TCP stack from being vulnerable.</p>
|
||||||
---
|
<p>In 2012, I attended ICFP in Copenhagen while a PhD student at ITU Copenhagen. There, <a href="https://www.cl.cam.ac.uk/~pes20/">Peter Sewell</a> gave an invited talk "Tales from the jungle" about rigorous methods for real-world infrastructure (C semantics, hardware (concurrency) behaviour of CPUs, TCP/IP, and likely more). Working on formal specifications myself in (<a href="https://en.itu.dk/-/media/EN/Research/PhD-Programme/PhD-defences/2013/130731-Hannes-Mehnert-PhD-dissertation-finalpdf.pdf">my dissertation</a>), and having a strong interest in real systems, I was immediately hooked by his perspective.</p>
|
||||||
|
<p>To dive a bit more into <a href="https://www.cl.cam.ac.uk/~pes20/Netsem/">network semantics</a>, the work done on TCP by Peter Sewell, et al., is a formal specification (or a model) of TCP/IP and the Unix sockets API developed in HOL4. It is a label transition system with nondeterministic choices, and the model itself is executable. It has been validated with the real world by collecting thousands of traces on Linux, Windows, and FreeBSD, which have been checked by the model for validity. This copes with the different implementations of the English prose of the RFCs. The network semantics research found several issues in existing TCP stacks and reported them upstream to have them fixed (though, there still is some special treatment, e.g., for the "BSD listen bug").</p>
|
||||||
The [Transmission Control Protocol (TCP)](https://en.wikipedia.org/wiki/Transmission_Control_Protocol) is one of the main Internet protocols. Usually spoken on top of the Internet Protocol (legacy version 4 or version 6), it provides a reliable, ordered, and error-checked stream of octets. When an application uses TCP, they get these properties for free (in contrast to UDP).
|
<p>In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the <a href="https://dl.acm.org/doi/10.1145/3243650">journal paper</a> (<a href="http://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf">author's copy</a>) published. At the same time, the <a href="https://mirage.io">MirageOS</a> melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.</p>
|
||||||
|
<p>My intention was to understand TCP better and use the specification as a basis for a TCP stack for MirageOS. The <a href="https://github.com/mirage/mirage-tcpip">existing one</a> (which is still used) has technical debt: a high issue to number of lines ratio. The Lwt monad is ubiquitous, which makes testing and debugging pretty hard, and also utilising multiple cores with OCaml Multicore won't be easy. Plus it has various resource leaks, and there is no active maintainer. But honestly, it works fine on a local network, and with well-behaved traffic. It doesn't work that well on the wild Internet with a variety of broken implementations. Apart from resource leakage, which made me implement things such as restart-on-failure in <a href="https://github.com/robur-coop/albatross">Albatross</a>, there are certain connection states which will never be exited.</p>
|
||||||
As common for Internet protocols, TCP is specified in a series of so-called requests for comments (RFC). The latest revised version from August 2022 is [RFC 9293](https://datatracker.ietf.org/doc/html/rfc9293); the initial one was [RFC 793](https://datatracker.ietf.org/doc/html/rfc793) from September 1981.
|
<h1 id="the-rise-of-µtcp">The rise of <a href="https://github.com/robur-coop/utcp">µTCP</a></h1>
|
||||||
|
<p>Back in Cambridge, I didn't manage to write a TCP stack based on the model, but in 2019, I restarted that work and got µTCP (the formal model manually translated to OCaml) to compile and do TCP session setup and teardown. Since it was a model that uses nondeterminism, this couldn't be translated one-to-one into an executable program, but there are places where decisions have to be made. Due to other projects, I worked only briefly in 2021 and 2022 on µTCP, but finally in the Summer of 2023, I motivated myself to push µTCP into a usable state. So far I've spend 25 days in 2023 on µTCP. Thanks to <a href="https://tarides.com">Tarides</a> for supporting my work.</p>
|
||||||
# My brief personal TCP story
|
<p>Since late August, we have been running some unikernels using µTCP, e.g., the <a href="https://retreat.mirage.io">retreat</a> website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.</p>
|
||||||
|
<h1 id="the-secondary-nameserver-issue">The secondary nameserver issue</h1>
|
||||||
My interest in TCP started back in 2006 when we worked on a [network stack in Dylan](https://github.com/dylan-hackers/network-night-vision) (these days abandoned). Ever since, I wanted to understand the implementation tradeoffs in more detail, including attacks and how to prevent a TCP stack from being vulnerable.
|
<p>One of our secondary nameservers attempts to receive zones (via AXFR using TCP) from another nameserver that is currently not running. Thus it replies to each SYN packet a corresponding RST. Below I graphed the network utilisation (send data/packets is positive y-axis, receive part on the negative) over time (on the x-axis) on the left and memory usage (bytes on y-axis) over time (x-axis) on the right of our nameserver. You can observe that both increases over time, and roughly every 3 hours, the unikernel hits its configured memory limit (64 MB), crashes with <em>out of memory</em>, and is restarted. The graph below is using the <code>mirage-tcpip</code> stack.</p>
|
||||||
|
<p><a href="/static/img/a.ns.mtcp.png"><img src="/static/img/a.ns.mtcp.png" width="750" /></a></p>
|
||||||
In 2012, I attended ICFP in Copenhagen while a PhD student at ITU Copenhagen. There, [Peter Sewell](https://www.cl.cam.ac.uk/~pes20/) gave an invited talk "Tales from the jungle" about rigorous methods for real-world infrastructure (C semantics, hardware (concurrency) behaviour of CPUs, TCP/IP, and likely more). Working on formal specifications myself in ([my dissertation](https://en.itu.dk/-/media/EN/Research/PhD-Programme/PhD-defences/2013/130731-Hannes-Mehnert-PhD-dissertation-finalpdf.pdf)), and having a strong interest in real systems, I was immediately hooked by his perspective.
|
<p>Now, after switching over to µTCP, graphed below, there's much less network utilisation and the memory limit is only reached after 36 hours, which is a great result. Though, still it is not very satisfying that the unikernel leaks memory. On their left side, both graphs contain a few hours of <code>mirage-tcpip</code>, and shortly after 20:00 on Nov 23rd, µTCP got deployed.</p>
|
||||||
|
<p><a href="/static/img/a.ns.mtcp-utcp.png"><img src="/static/img/a.ns.mtcp-utcp.png" width="750" /></a></p>
|
||||||
To dive a bit more into [network semantics](https://www.cl.cam.ac.uk/~pes20/Netsem/), the work done on TCP by Peter Sewell, et al., is a formal specification (or a model) of TCP/IP and the Unix sockets API developed in HOL4. It is a label transition system with nondeterministic choices, and the model itself is executable. It has been validated with the real world by collecting thousands of traces on Linux, Windows, and FreeBSD, which have been checked by the model for validity. This copes with the different implementations of the English prose of the RFCs. The network semantics research found several issues in existing TCP stacks and reported them upstream to have them fixed (though, there still is some special treatment, e.g., for the "BSD listen bug").
|
<p>Investigating the involved parts showed that an unestablished TCP connection has been registered at the MirageOS layer, but the pure core does not expose an event from the received RST that the connection has been cancelled. This means the MirageOS layer piles up all the connection attempts, and it doesn't inform the application that the connection couldn't be established. Note that the MirageOS layer is not code derived from the formal model, but boilerplate for (a) effectful side-effects (IO) and (b) meeting the needs of the <a href="https://github.com/mirage/mirage-tcpip/blob/v8.0.0/src/core/tcp.ml">TCP.S</a> module type (so that µTCP can be used as a drop-in replacement for mirage-tcpip). Once this was well understood, developing the <a href="https://github.com/robur-coop/utcp/commit/67fc49468e6b75b96a481ebe44dd11ce4bb76e6c">required code changes</a> was straightforward. The graph shows that the fix was deployed at 15:25. The memory usage is constant afterwards, but the network utilisation increased enormously.</p>
|
||||||
|
<p><a href="/static/img/a.ns.utcp-ev.png"><img src="/static/img/a.ns.utcp-ev.png" width="750" /></a></p>
|
||||||
In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the [journal paper](https://dl.acm.org/doi/10.1145/3243650) ([author's copy](http://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf)) published. At the same time, the [MirageOS](https://mirage.io) melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.
|
<p>Now, the network utilisation is unwanted. This was hidden by the application waiting forever while the TCP connection getting established. Our bug fix uncovered another issue -- a tight loop:</p>
|
||||||
|
<ul>
|
||||||
My intention was to understand TCP better and use the specification as a basis for a TCP stack for MirageOS. The [existing one](https://github.com/mirage/mirage-tcpip) (which is still used) has technical debt: a high issue to number of lines ratio. The Lwt monad is ubiquitous, which makes testing and debugging pretty hard, and also utilising multiple cores with OCaml Multicore won't be easy. Plus it has various resource leaks, and there is no active maintainer. But honestly, it works fine on a local network, and with well-behaved traffic. It doesn't work that well on the wild Internet with a variety of broken implementations. Apart from resource leakage, which made me implement things such as restart-on-failure in [Albatross](https://github.com/robur-coop/albatross), there are certain connection states which will never be exited.
|
<li>The nameserver attempts to connect to the other nameserver (<code>request</code>);
|
||||||
|
</li>
|
||||||
# The rise of [µTCP](https://github.com/robur-coop/utcp)
|
<li>This results in a <code>TCP.create_connection</code> which errors after one roundtrip;
|
||||||
|
</li>
|
||||||
Back in Cambridge, I didn't manage to write a TCP stack based on the model, but in 2019, I restarted that work and got µTCP (the formal model manually translated to OCaml) to compile and do TCP session setup and teardown. Since it was a model that uses nondeterminism, this couldn't be translated one-to-one into an executable program, but there are places where decisions have to be made. Due to other projects, I worked only briefly in 2021 and 2022 on µTCP, but finally in the Summer of 2023, I motivated myself to push µTCP into a usable state. So far I've spend 25 days in 2023 on µTCP. Thanks to [Tarides](https://tarides.com) for supporting my work.
|
<li>This leads to a <code>close</code>, which attempts a <code>request</code> again.
|
||||||
|
</li>
|
||||||
Since late August, we have been running some unikernels using µTCP, e.g., the [retreat](https://retreat.mirage.io) website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.
|
</ul>
|
||||||
|
<p>This is unnecessary since the DNS server code has a timer to attempt to connect to the remote nameserver periodically (but takes a break between attempts). After understanding this behaviour, we worked on <a href="https://github.com/mirage/ocaml-dns/pull/347">the fix</a> and redeployed the nameserver again. On the left edge, the has the tight loop (so you have a baseline for comparison), and at 16:05, we deployed the fix. Since then it looks pretty smooth, both in memory usage and in network utilisation.</p>
|
||||||
# The secondary nameserver issue
|
<p><a href="/static/img/a.ns.utcp-fixed.png"><img src="/static/img/a.ns.utcp-fixed.png" width="750" /></a></p>
|
||||||
|
<p>To give you the entire picture, below is the graph where you can spot the <code>mirage-tcpip</code> stack (lots of network, restarting every 3 hours), µTCP-without-informing-application (run for 3 * ~36 hours), DNS-server-high-network-utilization (which only lasted for a brief period, thus it is more a point in the graph), and finally the unikernel with both fixes applied.</p>
|
||||||
One of our secondary nameservers attempts to receive zones (via AXFR using TCP) from another nameserver that is currently not running. Thus it replies to each SYN packet a corresponding RST. Below I graphed the network utilisation (send data/packets is positive y-axis, receive part on the negative) over time (on the x-axis) on the left and memory usage (bytes on y-axis) over time (x-axis) on the right of our nameserver. You can observe that both increases over time, and roughly every 3 hours, the unikernel hits its configured memory limit (64 MB), crashes with *out of memory*, and is restarted. The graph below is using the `mirage-tcpip` stack.
|
<p><a href="/static/img/a.ns.all.png"><img src="/static/img/a.ns.all.png" width="750" /></a></p>
|
||||||
|
<h1 id="conclusion">Conclusion</h1>
|
||||||
[<img src="/static/img/a.ns.mtcp.png" width="750" />](/static/img/a.ns.mtcp.png)
|
<p>What can we learn from that? Choosing convenient tooling is crucial for effective debugging. Also, fixing one issue may uncover other issues. And of course, the <code>mirage-tcpip</code> was running with the DNS-server that had a tight reconnect loop. But, below the line: should such an application lead to memory leaks? I don't think so. My approach is that all core network libraries should work in a non-resource-leaky way with any kind of application on top of it. When one TCP connection returns an error (and thus is destroyed), the TCP stack should have no more resources used for that connection.</p>
|
||||||
|
<p>We'll take more time to investigate issues of µTCP in production, plan to write further documentation and blog posts, and hopefully soon will be ready for an initial public release. In the meantime, you can follow our development repository.</p>
|
||||||
Now, after switching over to µTCP, graphed below, there's much less network utilisation and the memory limit is only reached after 36 hours, which is a great result. Though, still it is not very satisfying that the unikernel leaks memory. On their left side, both graphs contain a few hours of `mirage-tcpip`, and shortly after 20:00 on Nov 23rd, µTCP got deployed.
|
<p>We at <a href="https://robur.coop">Robur</a> are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by <a href="https://aenderwerk.de/donate/">donating</a> (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).</p>
|
||||||
|
<p>If you have any questions, reach us best via eMail to team AT robur DOT coop.</p>
|
||||||
[<img src="/static/img/a.ns.mtcp-utcp.png" width="750" />](/static/img/a.ns.mtcp-utcp.png)
|
</article></div></div></main></body></html>
|
||||||
|
|
||||||
Investigating the involved parts showed that an unestablished TCP connection has been registered at the MirageOS layer, but the pure core does not expose an event from the received RST that the connection has been cancelled. This means the MirageOS layer piles up all the connection attempts, and it doesn't inform the application that the connection couldn't be established. Note that the MirageOS layer is not code derived from the formal model, but boilerplate for (a) effectful side-effects (IO) and (b) meeting the needs of the [TCP.S](https://github.com/mirage/mirage-tcpip/blob/v8.0.0/src/core/tcp.ml) module type (so that µTCP can be used as a drop-in replacement for mirage-tcpip). Once this was well understood, developing the [required code changes](https://github.com/robur-coop/utcp/commit/67fc49468e6b75b96a481ebe44dd11ce4bb76e6c) was straightforward. The graph shows that the fix was deployed at 15:25. The memory usage is constant afterwards, but the network utilisation increased enormously.
|
|
||||||
|
|
||||||
[<img src="/static/img/a.ns.utcp-ev.png" width="750" />](/static/img/a.ns.utcp-ev.png)
|
|
||||||
|
|
||||||
Now, the network utilisation is unwanted. This was hidden by the application waiting forever while the TCP connection getting established. Our bug fix uncovered another issue -- a tight loop:
|
|
||||||
- The nameserver attempts to connect to the other nameserver (`request`);
|
|
||||||
- This results in a `TCP.create_connection` which errors after one roundtrip;
|
|
||||||
- This leads to a `close`, which attempts a `request` again.
|
|
||||||
|
|
||||||
This is unnecessary since the DNS server code has a timer to attempt to connect to the remote nameserver periodically (but takes a break between attempts). After understanding this behaviour, we worked on [the fix](https://github.com/mirage/ocaml-dns/pull/347) and redeployed the nameserver again. On the left edge, the has the tight loop (so you have a baseline for comparison), and at 16:05, we deployed the fix. Since then it looks pretty smooth, both in memory usage and in network utilisation.
|
|
||||||
|
|
||||||
[<img src="/static/img/a.ns.utcp-fixed.png" width="750" />](/static/img/a.ns.utcp-fixed.png)
|
|
||||||
|
|
||||||
To give you the entire picture, below is the graph where you can spot the `mirage-tcpip` stack (lots of network, restarting every 3 hours), µTCP-without-informing-application (run for 3 * ~36 hours), DNS-server-high-network-utilization (which only lasted for a brief period, thus it is more a point in the graph), and finally the unikernel with both fixes applied.
|
|
||||||
|
|
||||||
[<img src="/static/img/a.ns.all.png" width="750" />](/static/img/a.ns.all.png)
|
|
||||||
|
|
||||||
# Conclusion
|
|
||||||
|
|
||||||
What can we learn from that? Choosing convenient tooling is crucial for effective debugging. Also, fixing one issue may uncover other issues. And of course, the `mirage-tcpip` was running with the DNS-server that had a tight reconnect loop. But, below the line: should such an application lead to memory leaks? I don't think so. My approach is that all core network libraries should work in a non-resource-leaky way with any kind of application on top of it. When one TCP connection returns an error (and thus is destroyed), the TCP stack should have no more resources used for that connection.
|
|
||||||
|
|
||||||
We'll take more time to investigate issues of µTCP in production, plan to write further documentation and blog posts, and hopefully soon will be ready for an initial public release. In the meantime, you can follow our development repository.
|
|
||||||
|
|
||||||
We at [Robur](https://robur.coop) are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by [donating](https://aenderwerk.de/donate/) (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).
|
|
||||||
|
|
||||||
If you have any questions, reach us best via eMail to team AT robur DOT coop.
|
|
||||||
259
Posts/Traceroute
259
Posts/Traceroute
|
|
@ -1,17 +1,8 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Traceroute
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Traceroute</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Traceroute" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Traceroute</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2020-06-24 (last updated: 2021-11-19)</span><article><h2 id="traceroute">Traceroute</h2>
|
||||||
author: hannes
|
<p>Is a diagnostic utility which displays the route and measures transit delays of
|
||||||
tags: mirageos, protocol
|
packets across an Internet protocol (IP) network.</p>
|
||||||
abstract: A MirageOS unikernel which traces the path between itself and a remote host.
|
<pre><code class="language-bash">$ doas solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --host=198.167.222.207
|
||||||
---
|
|
||||||
|
|
||||||
## Traceroute
|
|
||||||
|
|
||||||
Is a diagnostic utility which displays the route and measures transit delays of
|
|
||||||
packets across an Internet protocol (IP) network.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ doas solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --host=198.167.222.207
|
|
||||||
| ___|
|
| ___|
|
||||||
__| _ \ | _ \ __ \
|
__| _ \ | _ \ __ \
|
||||||
\__ \ ( | | ( | ) |
|
\__ \ ( | | ( | ) |
|
||||||
|
|
@ -22,7 +13,7 @@ Solo5: reserved @ (0x0 - 0xfffff)
|
||||||
Solo5: text @ (0x100000 - 0x212fff)
|
Solo5: text @ (0x100000 - 0x212fff)
|
||||||
Solo5: rodata @ (0x213000 - 0x24bfff)
|
Solo5: rodata @ (0x213000 - 0x24bfff)
|
||||||
Solo5: data @ (0x24c000 - 0x317fff)
|
Solo5: data @ (0x24c000 - 0x317fff)
|
||||||
Solo5: heap >= 0x318000 < stack < 0x20000000
|
Solo5: heap >= 0x318000 < stack < 0x20000000
|
||||||
2020-06-22 15:41:25 -00:00: INF [netif] Plugging into service with mac 76:9b:36:e0:e5:74 mtu 1500
|
2020-06-22 15:41:25 -00:00: INF [netif] Plugging into service with mac 76:9b:36:e0:e5:74 mtu 1500
|
||||||
2020-06-22 15:41:25 -00:00: INF [ethernet] Connected Ethernet interface 76:9b:36:e0:e5:74
|
2020-06-22 15:41:25 -00:00: INF [ethernet] Connected Ethernet interface 76:9b:36:e0:e5:74
|
||||||
2020-06-22 15:41:25 -00:00: INF [ARP] Sending gratuitous ARP for 10.0.42.2 (76:9b:36:e0:e5:74)
|
2020-06-22 15:41:25 -00:00: INF [ARP] Sending gratuitous ARP for 10.0.42.2 (76:9b:36:e0:e5:74)
|
||||||
|
|
@ -45,9 +36,8 @@ Solo5: heap >= 0x318000 < stack < 0x20000000
|
||||||
2020-06-22 15:41:27 -00:00: INF [application] 16 80.67.10.147 47.213ms
|
2020-06-22 15:41:27 -00:00: INF [application] 16 80.67.10.147 47.213ms
|
||||||
2020-06-22 15:41:27 -00:00: INF [application] 17 198.167.222.207 48.598ms
|
2020-06-22 15:41:27 -00:00: INF [application] 17 198.167.222.207 48.598ms
|
||||||
Solo5: solo5_exit(0) called
|
Solo5: solo5_exit(0) called
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This means with a traceroute utility you can investigate which route is taken
|
||||||
This means with a traceroute utility you can investigate which route is taken
|
|
||||||
to a destination host, and what the round trip time(s) on the path are. The
|
to a destination host, and what the round trip time(s) on the path are. The
|
||||||
sample output above is taken from a virtual machine on my laptop to the remote
|
sample output above is taken from a virtual machine on my laptop to the remote
|
||||||
host 198.167.222.207. You can see there are 17 hops between us, with the first
|
host 198.167.222.207. You can see there are 17 hops between us, with the first
|
||||||
|
|
@ -56,17 +46,15 @@ using private IP addresses, and are my home network. The round trip time of the
|
||||||
fourth hop is much higher, this is the first hop on the other side of my DSL
|
fourth hop is much higher, this is the first hop on the other side of my DSL
|
||||||
modem. You can see various hops on the public Internet: the packets pass from
|
modem. You can see various hops on the public Internet: the packets pass from
|
||||||
my Internet provider's backbone across some exchange points to the destination
|
my Internet provider's backbone across some exchange points to the destination
|
||||||
Internet provider somewhere in Sweden.
|
Internet provider somewhere in Sweden.</p>
|
||||||
|
<p>The implementation of traceroute relies mainly on the time-to-live (ttl) field
|
||||||
The implementation of traceroute relies mainly on the time-to-live (ttl) field
|
(in IPv6 lingua it is "hop limit") of IP packets, which is meant to avoid route
|
||||||
(in IPv6 lingua it is "hop limit") of IP packets, which is meant to avoid route
|
|
||||||
cycles that would infinitely forward IP packets in circles. Every router, when
|
cycles that would infinitely forward IP packets in circles. Every router, when
|
||||||
forwarding an IP packet, first checks that the ttl field is greater than zero,
|
forwarding an IP packet, first checks that the ttl field is greater than zero,
|
||||||
and then forwards the IP packet where the ttl is decreased by one. If the ttl
|
and then forwards the IP packet where the ttl is decreased by one. If the ttl
|
||||||
field is zero, instead of forwarding, an ICMP time exceeded packet is sent back
|
field is zero, instead of forwarding, an ICMP time exceeded packet is sent back
|
||||||
to the source.
|
to the source.</p>
|
||||||
|
<p>Traceroute works by exploiting this mechanism: a series of IP packets with
|
||||||
Traceroute works by exploiting this mechanism: a series of IP packets with
|
|
||||||
increasing ttls is sent to the destination. Since upfront the length of the
|
increasing ttls is sent to the destination. Since upfront the length of the
|
||||||
path is unknown, it is a reactive system: first send an IP packet with a ttl of
|
path is unknown, it is a reactive system: first send an IP packet with a ttl of
|
||||||
one, if a ICMP time exceeded packet is returned, send an IP packet with a ttl of
|
one, if a ICMP time exceeded packet is returned, send an IP packet with a ttl of
|
||||||
|
|
@ -74,18 +62,16 @@ two, etc. -- until an ICMP packet of type destination unreachable is received.
|
||||||
Since some hosts do not reply with a time exceeded message, it is crucial for
|
Since some hosts do not reply with a time exceeded message, it is crucial for
|
||||||
not getting stuck to use a timeout for each packet: when the timeout is reached,
|
not getting stuck to use a timeout for each packet: when the timeout is reached,
|
||||||
an IP packet with an increased ttl is sent and an unknown for the ttl is
|
an IP packet with an increased ttl is sent and an unknown for the ttl is
|
||||||
printed (see the fifth hop in the example above).
|
printed (see the fifth hop in the example above).</p>
|
||||||
|
<p>The packets send out are conventionally UDP packets without payload. From a
|
||||||
The packets send out are conventionally UDP packets without payload. From a
|
|
||||||
development perspective, one question is how to correlate the ICMP packet
|
development perspective, one question is how to correlate the ICMP packet
|
||||||
with the sent UDP packet. Conveniently, ICMP packets contain the IP header and
|
with the sent UDP packet. Conveniently, ICMP packets contain the IP header and
|
||||||
the first eight bytes of the next protocol - the UDP header containing source
|
the first eight bytes of the next protocol - the UDP header containing source
|
||||||
port, destination port, checksum, and payload length (each fields of size two
|
port, destination port, checksum, and payload length (each fields of size two
|
||||||
bytes). This means when we record the outgoing ports together with the sent
|
bytes). This means when we record the outgoing ports together with the sent
|
||||||
timestamp, and correlate the later received ICMP packet to the sent packet.
|
timestamp, and correlate the later received ICMP packet to the sent packet.
|
||||||
Great.
|
Great.</p>
|
||||||
|
<p>But as a functional programmer, let's figure whether we can abolish the
|
||||||
But as a functional programmer, let's figure whether we can abolish the
|
|
||||||
(globally shared) state. Since the ICMP packet contains the original IP
|
(globally shared) state. Since the ICMP packet contains the original IP
|
||||||
header and the first eight bytes of the UDP header, this is where we will
|
header and the first eight bytes of the UDP header, this is where we will
|
||||||
embed data. As described above, the data is the sent timestamp and the value
|
embed data. As described above, the data is the sent timestamp and the value
|
||||||
|
|
@ -95,13 +81,10 @@ round trip time. Taking the source and destination port are 32 bits, using 5 for
|
||||||
ttl, remaining are 27 bits (an unsigned value up to 134217727). Looking at the
|
ttl, remaining are 27 bits (an unsigned value up to 134217727). Looking at the
|
||||||
decimal representation, 1 second is likely too small, 13 seconds are sufficient
|
decimal representation, 1 second is likely too small, 13 seconds are sufficient
|
||||||
for the round trip time measurement. This implies our precision is 100ns, by
|
for the round trip time measurement. This implies our precision is 100ns, by
|
||||||
counting the digits.
|
counting the digits.</p>
|
||||||
|
<p>Finally to the code. First we need forth and back conversions between ports
|
||||||
Finally to the code. First we need forth and back conversions between ports
|
and ttl, timestamp:</p>
|
||||||
and ttl, timestamp:
|
<pre><code class="language-OCaml">(* takes a time-to-live (int) and timestamp (int64, nanoseconda), encodes them
|
||||||
|
|
||||||
```OCaml
|
|
||||||
(* takes a time-to-live (int) and timestamp (int64, nanoseconda), encodes them
|
|
||||||
into 16 bit source port and 16 bit destination port:
|
into 16 bit source port and 16 bit destination port:
|
||||||
- the timestamp precision is 100ns (thus, it is divided by 100)
|
- the timestamp precision is 100ns (thus, it is divided by 100)
|
||||||
- use the bits 27-11 of the timestamp as source port
|
- use the bits 27-11 of the timestamp as source port
|
||||||
|
|
@ -126,28 +109,21 @@ let ttl_ts_of_ports src_port dst_port =
|
||||||
in
|
in
|
||||||
let ts = Int64.mul ts 100L in
|
let ts = Int64.mul ts 100L in
|
||||||
ttl, ts
|
ttl, ts
|
||||||
```
|
</code></pre>
|
||||||
|
<p>They should be inverse over the range of valid input: ports are 16 bit numbers,
|
||||||
They should be inverse over the range of valid input: ports are 16 bit numbers,
|
ttl expected to be at most 31, ts a int64 expressed in nanoseconds.</p>
|
||||||
ttl expected to be at most 31, ts a int64 expressed in nanoseconds.
|
<p>Related is the function to print out one hop and round trip measurement:</p>
|
||||||
|
<pre><code class="language-OCaml">(* write a log line of a hop: the number, IP address, and round trip time *)
|
||||||
Related is the function to print out one hop and round trip measurement:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
(* write a log line of a hop: the number, IP address, and round trip time *)
|
|
||||||
let log_one now ttl sent ip =
|
let log_one now ttl sent ip =
|
||||||
let now = Int64.(mul (logand (div now 100L) 0x7FFFFFFL) 100L) in
|
let now = Int64.(mul (logand (div now 100L) 0x7FFFFFFL) 100L) in
|
||||||
let duration = Mtime.Span.of_uint64_ns (Int64.sub now sent) in
|
let duration = Mtime.Span.of_uint64_ns (Int64.sub now sent) in
|
||||||
Logs.info (fun m -> m "%2d %a %a" ttl Ipaddr.V4.pp ip Mtime.Span.pp duration)
|
Logs.info (fun m -> m "%2d %a %a" ttl Ipaddr.V4.pp ip Mtime.Span.pp duration)
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The most logic is when a ICMP packet is received:</p>
|
||||||
The most logic is when a ICMP packet is received:
|
<pre><code class="language-OCaml">module Icmp = struct
|
||||||
|
|
||||||
```OCaml
|
|
||||||
module Icmp = struct
|
|
||||||
type t = {
|
type t = {
|
||||||
send : int -> unit Lwt.t ;
|
send : int -> unit Lwt.t ;
|
||||||
log : int -> int64 -> Ipaddr.V4.t -> unit ;
|
log : int -> int64 -> Ipaddr.V4.t -> unit ;
|
||||||
task_done : unit Lwt.u ;
|
task_done : unit Lwt.u ;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -160,21 +136,21 @@ module Icmp = struct
|
||||||
let open Icmpv4_packet in
|
let open Icmpv4_packet in
|
||||||
(* Decode the received buffer (the IP header has been cut off already). *)
|
(* Decode the received buffer (the IP header has been cut off already). *)
|
||||||
match Unmarshal.of_cstruct buf with
|
match Unmarshal.of_cstruct buf with
|
||||||
| Error s ->
|
| Error s ->
|
||||||
Lwt.fail_with (Fmt.strf "ICMP: error parsing message from %a: %s" Ipaddr.V4.pp src s)
|
Lwt.fail_with (Fmt.strf "ICMP: error parsing message from %a: %s" Ipaddr.V4.pp src s)
|
||||||
| Ok (message, payload) ->
|
| Ok (message, payload) ->
|
||||||
let open Icmpv4_wire in
|
let open Icmpv4_wire in
|
||||||
(* There are two interesting cases: Time exceeded (-> send next packet),
|
(* There are two interesting cases: Time exceeded (-> send next packet),
|
||||||
and Destination (port) unreachable (-> we reached the final host and can exit) *)
|
and Destination (port) unreachable (-> we reached the final host and can exit) *)
|
||||||
match message.ty with
|
match message.ty with
|
||||||
| Time_exceeded ->
|
| Time_exceeded ->
|
||||||
(* Decode the payload, which should be an IPv4 header and a protocol header *)
|
(* Decode the payload, which should be an IPv4 header and a protocol header *)
|
||||||
begin match Ipv4_packet.Unmarshal.header_of_cstruct payload with
|
begin match Ipv4_packet.Unmarshal.header_of_cstruct payload with
|
||||||
| Ok (pkt, off) when
|
| Ok (pkt, off) when
|
||||||
(* Ensure this packet matches our sent packet: the protocol is UDP
|
(* Ensure this packet matches our sent packet: the protocol is UDP
|
||||||
and the destination address is the host we're tracing *)
|
and the destination address is the host we're tracing *)
|
||||||
pkt.Ipv4_packet.proto = Ipv4_packet.Marshal.protocol_to_int `UDP &&
|
pkt.Ipv4_packet.proto = Ipv4_packet.Marshal.protocol_to_int `UDP &&
|
||||||
Ipaddr.V4.compare pkt.Ipv4_packet.dst (Key_gen.host ()) = 0 ->
|
Ipaddr.V4.compare pkt.Ipv4_packet.dst (Key_gen.host ()) = 0 ->
|
||||||
let src_port = Cstruct.BE.get_uint16 payload off
|
let src_port = Cstruct.BE.get_uint16 payload off
|
||||||
and dst_port = Cstruct.BE.get_uint16 payload (off + 2)
|
and dst_port = Cstruct.BE.get_uint16 payload (off + 2)
|
||||||
in
|
in
|
||||||
|
|
@ -186,24 +162,24 @@ module Icmp = struct
|
||||||
t.log ttl sent src;
|
t.log ttl sent src;
|
||||||
(* Sent out the next UDP packet with an increased ttl. *)
|
(* Sent out the next UDP packet with an increased ttl. *)
|
||||||
let ttl' = succ ttl in
|
let ttl' = succ ttl in
|
||||||
Logs.debug (fun m -> m "ICMP time exceeded from %a to %a, now sending with ttl %d"
|
Logs.debug (fun m -> m "ICMP time exceeded from %a to %a, now sending with ttl %d"
|
||||||
Ipaddr.V4.pp src Ipaddr.V4.pp dst ttl');
|
Ipaddr.V4.pp src Ipaddr.V4.pp dst ttl');
|
||||||
t.send ttl'
|
t.send ttl'
|
||||||
| Ok (pkt, _) ->
|
| Ok (pkt, _) ->
|
||||||
(* Some stray ICMP packet. *)
|
(* Some stray ICMP packet. *)
|
||||||
Logs.debug (fun m -> m "unsolicited time exceeded from %a to %a (proto %X dst %a)"
|
Logs.debug (fun m -> m "unsolicited time exceeded from %a to %a (proto %X dst %a)"
|
||||||
Ipaddr.V4.pp src Ipaddr.V4.pp dst pkt.Ipv4_packet.proto Ipaddr.V4.pp pkt.Ipv4_packet.dst);
|
Ipaddr.V4.pp src Ipaddr.V4.pp dst pkt.Ipv4_packet.proto Ipaddr.V4.pp pkt.Ipv4_packet.dst);
|
||||||
Lwt.return_unit
|
Lwt.return_unit
|
||||||
| Error e ->
|
| Error e ->
|
||||||
(* Decoding error. *)
|
(* Decoding error. *)
|
||||||
Logs.warn (fun m -> m "couldn't parse ICMP time exceeded payload (IPv4) (%a -> %a) %s"
|
Logs.warn (fun m -> m "couldn't parse ICMP time exceeded payload (IPv4) (%a -> %a) %s"
|
||||||
Ipaddr.V4.pp src Ipaddr.V4.pp dst e);
|
Ipaddr.V4.pp src Ipaddr.V4.pp dst e);
|
||||||
Lwt.return_unit
|
Lwt.return_unit
|
||||||
end
|
end
|
||||||
| Destination_unreachable when Ipaddr.V4.compare src (Key_gen.host ()) = 0 ->
|
| Destination_unreachable when Ipaddr.V4.compare src (Key_gen.host ()) = 0 ->
|
||||||
(* We reached the final host, and the destination port was not listened to *)
|
(* We reached the final host, and the destination port was not listened to *)
|
||||||
begin match Ipv4_packet.Unmarshal.header_of_cstruct payload with
|
begin match Ipv4_packet.Unmarshal.header_of_cstruct payload with
|
||||||
| Ok (_, off) ->
|
| Ok (_, off) ->
|
||||||
let src_port = Cstruct.BE.get_uint16 payload off
|
let src_port = Cstruct.BE.get_uint16 payload off
|
||||||
and dst_port = Cstruct.BE.get_uint16 payload (off + 2)
|
and dst_port = Cstruct.BE.get_uint16 payload (off + 2)
|
||||||
in
|
in
|
||||||
|
|
@ -214,24 +190,21 @@ module Icmp = struct
|
||||||
(* Wakeup the waiter task to exit the unikernel. *)
|
(* Wakeup the waiter task to exit the unikernel. *)
|
||||||
Lwt.wakeup t.task_done ();
|
Lwt.wakeup t.task_done ();
|
||||||
Lwt.return_unit
|
Lwt.return_unit
|
||||||
| Error e ->
|
| Error e ->
|
||||||
(* Decoding error. *)
|
(* Decoding error. *)
|
||||||
Logs.warn (fun m -> m "couldn't parse ICMP unreachable payload (IPv4) (%a -> %a) %s"
|
Logs.warn (fun m -> m "couldn't parse ICMP unreachable payload (IPv4) (%a -> %a) %s"
|
||||||
Ipaddr.V4.pp src Ipaddr.V4.pp dst e);
|
Ipaddr.V4.pp src Ipaddr.V4.pp dst e);
|
||||||
Lwt.return_unit
|
Lwt.return_unit
|
||||||
end
|
end
|
||||||
| ty ->
|
| ty ->
|
||||||
Logs.debug (fun m -> m "ICMP unknown ty %s from %a to %a: %a"
|
Logs.debug (fun m -> m "ICMP unknown ty %s from %a to %a: %a"
|
||||||
(ty_to_string ty) Ipaddr.V4.pp src Ipaddr.V4.pp dst
|
(ty_to_string ty) Ipaddr.V4.pp src Ipaddr.V4.pp dst
|
||||||
Cstruct.hexdump_pp payload);
|
Cstruct.hexdump_pp payload);
|
||||||
Lwt.return_unit
|
Lwt.return_unit
|
||||||
end
|
end
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Now, the remaining main unikernel is the module <code>Main</code>:</p>
|
||||||
Now, the remaining main unikernel is the module `Main`:
|
<pre><code class="language-OCaml">module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.S) (N : Mirage_net.S) = struct
|
||||||
|
|
||||||
```OCaml
|
|
||||||
module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.S) (N : Mirage_net.S) = struct
|
|
||||||
module ETH = Ethernet.Make(N)
|
module ETH = Ethernet.Make(N)
|
||||||
module ARP = Arp.Make(ETH)(Time)
|
module ARP = Arp.Make(ETH)(Time)
|
||||||
module IPV4 = Static_ipv4.Make(R)(M)(ETH)(ARP)
|
module IPV4 = Static_ipv4.Make(R)(M)(ETH)(ARP)
|
||||||
|
|
@ -245,10 +218,10 @@ module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.
|
||||||
(* This is called by the ICMP handler which successfully received a
|
(* This is called by the ICMP handler which successfully received a
|
||||||
time exceeded, thus we cancel the timeout task. *)
|
time exceeded, thus we cancel the timeout task. *)
|
||||||
(match !to_cancel with
|
(match !to_cancel with
|
||||||
| None -> ()
|
| None -> ()
|
||||||
| Some t -> Lwt.cancel t ; to_cancel := None);
|
| Some t -> Lwt.cancel t ; to_cancel := None);
|
||||||
(* Our hop limit is 31 - 5 bit - should be sufficient for most networks. *)
|
(* Our hop limit is 31 - 5 bit - should be sufficient for most networks. *)
|
||||||
if ttl > 31 then
|
if ttl > 31 then
|
||||||
Lwt.return_unit
|
Lwt.return_unit
|
||||||
else
|
else
|
||||||
(* Create a timeout task which:
|
(* Create a timeout task which:
|
||||||
|
|
@ -257,11 +230,11 @@ module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.
|
||||||
- sends another packet with increased ttl
|
- sends another packet with increased ttl
|
||||||
*)
|
*)
|
||||||
let cancel =
|
let cancel =
|
||||||
Lwt.catch (fun () ->
|
Lwt.catch (fun () ->
|
||||||
Time.sleep_ns (Duration.of_ms (Key_gen.timeout ())) >>= fun () ->
|
Time.sleep_ns (Duration.of_ms (Key_gen.timeout ())) >>= fun () ->
|
||||||
Logs.info (fun m -> m "%2d *" ttl);
|
Logs.info (fun m -> m "%2d *" ttl);
|
||||||
send_udp udp (succ ttl))
|
send_udp udp (succ ttl))
|
||||||
(function Lwt.Canceled -> Lwt.return_unit | exc -> Lwt.fail exc)
|
(function Lwt.Canceled -> Lwt.return_unit | exc -> Lwt.fail exc)
|
||||||
in
|
in
|
||||||
(* Assign this timeout task. *)
|
(* Assign this timeout task. *)
|
||||||
to_cancel := Some cancel;
|
to_cancel := Some cancel;
|
||||||
|
|
@ -269,26 +242,26 @@ module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.
|
||||||
and current timestamp. *)
|
and current timestamp. *)
|
||||||
let src_port, dst_port = ports_of_ttl_ts ttl (M.elapsed_ns ()) in
|
let src_port, dst_port = ports_of_ttl_ts ttl (M.elapsed_ns ()) in
|
||||||
(* Send packet via UDP. *)
|
(* Send packet via UDP. *)
|
||||||
UDP.write ~ttl ~src_port ~dst:(Key_gen.host ()) ~dst_port udp Cstruct.empty >>= function
|
UDP.write ~ttl ~src_port ~dst:(Key_gen.host ()) ~dst_port udp Cstruct.empty >>= function
|
||||||
| Ok () -> Lwt.return_unit
|
| Ok () -> Lwt.return_unit
|
||||||
| Error e -> Lwt.fail_with (Fmt.strf "while sending udp frame %a" UDP.pp_error e)
|
| Error e -> Lwt.fail_with (Fmt.strf "while sending udp frame %a" UDP.pp_error e)
|
||||||
|
|
||||||
(* The main unikernel entry point. *)
|
(* The main unikernel entry point. *)
|
||||||
let start () () () net =
|
let start () () () net =
|
||||||
let cidr = Key_gen.ipv4 ()
|
let cidr = Key_gen.ipv4 ()
|
||||||
and gateway = Key_gen.ipv4_gateway ()
|
and gateway = Key_gen.ipv4_gateway ()
|
||||||
in
|
in
|
||||||
let log_one = fun port ip -> log_one (M.elapsed_ns ()) port ip
|
let log_one = fun port ip -> log_one (M.elapsed_ns ()) port ip
|
||||||
(* Create a task to wait on and a waiter to wakeup. *)
|
(* Create a task to wait on and a waiter to wakeup. *)
|
||||||
and t, w = Lwt.task ()
|
and t, w = Lwt.task ()
|
||||||
in
|
in
|
||||||
(* Setup network stack: ethernet, ARP, IPv4, UDP, and ICMP. *)
|
(* Setup network stack: ethernet, ARP, IPv4, UDP, and ICMP. *)
|
||||||
ETH.connect net >>= fun eth ->
|
ETH.connect net >>= fun eth ->
|
||||||
ARP.connect eth >>= fun arp ->
|
ARP.connect eth >>= fun arp ->
|
||||||
IPV4.connect ~cidr ~gateway eth arp >>= fun ip ->
|
IPV4.connect ~cidr ~gateway eth arp >>= fun ip ->
|
||||||
UDP.connect ip >>= fun udp ->
|
UDP.connect ip >>= fun udp ->
|
||||||
let send = send_udp udp in
|
let send = send_udp udp in
|
||||||
Icmp.connect send log_one w >>= fun icmp ->
|
Icmp.connect send log_one w >>= fun icmp ->
|
||||||
|
|
||||||
(* The callback cascade for an incoming network packet. *)
|
(* The callback cascade for an incoming network packet. *)
|
||||||
let ethif_listener =
|
let ethif_listener =
|
||||||
|
|
@ -296,86 +269,78 @@ module Main (R : Mirage_random.S) (M : Mirage_clock.MCLOCK) (Time : Mirage_time.
|
||||||
~arpv4:(ARP.input arp)
|
~arpv4:(ARP.input arp)
|
||||||
~ipv4:(
|
~ipv4:(
|
||||||
IPV4.input
|
IPV4.input
|
||||||
~tcp:(fun ~src:_ ~dst:_ _ -> Lwt.return_unit)
|
~tcp:(fun ~src:_ ~dst:_ _ -> Lwt.return_unit)
|
||||||
~udp:(fun ~src:_ ~dst:_ _ -> Lwt.return_unit)
|
~udp:(fun ~src:_ ~dst:_ _ -> Lwt.return_unit)
|
||||||
~default:(fun ~proto ~src ~dst buf ->
|
~default:(fun ~proto ~src ~dst buf ->
|
||||||
match proto with
|
match proto with
|
||||||
| 1 -> Icmp.input icmp ~src ~dst buf
|
| 1 -> Icmp.input icmp ~src ~dst buf
|
||||||
| _ -> Lwt.return_unit)
|
| _ -> Lwt.return_unit)
|
||||||
ip)
|
ip)
|
||||||
~ipv6:(fun _ -> Lwt.return_unit)
|
~ipv6:(fun _ -> Lwt.return_unit)
|
||||||
eth
|
eth
|
||||||
in
|
in
|
||||||
(* Start the callback in a separate asynchronous task. *)
|
(* Start the callback in a separate asynchronous task. *)
|
||||||
Lwt.async (fun () ->
|
Lwt.async (fun () ->
|
||||||
N.listen net ~header_size:Ethernet_wire.sizeof_ethernet ethif_listener >|= function
|
N.listen net ~header_size:Ethernet_wire.sizeof_ethernet ethif_listener >|= function
|
||||||
| Ok () -> ()
|
| Ok () -> ()
|
||||||
| Error e -> Logs.err (fun m -> m "netif error %a" N.pp_error e));
|
| Error e -> Logs.err (fun m -> m "netif error %a" N.pp_error e));
|
||||||
(* Send the initial UDP packet with a ttl of 1. This entails the domino
|
(* Send the initial UDP packet with a ttl of 1. This entails the domino
|
||||||
effect to receive ICMP packets, send out another UDP packet with ttl
|
effect to receive ICMP packets, send out another UDP packet with ttl
|
||||||
increased by one, etc. - until a destination unreachable is received,
|
increased by one, etc. - until a destination unreachable is received,
|
||||||
or the hop limit is reached. *)
|
or the hop limit is reached. *)
|
||||||
send 1 >>= fun () ->
|
send 1 >>= fun () ->
|
||||||
t
|
t
|
||||||
end
|
end
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The configuration (<code>config.ml</code>) for this unikernel is as follows:</p>
|
||||||
The configuration (`config.ml`) for this unikernel is as follows:
|
<pre><code class="language-OCaml">open Mirage
|
||||||
|
|
||||||
```OCaml
|
|
||||||
open Mirage
|
|
||||||
|
|
||||||
let host =
|
let host =
|
||||||
let doc = Key.Arg.info ~doc:"The host to trace." ["host"] in
|
let doc = Key.Arg.info ~doc:"The host to trace." ["host"] in
|
||||||
Key.(create "host" Arg.(opt ipv4_address (Ipaddr.V4.of_string_exn "141.1.1.1") doc))
|
Key.(create "host" Arg.(opt ipv4_address (Ipaddr.V4.of_string_exn "141.1.1.1") doc))
|
||||||
|
|
||||||
let timeout =
|
let timeout =
|
||||||
let doc = Key.Arg.info ~doc:"Timeout (in millisecond)" ["timeout"] in
|
let doc = Key.Arg.info ~doc:"Timeout (in millisecond)" ["timeout"] in
|
||||||
Key.(create "timeout" Arg.(opt int 1000 doc))
|
Key.(create "timeout" Arg.(opt int 1000 doc))
|
||||||
|
|
||||||
let ipv4 =
|
let ipv4 =
|
||||||
let doc = Key.Arg.info ~doc:"IPv4 address" ["ipv4"] in
|
let doc = Key.Arg.info ~doc:"IPv4 address" ["ipv4"] in
|
||||||
Key.(create "ipv4" Arg.(required ipv4 doc))
|
Key.(create "ipv4" Arg.(required ipv4 doc))
|
||||||
|
|
||||||
let ipv4_gateway =
|
let ipv4_gateway =
|
||||||
let doc = Key.Arg.info ~doc:"IPv4 gateway" ["ipv4-gateway"] in
|
let doc = Key.Arg.info ~doc:"IPv4 gateway" ["ipv4-gateway"] in
|
||||||
Key.(create "ipv4-gateway" Arg.(required ipv4_address doc))
|
Key.(create "ipv4-gateway" Arg.(required ipv4_address doc))
|
||||||
|
|
||||||
let main =
|
let main =
|
||||||
let packages = [
|
let packages = [
|
||||||
package ~sublibs:["ipv4"; "udp"; "icmpv4"] "tcpip";
|
package ~sublibs:["ipv4"; "udp"; "icmpv4"] "tcpip";
|
||||||
package "ethernet";
|
package "ethernet";
|
||||||
package "arp-mirage";
|
package "arp-mirage";
|
||||||
package "mirage-protocols";
|
package "mirage-protocols";
|
||||||
package "mtime";
|
package "mtime";
|
||||||
] in
|
] in
|
||||||
foreign
|
foreign
|
||||||
~keys:[Key.abstract ipv4 ; Key.abstract ipv4_gateway ; Key.abstract host ; Key.abstract timeout]
|
~keys:[Key.abstract ipv4 ; Key.abstract ipv4_gateway ; Key.abstract host ; Key.abstract timeout]
|
||||||
~packages
|
~packages
|
||||||
"Unikernel.Main"
|
"Unikernel.Main"
|
||||||
(random @-> mclock @-> time @-> network @-> job)
|
(random @-> mclock @-> time @-> network @-> job)
|
||||||
|
|
||||||
let () =
|
let () =
|
||||||
register "traceroute"
|
register "traceroute"
|
||||||
[ main $ default_random $ default_monotonic_clock $ default_time $ default_network ]
|
[ main $ default_random $ default_monotonic_clock $ default_time $ default_network ]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>And voila, that's all the code. If you copy it together (or download the two
|
||||||
And voila, that's all the code. If you copy it together (or download the two
|
files from <a href="https://github.com/roburio/traceroute">the GitHub repository</a>),
|
||||||
files from [the GitHub repository](https://github.com/roburio/traceroute)),
|
and have OCaml, opam, and <a href="https://mirage.io/wiki/install">mirage (>= 3.8.0)</a> installed,
|
||||||
and have OCaml, opam, and [mirage (>= 3.8.0)](https://mirage.io/wiki/install) installed,
|
you should be able to:</p>
|
||||||
you should be able to:
|
<pre><code class="language-bash">$ mirage configure -t hvt
|
||||||
```bash
|
|
||||||
$ mirage configure -t hvt
|
|
||||||
$ make depend
|
$ make depend
|
||||||
$ make
|
$ make
|
||||||
$ solo5-hvt --net:service=tap0 -- traceroute.hvt ...
|
$ solo5-hvt --net:service=tap0 -- traceroute.hvt ...
|
||||||
... get the output shown at top ...
|
... get the output shown at top ...
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Enhancements may be to use a different protocol (TCP? or any other protocol ID (may be used to encode more information), encode data into IPv4 ID, or the full 8 bytes of the upper protocol), encrypt/authenticate the data transmitted (and verify it has not been tampered with in the ICMP reply), improve error handling and recovery, send multiple packets for improved round trip time measurements, ...</p>
|
||||||
Enhancements may be to use a different protocol (TCP? or any other protocol ID (may be used to encode more information), encode data into IPv4 ID, or the full 8 bytes of the upper protocol), encrypt/authenticate the data transmitted (and verify it has not been tampered with in the ICMP reply), improve error handling and recovery, send multiple packets for improved round trip time measurements, ...
|
<p>If you develop enhancements you'd like to share, please sent a pull request to the git repository.</p>
|
||||||
|
<p>Motivation for this traceroute unikernel was while talking with <a href="https://twitter.com/networkservice">Aaron</a> and <a href="https://github.com/phaer">Paul</a>, who contributed several patches to the IP stack which pass the ttl through.</p>
|
||||||
If you develop enhancements you'd like to share, please sent a pull request to the git repository.
|
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
Motivation for this traceroute unikernel was while talking with [Aaron](https://twitter.com/networkservice) and [Paul](https://github.com/phaer), who contributed several patches to the IP stack which pass the ttl through.
|
|
||||||
|
|
||||||
If you want to support our work on MirageOS unikernels, please [donate to robur](https://robur.coop/Donate). I'm interested in feedback, either via [twitter](https://twitter.com/h4nnes), [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
289
Posts/VMM
289
Posts/VMM
|
|
@ -1,133 +1,112 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Albatross - provisioning, deploying, managing, and monitoring virtual machines
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Albatross - provisioning, deploying, managing, and monitoring virtual machines</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Albatross - provisioning, deploying, managing, and monitoring virtual machines" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a><a href="/tags/provisioning" class="tag">provisioning</a></div><span class="date">Published: 2017-07-10 (last updated: 2023-05-16)</span><article><p>EDIT (2023-05-16): Please take a look at <a href="/Posts/Albatross">the updated article</a>.</p>
|
||||||
author: hannes
|
<h2 id="how-to-deploy-unikernels">How to deploy unikernels?</h2>
|
||||||
tags: mirageos, deployment, provisioning
|
<p>MirageOS has a pretty good story on how to compose your OCaml libraries into a
|
||||||
abstract: all we need is X.509
|
virtual machine image. The <code>mirage</code> command line utility contains all the
|
||||||
---
|
|
||||||
|
|
||||||
EDIT (2023-05-16): Please take a look at [the updated article](/Posts/Albatross).
|
|
||||||
|
|
||||||
## How to deploy unikernels?
|
|
||||||
|
|
||||||
MirageOS has a pretty good story on how to compose your OCaml libraries into a
|
|
||||||
virtual machine image. The `mirage` command line utility contains all the
|
|
||||||
knowledge about which backend requires which library. This enables it to write a
|
knowledge about which backend requires which library. This enables it to write a
|
||||||
unikernel using abstract interfaces (such as a network device). Additionally the
|
unikernel using abstract interfaces (such as a network device). Additionally the
|
||||||
`mirage` utility can compile for any backend. (It is still unclear whether this
|
<code>mirage</code> utility can compile for any backend. (It is still unclear whether this
|
||||||
is a sustainable idea, since the `mirage` tool needs to be adjusted for every
|
is a sustainable idea, since the <code>mirage</code> tool needs to be adjusted for every
|
||||||
new backend, but also for additional implementations of an interface.)
|
new backend, but also for additional implementations of an interface.)</p>
|
||||||
|
<p>Once a virtual machine image has been created, it needs to be deployed. I run
|
||||||
Once a virtual machine image has been created, it needs to be deployed. I run
|
|
||||||
my own physical hardware, with all the associated upsides and downsides.
|
my own physical hardware, with all the associated upsides and downsides.
|
||||||
Specifically I run several physical [FreeBSD](https://freebsd.org) machines on
|
Specifically I run several physical <a href="https://freebsd.org">FreeBSD</a> machines on
|
||||||
the Internet, and use the [bhyve](http://bhyve.org) hypervisor with MirageOS as
|
the Internet, and use the <a href="http://bhyve.org">bhyve</a> hypervisor with MirageOS as
|
||||||
described [earlier](/Posts/Solo5). Recently, Martin
|
described <a href="/Posts/Solo5">earlier</a>. Recently, Martin
|
||||||
Lucina
|
Lucina
|
||||||
[developed](https://github.com/Solo5/solo5/pull/171/commits/e67a007b75fa3fcee5c082aab04c9fe9e897d779)
|
<a href="https://github.com/Solo5/solo5/pull/171/commits/e67a007b75fa3fcee5c082aab04c9fe9e897d779">developed</a>
|
||||||
a
|
a
|
||||||
[`vmm`](https://svnweb.freebsd.org/base/head/sys/amd64/include/vmm.h?view=markup)
|
<a href="https://svnweb.freebsd.org/base/head/sys/amd64/include/vmm.h?view=markup"><code>vmm</code></a>
|
||||||
backend for [Solo5](https://github.com/solo5/solo5). This means there is no
|
backend for <a href="https://github.com/solo5/solo5">Solo5</a>. This means there is no
|
||||||
need to use virtio anymore, or grub2-bhyve, or the bhyve binary (which links
|
need to use virtio anymore, or grub2-bhyve, or the bhyve binary (which links
|
||||||
`libvmmapi` that already had a [security
|
<code>libvmmapi</code> that already had a <a href="https://www.freebsd.org/security/advisories/FreeBSD-SA-16:38.bhyve.asc">security
|
||||||
advisory](https://www.freebsd.org/security/advisories/FreeBSD-SA-16:38.bhyve.asc)).
|
advisory</a>).
|
||||||
Instead of the bhyve binary, a ~70kB small `ukvm-bin` binary (dynamically
|
Instead of the bhyve binary, a ~70kB small <code>ukvm-bin</code> binary (dynamically
|
||||||
linking libc) can be used which is the solo5 virtual machine monitor on the host
|
linking libc) can be used which is the solo5 virtual machine monitor on the host
|
||||||
side.
|
side.</p>
|
||||||
|
<p>Until now, I manually created and deployed virtual machines using shell scripts,
|
||||||
Until now, I manually created and deployed virtual machines using shell scripts,
|
|
||||||
ssh logins, and a network file system shared with the FreeBSD virtual machine
|
ssh logins, and a network file system shared with the FreeBSD virtual machine
|
||||||
which builds my MirageOS unikernels.
|
which builds my MirageOS unikernels.</p>
|
||||||
|
<p>But there are several drawbacks with this approach, the biggest is that sharing
|
||||||
But there are several drawbacks with this approach, the biggest is that sharing
|
|
||||||
resources is hard - to enable a friend to run their unikernel on my server,
|
resources is hard - to enable a friend to run their unikernel on my server,
|
||||||
they'll need to have a user account, and even privileged permissions to
|
they'll need to have a user account, and even privileged permissions to
|
||||||
create virtual network interfaces and execute virtual machines.
|
create virtual network interfaces and execute virtual machines.</p>
|
||||||
|
<p>To get rid of these ad-hoc shell scripts and copying of virtual machine images,
|
||||||
To get rid of these ad-hoc shell scripts and copying of virtual machine images,
|
|
||||||
I developed an UNIX daemon which accomplishes the required work. This daemon
|
I developed an UNIX daemon which accomplishes the required work. This daemon
|
||||||
waits for (mutually!) authenticated network connections, and provides the
|
waits for (mutually!) authenticated network connections, and provides the
|
||||||
desired commands; to create a new virtual machine, to acquire a block device of
|
desired commands; to create a new virtual machine, to acquire a block device of
|
||||||
a given size, to destroy a virtual machine, to stream the console output of a
|
a given size, to destroy a virtual machine, to stream the console output of a
|
||||||
virtual machine.
|
virtual machine.</p>
|
||||||
|
<h2 id="system-design">System design</h2>
|
||||||
## System design
|
<p>The system bears minimalistic characteristics. The single interface to the
|
||||||
|
|
||||||
The system bears minimalistic characteristics. The single interface to the
|
|
||||||
outside world is a TLS stream over TCP. Internally, there is a family of
|
outside world is a TLS stream over TCP. Internally, there is a family of
|
||||||
processes, one of which has superuser privileges, communicating via unix domain
|
processes, one of which has superuser privileges, communicating via unix domain
|
||||||
sockets. The processes do not need any persistent storage (apart from the
|
sockets. The processes do not need any persistent storage (apart from the
|
||||||
revocation lists). A brief enumeration of the processes is provided below:
|
revocation lists). A brief enumeration of the processes is provided below:</p>
|
||||||
|
<ul>
|
||||||
* `vmmd` (superuser privileges), which terminates TLS sessions, proxies messages, and creates and destroys virtual machines (including setup and teardown of network interfaces and virtual block devices)
|
<li><code>vmmd</code> (superuser privileges), which terminates TLS sessions, proxies messages, and creates and destroys virtual machines (including setup and teardown of network interfaces and virtual block devices)
|
||||||
* `vmm_stats` periodically gathers resource usage and network interface statistics
|
</li>
|
||||||
* `vmm_console` reads console output of every provided fifo, and stores this in a ringbuffer, replaying to a client on demand
|
<li><code>vmm_stats</code> periodically gathers resource usage and network interface statistics
|
||||||
* `vmm_log` consumes the event log (login, starting, and stopping of virtual machines)
|
</li>
|
||||||
|
<li><code>vmm_console</code> reads console output of every provided fifo, and stores this in a ringbuffer, replaying to a client on demand
|
||||||
The system uses X.509 certificates as tokens. These are authenticated key value
|
</li>
|
||||||
stores. There are four shapes of certificates: a *virtual machine certificate*
|
<li><code>vmm_log</code> consumes the event log (login, starting, and stopping of virtual machines)
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
<p>The system uses X.509 certificates as tokens. These are authenticated key value
|
||||||
|
stores. There are four shapes of certificates: a <em>virtual machine certificate</em>
|
||||||
which embeds the entire virtual machine image, together with configuration
|
which embeds the entire virtual machine image, together with configuration
|
||||||
information (resource usage, how many and which network interfaces, block device
|
information (resource usage, how many and which network interfaces, block device
|
||||||
access); a *command certificate* (for interactive use, allowing (a subset of)
|
access); a <em>command certificate</em> (for interactive use, allowing (a subset of)
|
||||||
commands such as attaching to console output); a *revocation certificate* which
|
commands such as attaching to console output); a <em>revocation certificate</em> which
|
||||||
contains a list of revoked certificates; and a *delegation certificate* to
|
contains a list of revoked certificates; and a <em>delegation certificate</em> to
|
||||||
distribute resources to someone else (an intermediate CA certificate).
|
distribute resources to someone else (an intermediate CA certificate).</p>
|
||||||
|
<p>The resources which can be controlled are CPUs, memory consumption, block
|
||||||
The resources which can be controlled are CPUs, memory consumption, block
|
|
||||||
storage, and access to bridge interfaces (virtual switches) - encoded in the
|
storage, and access to bridge interfaces (virtual switches) - encoded in the
|
||||||
virtual machine and delegation certificates. Additionally, delegation
|
virtual machine and delegation certificates. Additionally, delegation
|
||||||
certificates can limit the number of virtual machines.
|
certificates can limit the number of virtual machines.</p>
|
||||||
|
<p>Leveraging the X.509 system ensures that the client always has to present a
|
||||||
Leveraging the X.509 system ensures that the client always has to present a
|
|
||||||
certificate chain from the root certificate. Each intermediate certificate is a
|
certificate chain from the root certificate. Each intermediate certificate is a
|
||||||
delegation certificate, which may further restrict resources. The serial
|
delegation certificate, which may further restrict resources. The serial
|
||||||
numbers of the chain is used as unique identifier for each virtual machine and
|
numbers of the chain is used as unique identifier for each virtual machine and
|
||||||
other certificates. The chain restricts access of the leaf certificate as well:
|
other certificates. The chain restricts access of the leaf certificate as well:
|
||||||
only the subtree of the chain can be viewed. E.g. if there are delegations to
|
only the subtree of the chain can be viewed. E.g. if there are delegations to
|
||||||
both Alice and Bob from the root certificate, they can not see each other
|
both Alice and Bob from the root certificate, they can not see each other
|
||||||
virtual machines.
|
virtual machines.</p>
|
||||||
|
<p>Connecting to the vmmd requires a TLS client, a CA certificate, a leaf
|
||||||
Connecting to the vmmd requires a TLS client, a CA certificate, a leaf
|
|
||||||
certificate (and the delegation chain) and its private key. In the background,
|
certificate (and the delegation chain) and its private key. In the background,
|
||||||
it is a multi-step process using TLS: first, the client establishes a TLS
|
it is a multi-step process using TLS: first, the client establishes a TLS
|
||||||
connection where it authenticates the server using the CA certificate, then the
|
connection where it authenticates the server using the CA certificate, then the
|
||||||
server demands a TLS renegotiation where it requires the client to authenticate
|
server demands a TLS renegotiation where it requires the client to authenticate
|
||||||
with its leaf certificate and private key. Using renegotiation over the
|
with its leaf certificate and private key. Using renegotiation over the
|
||||||
encrypted channel prevents passive observers to see the client certificate in
|
encrypted channel prevents passive observers to see the client certificate in
|
||||||
clear.
|
clear.</p>
|
||||||
|
<p>Depending on the leaf certificate, the server logic is slightly different. A
|
||||||
Depending on the leaf certificate, the server logic is slightly different. A
|
|
||||||
command certificate opens an interactive session where - depending on
|
command certificate opens an interactive session where - depending on
|
||||||
permissions encoded in the certificate - different commands can be issued: the
|
permissions encoded in the certificate - different commands can be issued: the
|
||||||
console output can be streamed, the event log can be viewed, virtual machines
|
console output can be streamed, the event log can be viewed, virtual machines
|
||||||
can be destroyed, statistics can be collected, and block devices can be managed.
|
can be destroyed, statistics can be collected, and block devices can be managed.</p>
|
||||||
|
<p>When a virtual machine certificate is presented, the desired resource usage is
|
||||||
When a virtual machine certificate is presented, the desired resource usage is
|
|
||||||
checked against the resource policies in the delegation certificate chain and
|
checked against the resource policies in the delegation certificate chain and
|
||||||
the currently running virtual machines. If sufficient resources are free, the
|
the currently running virtual machines. If sufficient resources are free, the
|
||||||
embedded virtual machine is started. In addition to other resource information,
|
embedded virtual machine is started. In addition to other resource information,
|
||||||
a delegation certificate may embed IP usage, listing the network configuration
|
a delegation certificate may embed IP usage, listing the network configuration
|
||||||
(gateway and netmask), and which addresses you're supposed to use. Boot
|
(gateway and netmask), and which addresses you're supposed to use. Boot
|
||||||
arguments can be encoded in the certificate as well, they are just passed to the
|
arguments can be encoded in the certificate as well, they are just passed to the
|
||||||
virtual machine (for easy deployment of off-the-shelf systems).
|
virtual machine (for easy deployment of off-the-shelf systems).</p>
|
||||||
|
<p>If a revocation certificate is presented, the embodied revocation list is
|
||||||
If a revocation certificate is presented, the embodied revocation list is
|
|
||||||
verified, and stored on the host system. Revocation is enforced by destroying
|
verified, and stored on the host system. Revocation is enforced by destroying
|
||||||
any revoked virtual machines and terminating any revoked interactive sessions.
|
any revoked virtual machines and terminating any revoked interactive sessions.
|
||||||
If a delegation certificate is revoked, additionally the connected block devices
|
If a delegation certificate is revoked, additionally the connected block devices
|
||||||
are destroyed.
|
are destroyed.</p>
|
||||||
|
<p>The maximum size of a virtual machine image embedded into a X.509 certificate
|
||||||
The maximum size of a virtual machine image embedded into a X.509 certificate
|
|
||||||
transferred over TLS is 2 ^ 24 - 1 bytes, roughly 16 MB. If this turns out to
|
transferred over TLS is 2 ^ 24 - 1 bytes, roughly 16 MB. If this turns out to
|
||||||
be not sufficient, compression may help. Or staging of deployment.
|
be not sufficient, compression may help. Or staging of deployment.</p>
|
||||||
|
<h2 id="an-example">An example</h2>
|
||||||
## An example
|
<p>Instructions on how to setup <code>vmmd</code> and the certificate authority are in the
|
||||||
|
README file of the <a href="https://github.com/hannesm/albatross"><code>albatross</code> git repository</a>. Here
|
||||||
Instructions on how to setup `vmmd` and the certificate authority are in the
|
is some (stripped) terminal output:</p>
|
||||||
README file of the [`albatross` git repository](https://github.com/hannesm/albatross). Here
|
<pre><code class="language-bash">> openssl x509 -text -noout -in admin.pem
|
||||||
is some (stripped) terminal output:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
> openssl x509 -text -noout -in admin.pem
|
|
||||||
Certificate:
|
Certificate:
|
||||||
Data:
|
Data:
|
||||||
Serial Number: b7:aa:77:f6:ca:08:ee:6a
|
Serial Number: b7:aa:77:f6:ca:08:ee:6a
|
||||||
|
|
@ -138,7 +117,7 @@ Certificate:
|
||||||
1.3.6.1.4.1.49836.42.42: ....
|
1.3.6.1.4.1.49836.42.42: ....
|
||||||
1.3.6.1.4.1.49836.42.0: ...
|
1.3.6.1.4.1.49836.42.0: ...
|
||||||
|
|
||||||
> openssl asn1parse -in admin.pem
|
> openssl asn1parse -in admin.pem
|
||||||
403:d=4 hl=2 l= 18 cons: SEQUENCE
|
403:d=4 hl=2 l= 18 cons: SEQUENCE
|
||||||
405:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.42
|
405:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.42
|
||||||
417:d=5 hl=2 l= 4 prim: OCTET STRING [HEX DUMP]:03020780
|
417:d=5 hl=2 l= 4 prim: OCTET STRING [HEX DUMP]:03020780
|
||||||
|
|
@ -146,7 +125,7 @@ Certificate:
|
||||||
425:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.0
|
425:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.0
|
||||||
437:d=5 hl=2 l= 3 prim: OCTET STRING [HEX DUMP]:020100
|
437:d=5 hl=2 l= 3 prim: OCTET STRING [HEX DUMP]:020100
|
||||||
|
|
||||||
> openssl asn1parse -in hello.pem
|
> openssl asn1parse -in hello.pem
|
||||||
410:d=4 hl=2 l= 18 cons: SEQUENCE
|
410:d=4 hl=2 l= 18 cons: SEQUENCE
|
||||||
412:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.42
|
412:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.42
|
||||||
424:d=5 hl=2 l= 4 prim: OCTET STRING [HEX DUMP]:03020520
|
424:d=5 hl=2 l= 4 prim: OCTET STRING [HEX DUMP]:03020520
|
||||||
|
|
@ -159,25 +138,20 @@ Certificate:
|
||||||
469:d=4 hl=5 l=3054024 cons: SEQUENCE
|
469:d=4 hl=5 l=3054024 cons: SEQUENCE
|
||||||
474:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.9
|
474:d=5 hl=2 l= 10 prim: OBJECT :1.3.6.1.4.1.49836.42.9
|
||||||
486:d=5 hl=5 l=3054007 prim: OCTET STRING [HEX DUMP]:A0832E99B204832E99AD7F454C46
|
486:d=5 hl=5 l=3054007 prim: OCTET STRING [HEX DUMP]:A0832E99B204832E99AD7F454C46
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The MirageOS private enterprise number is 1.3.6.1.4.1.49836, I use the arc 42
|
||||||
The MirageOS private enterprise number is 1.3.6.1.4.1.49836, I use the arc 42
|
here. I use 0 as version (an integer), where 0 is the current version.</p>
|
||||||
here. I use 0 as version (an integer), where 0 is the current version.
|
<p>42 is a bit string representing the permissions. 5 the amount of memory, 6 the
|
||||||
|
|
||||||
42 is a bit string representing the permissions. 5 the amount of memory, 6 the
|
|
||||||
CPU id, and 9 finally the virtual machine image (as ELF binary). If you're
|
CPU id, and 9 finally the virtual machine image (as ELF binary). If you're
|
||||||
eager to see more, look into the `Vmm_asn` module.
|
eager to see more, look into the <code>Vmm_asn</code> module.</p>
|
||||||
|
<p>Using a command certificate establishes an interactive session where you can
|
||||||
Using a command certificate establishes an interactive session where you can
|
|
||||||
review the event log, see all currently running virtual machines, or attach to
|
review the event log, see all currently running virtual machines, or attach to
|
||||||
the console (which is then streamed, if new console output appears while the
|
the console (which is then streamed, if new console output appears while the
|
||||||
interactive session is active, you'll be notified). The `db` file is used to
|
interactive session is active, you'll be notified). The <code>db</code> file is used to
|
||||||
translate between the internal names (mentioned above, hashed serial numbers) to
|
translate between the internal names (mentioned above, hashed serial numbers) to
|
||||||
common names of the certificates - both on command input (`attach hello`) and
|
common names of the certificates - both on command input (<code>attach hello</code>) and
|
||||||
output.
|
output.</p>
|
||||||
|
<pre><code class="language-bash">> vmm_client cacert.pem admin.bundle admin.key localhost:1025 --db dev.db
|
||||||
```bash
|
|
||||||
> vmm_client cacert.pem admin.bundle admin.key localhost:1025 --db dev.db
|
|
||||||
$ info
|
$ info
|
||||||
info sn.nqsb.io: 'cpuset' '-l' '7' '/tmp/vmm/ukvm-bin.net' '--net=tap27' '--' '/tmp/81363f.0237f3.img' 91540 taps tap27
|
info sn.nqsb.io: 'cpuset' '-l' '7' '/tmp/vmm/ukvm-bin.net' '--net=tap27' '--' '/tmp/81363f.0237f3.img' 91540 taps tap27
|
||||||
info nqsbio: 'cpuset' '-l' '5' '/tmp/vmm/ukvm-bin.net' '--net=tap26' '--' '/tmp/81363f.43a0ff.img' 91448 taps tap26
|
info nqsbio: 'cpuset' '-l' '5' '/tmp/vmm/ukvm-bin.net' '--net=tap26' '--' '/tmp/81363f.43a0ff.img' 91448 taps tap26
|
||||||
|
|
@ -204,32 +178,26 @@ console hello: 2017-07-09 18:44:52 +00:00 Solo5: unused @ (0x0 - 0xfffff)
|
||||||
console hello: 2017-07-09 18:44:52 +00:00 Solo5: text @ (0x100000 - 0x1e4fff)
|
console hello: 2017-07-09 18:44:52 +00:00 Solo5: text @ (0x100000 - 0x1e4fff)
|
||||||
console hello: 2017-07-09 18:44:52 +00:00 Solo5: rodata @ (0x1e5000 - 0x217fff)
|
console hello: 2017-07-09 18:44:52 +00:00 Solo5: rodata @ (0x1e5000 - 0x217fff)
|
||||||
console hello: 2017-07-09 18:44:52 +00:00 Solo5: data @ (0x218000 - 0x2cffff)
|
console hello: 2017-07-09 18:44:52 +00:00 Solo5: data @ (0x218000 - 0x2cffff)
|
||||||
console hello: 2017-07-09 18:44:52 +00:00 Solo5: heap >= 0x2d0000 < stack < 0x20000000
|
console hello: 2017-07-09 18:44:52 +00:00 Solo5: heap >= 0x2d0000 < stack < 0x20000000
|
||||||
console hello: 2017-07-09 18:44:52 +00:00 STUB: getenv() called
|
console hello: 2017-07-09 18:44:52 +00:00 STUB: getenv() called
|
||||||
console hello: 2017-07-09 18:44:52 +00:00 2017-07-09 18:44:52 -00:00: INF [application] hello
|
console hello: 2017-07-09 18:44:52 +00:00 2017-07-09 18:44:52 -00:00: INF [application] hello
|
||||||
console hello: 2017-07-09 18:44:53 +00:00 2017-07-09 18:44:53 -00:00: INF [application] hello
|
console hello: 2017-07-09 18:44:53 +00:00 2017-07-09 18:44:53 -00:00: INF [application] hello
|
||||||
console hello: 2017-07-09 18:44:54 +00:00 2017-07-09 18:44:54 -00:00: INF [application] hello
|
console hello: 2017-07-09 18:44:54 +00:00 2017-07-09 18:44:54 -00:00: INF [application] hello
|
||||||
console hello: 2017-07-09 18:44:55 +00:00 2017-07-09 18:44:55 -00:00: INF [application] hello
|
console hello: 2017-07-09 18:44:55 +00:00 2017-07-09 18:44:55 -00:00: INF [application] hello
|
||||||
```
|
</code></pre>
|
||||||
|
<p>If you use a virtual machine certificate, depending on allowed resource the
|
||||||
If you use a virtual machine certificate, depending on allowed resource the
|
virtual machine is started or not:</p>
|
||||||
virtual machine is started or not:
|
<pre><code class="language-bash">> vmm_client cacert.pem hello.bundle hello.key localhost:1025
|
||||||
|
|
||||||
```bash
|
|
||||||
> vmm_client cacert.pem hello.bundle hello.key localhost:1025
|
|
||||||
success VM started
|
success VM started
|
||||||
```
|
</code></pre>
|
||||||
|
<h2 id="sharing-is-caring">Sharing is caring</h2>
|
||||||
## Sharing is caring
|
<p>Deploying unikernels is now easier for myself on my physical machine. That's
|
||||||
|
fine. Another aspect comes <em>for free</em> by reusing X.509: further delegation (and
|
||||||
Deploying unikernels is now easier for myself on my physical machine. That's
|
|
||||||
fine. Another aspect comes *for free* by reusing X.509: further delegation (and
|
|
||||||
limiting thereof). Within a delegation certificate, the basic constraints
|
limiting thereof). Within a delegation certificate, the basic constraints
|
||||||
extension must be present which marks this certificate as a CA certificate.
|
extension must be present which marks this certificate as a CA certificate.
|
||||||
This may as well contain a path length - how many other delegations may follow -
|
This may as well contain a path length - how many other delegations may follow -
|
||||||
or whether the resources may be shared further.
|
or whether the resources may be shared further.</p>
|
||||||
|
<p>If I delegate 2 virtual machines and 2GB of memory to Alice, and allow an
|
||||||
If I delegate 2 virtual machines and 2GB of memory to Alice, and allow an
|
|
||||||
arbitrary path length, she can issue tokens to her friend Carol and Dan, each up
|
arbitrary path length, she can issue tokens to her friend Carol and Dan, each up
|
||||||
to 2 virtual machines and 2 GB memory (but also less -- within the X.509 system
|
to 2 virtual machines and 2 GB memory (but also less -- within the X.509 system
|
||||||
even more, but vmmd will reject any resource increase in the chain) - who can
|
even more, but vmmd will reject any resource increase in the chain) - who can
|
||||||
|
|
@ -238,90 +206,75 @@ and vmmd will only start up to 2 virtual machines using 2GB of memory in total
|
||||||
(sum of Alice, Carol, and Dan deployed virtual machines). Alice may revoke any
|
(sum of Alice, Carol, and Dan deployed virtual machines). Alice may revoke any
|
||||||
issued delegation (using a revocation certificate described above) to free up
|
issued delegation (using a revocation certificate described above) to free up
|
||||||
some resources for herself. I don't need to interact when Alice or Dan share
|
some resources for herself. I don't need to interact when Alice or Dan share
|
||||||
their delegated resources further.
|
their delegated resources further.</p>
|
||||||
|
<h2 id="security">Security</h2>
|
||||||
## Security
|
<p>There are several security properties preserved by <code>vmmd</code>, such as the virtual
|
||||||
|
|
||||||
There are several security properties preserved by `vmmd`, such as the virtual
|
|
||||||
machine image is never transmitted in clear. Only properly authenticated
|
machine image is never transmitted in clear. Only properly authenticated
|
||||||
clients can create, destroy, gather statistics of _their_ virtual machines.
|
clients can create, destroy, gather statistics of <em>their</em> virtual machines.</p>
|
||||||
|
<p>Two disjoint paths in the delegation tree are not able to discover anything
|
||||||
Two disjoint paths in the delegation tree are not able to discover anything
|
|
||||||
about each other (apart from caches, which depend on how CPUs are delegated and
|
about each other (apart from caches, which depend on how CPUs are delegated and
|
||||||
their concrete physical layout). Only smaller amounts of resources can be
|
their concrete physical layout). Only smaller amounts of resources can be
|
||||||
delegated further down. Each running virtual machine image is strongly isolated
|
delegated further down. Each running virtual machine image is strongly isolated
|
||||||
from all other virtual machines.
|
from all other virtual machines.</p>
|
||||||
|
<p>As mentioned in the last section, delegations of delegations may end up in the
|
||||||
As mentioned in the last section, delegations of delegations may end up in the
|
|
||||||
hands of malicious people. Vmmd limits delegations to allocate resources on the
|
hands of malicious people. Vmmd limits delegations to allocate resources on the
|
||||||
host system, namely bridges and file systems. Only top delegations - directly
|
host system, namely bridges and file systems. Only top delegations - directly
|
||||||
signed by the certificate authority - create bridge interfaces (which are
|
signed by the certificate authority - create bridge interfaces (which are
|
||||||
explicitly named in the certificate) and file systems (one zfs for each top
|
explicitly named in the certificate) and file systems (one zfs for each top
|
||||||
delegation (to allow easy snapshots and backups)).
|
delegation (to allow easy snapshots and backups)).</p>
|
||||||
|
<p>The threat model is that clients have layer 2 access to the hosts network
|
||||||
The threat model is that clients have layer 2 access to the hosts network
|
|
||||||
interface card, and all guests share a single bridge (if this turns out to be a
|
interface card, and all guests share a single bridge (if this turns out to be a
|
||||||
problem, there are ways to restrict to a point-to-point interface with routed IP
|
problem, there are ways to restrict to a point-to-point interface with routed IP
|
||||||
addresses). A malicious virtual machine can try to hijack ethernet and IP
|
addresses). A malicious virtual machine can try to hijack ethernet and IP
|
||||||
addresses.
|
addresses.</p>
|
||||||
|
<p>Possible DoS scenarios include also to spawn VMs very fast (which immediately
|
||||||
Possible DoS scenarios include also to spawn VMs very fast (which immediately
|
|
||||||
crash) or generating a lot of console output. Both is indirectly handled by the
|
crash) or generating a lot of console output. Both is indirectly handled by the
|
||||||
control channel: to create a virtual machine image, you need to setup a TLS
|
control channel: to create a virtual machine image, you need to setup a TLS
|
||||||
connection (with two handshakes) and transfer the virtual machine image (there
|
connection (with two handshakes) and transfer the virtual machine image (there
|
||||||
is intentionally no "respawn on quit" option). The console output is read by a
|
is intentionally no "respawn on quit" option). The console output is read by a
|
||||||
single process with user privileges (in the future there may be one console
|
single process with user privileges (in the future there may be one console
|
||||||
reading process for each top delegation). It may further be rate limited as
|
reading process for each top delegation). It may further be rate limited as
|
||||||
well. The console stream is only ever sent to a single session, as soon as
|
well. The console stream is only ever sent to a single session, as soon as
|
||||||
someone attaches to the console in one session, all other sessions have this
|
someone attaches to the console in one session, all other sessions have this
|
||||||
console detached (and are notified about that).
|
console detached (and are notified about that).</p>
|
||||||
|
<p>The control channel itself can be rate limited using the host system firewall.</p>
|
||||||
The control channel itself can be rate limited using the host system firewall.
|
<p>The only information persistently stored on a block device are the certificate
|
||||||
|
|
||||||
The only information persistently stored on a block device are the certificate
|
|
||||||
revocation lists - virtual machine images, FIFOs, unix domain sockets are all
|
revocation lists - virtual machine images, FIFOs, unix domain sockets are all
|
||||||
stored in a memory-backed file system. A virtual machine with a lots of disk
|
stored in a memory-backed file system. A virtual machine with a lots of disk
|
||||||
operation may only delay or starve revocation list updates - if this turns out
|
operation may only delay or starve revocation list updates - if this turns out
|
||||||
to be a problem, the solution may be to use separate physical block devices for
|
to be a problem, the solution may be to use separate physical block devices for
|
||||||
the revocation lists and virtual block devices for clients.
|
the revocation lists and virtual block devices for clients.</p>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
## Conclusion
|
<p>I showed a minimalistic system to provision, deploy, and manage virtual machine
|
||||||
|
|
||||||
I showed a minimalistic system to provision, deploy, and manage virtual machine
|
|
||||||
images. It also allows to delegate resources (CPU, disk, ..) further. I'm
|
images. It also allows to delegate resources (CPU, disk, ..) further. I'm
|
||||||
pretty satisfied with the security properties of the system.
|
pretty satisfied with the security properties of the system.</p>
|
||||||
|
<p>The system embeds all data (configuration, resource policies, virtual machine
|
||||||
The system embeds all data (configuration, resource policies, virtual machine
|
|
||||||
images) into X.509 certificates, and does not rely on an external file transfer
|
images) into X.509 certificates, and does not rely on an external file transfer
|
||||||
protocol. An advantage thereof is that all deployed images have been signed
|
protocol. An advantage thereof is that all deployed images have been signed
|
||||||
with a private key.
|
with a private key.</p>
|
||||||
|
<p>All communication between the processes and between the client and the server
|
||||||
All communication between the processes and between the client and the server
|
|
||||||
use a wire protocol, with structured input and output - this enables more
|
use a wire protocol, with structured input and output - this enables more
|
||||||
advanced algorithms (e.g. automated scaling) and fancier user interfaces than
|
advanced algorithms (e.g. automated scaling) and fancier user interfaces than
|
||||||
the currently provided terminal based one.
|
the currently provided terminal based one.</p>
|
||||||
|
<p>The delegation mechanism allows to actually share computing resources in a
|
||||||
The delegation mechanism allows to actually share computing resources in a
|
|
||||||
decentralised way - without knowing the final recipient. Revocation is builtin,
|
decentralised way - without knowing the final recipient. Revocation is builtin,
|
||||||
which can at any point delete access of a subtree or individual virtual machine
|
which can at any point delete access of a subtree or individual virtual machine
|
||||||
to the system. Instead of requesting revocation lists during the handshake,
|
to the system. Instead of requesting revocation lists during the handshake,
|
||||||
they are pushed explicitly by the (sub)CA revoking a certificate.
|
they are pushed explicitly by the (sub)CA revoking a certificate.</p>
|
||||||
|
<p>While this system was designed for a physical server, it should be
|
||||||
While this system was designed for a physical server, it should be
|
|
||||||
straightforward to develop a Google compute engine / EC2 backend which extracts
|
straightforward to develop a Google compute engine / EC2 backend which extracts
|
||||||
the virtual machine image, commands, etc. from the certificate and deploys it to
|
the virtual machine image, commands, etc. from the certificate and deploys it to
|
||||||
your favourite cloud provider. A virtual machine image itself is only
|
your favourite cloud provider. A virtual machine image itself is only
|
||||||
processor-specific, and should be portable between different hypervisors - being
|
processor-specific, and should be portable between different hypervisors - being
|
||||||
it FreeBSD and VMM, Linux and KVM, or MacOSX and Hypervisor.Framework.
|
it FreeBSD and VMM, Linux and KVM, or MacOSX and Hypervisor.Framework.</p>
|
||||||
|
<p>The code is available <a href="https://github.com/hannesm/albatross">on GitHub</a>. If you want
|
||||||
The code is available [on GitHub](https://github.com/hannesm/albatross). If you want
|
|
||||||
to deploy your unikernel on my hardware, please send me a certificate signing
|
to deploy your unikernel on my hardware, please send me a certificate signing
|
||||||
request. I'm interested in feedback, either via
|
request. I'm interested in feedback, either via
|
||||||
[twitter](https://twitter.com/h4nnes) or open issues in the repository. This
|
<a href="https://twitter.com/h4nnes">twitter</a> or open issues in the repository. This
|
||||||
article itself is stored [in a different
|
article itself is stored <a href="https://git.robur.io/hannes/hannes.robur.coop">in a different
|
||||||
repository](https://git.robur.io/hannes/hannes.robur.coop) (in case you have typo or
|
repository</a> (in case you have typo or
|
||||||
grammatical corrections).
|
grammatical corrections).</p>
|
||||||
|
<p>I'm very thankful to people who gave feedback on earlier versions of this
|
||||||
I'm very thankful to people who gave feedback on earlier versions of this
|
|
||||||
article, and who discussed the system design with me. These are Addie, Chris,
|
article, and who discussed the system design with me. These are Addie, Chris,
|
||||||
Christiano, Joe, mato, Mindy, Mort, and sg.
|
Christiano, Joe, mato, Mindy, Mort, and sg.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
126
Posts/X50907
126
Posts/X50907
|
|
@ -1,72 +1,54 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: X509 0.7
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>X509 0.7</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="X509 0.7" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>X509 0.7</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/security" class="tag">security</a><a href="/tags/tls" class="tag">tls</a></div><span class="date">Published: 2019-08-15 (last updated: 2021-11-19)</span><article><h2 id="cryptographic-material">Cryptographic material</h2>
|
||||||
author: hannes
|
<p>Once a private and public key pair is generated (doesn't matter whether it is plain RSA, DSA, ECC on any curve), this is fine from a scientific point of view, and can already be used for authenticating and encrypting. From a practical point of view, the public parts need to be exchanged and verified (usually a fingerprint or hash thereof). This leads to the struggle how to encode this cryptographic material, and how to embed an identity (or multiple), capabilities, and other information into it. <a href="https://en.wikipedia.org/wiki/X.509">X.509</a> is a standard to solve this encoding and embedding, and provides more functionality, such as establishing chains of trust and revocation of invalidated or compromised material. X.509 uses certificates, which contain the public key, and additional information (in a extensible key-value store), and are signed by an issuer, either the private key corresponding to the public key - a so-called self-signed certificate - or by a different private key, an authority one step up the chain. A rather long, but very good introduction to certificates by Mike Malone is <a href="https://smallstep.com/blog/everything-pki.html">available here</a>.</p>
|
||||||
tags: mirageos, security, tls
|
<h2 id="ocaml-ecosystem-evolving">OCaml ecosystem evolving</h2>
|
||||||
abstract: Five years since ocaml-x509 initial release, it has been reworked and used more widely
|
<p>More than 5 years ago David Kaloper and I <a href="https://mirage.io/blog/introducing-x509">released the initial ocaml-x509</a> package as part of our <a href="https://nqsb.io">TLS stack</a>, which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in <a href="https://tools.ietf.org/html/rfc6125">RFC 5280</a>). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's <a href="https://github.com/mirleft/ocaml-asn1-combinators">asn1-combinators</a> library changed much over time.</p>
|
||||||
---
|
<p>The OCaml ecosystem evolved over the years, which lead to some changes:</p>
|
||||||
|
<ul>
|
||||||
## Cryptographic material
|
<li>Camlp4 deprecation - we used camlp4 for stream parsers of PEM-encoded certificates, and sexplib.syntax to derive s-expression decoders and encoders;
|
||||||
|
</li>
|
||||||
Once a private and public key pair is generated (doesn't matter whether it is plain RSA, DSA, ECC on any curve), this is fine from a scientific point of view, and can already be used for authenticating and encrypting. From a practical point of view, the public parts need to be exchanged and verified (usually a fingerprint or hash thereof). This leads to the struggle how to encode this cryptographic material, and how to embed an identity (or multiple), capabilities, and other information into it. [X.509](https://en.wikipedia.org/wiki/X.509) is a standard to solve this encoding and embedding, and provides more functionality, such as establishing chains of trust and revocation of invalidated or compromised material. X.509 uses certificates, which contain the public key, and additional information (in a extensible key-value store), and are signed by an issuer, either the private key corresponding to the public key - a so-called self-signed certificate - or by a different private key, an authority one step up the chain. A rather long, but very good introduction to certificates by Mike Malone is [available here](https://smallstep.com/blog/everything-pki.html).
|
<li>Avoiding brittle ppx converters - which we used for s-expression decoders and encoders of certificates after camlp4 was deprecated;
|
||||||
|
</li>
|
||||||
## OCaml ecosystem evolving
|
<li>Build and release system iterations - initially oasis and a packed library, then topkg and ocamlbuild, now dune;
|
||||||
|
</li>
|
||||||
More than 5 years ago David Kaloper and I [released the initial ocaml-x509](https://mirage.io/blog/introducing-x509) package as part of our [TLS stack](https://nqsb.io), which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in [RFC 5280](https://tools.ietf.org/html/rfc6125)). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's [asn1-combinators](https://github.com/mirleft/ocaml-asn1-combinators) library changed much over time.
|
<li>Introduction of the <code>result</code> type in the standard library - we used to use <code>[ `Ok of certificate option | `Fail of failure ]</code>;
|
||||||
|
</li>
|
||||||
The OCaml ecosystem evolved over the years, which lead to some changes:
|
<li>No more leaking exceptions in the public API;
|
||||||
- Camlp4 deprecation - we used camlp4 for stream parsers of PEM-encoded certificates, and sexplib.syntax to derive s-expression decoders and encoders;
|
</li>
|
||||||
- Avoiding brittle ppx converters - which we used for s-expression decoders and encoders of certificates after camlp4 was deprecated;
|
<li>Usage of pretty-printers, esp with the <a href="https://erratique.ch/software/fmt">fmt</a> library <code>val pp : Format.formatter -> 'a -> unit</code>, instead of <code>val to_string : t -> string</code> functions;
|
||||||
- Build and release system iterations - initially oasis and a packed library, then topkg and ocamlbuild, now dune;
|
</li>
|
||||||
- Introduction of the `result` type in the standard library - we used to use `` [ `Ok of certificate option | `Fail of failure ] ``;
|
<li>Release of <a href="https://erratique.ch/software/ptime">ptime</a>, a platform-independent POSIX time support;
|
||||||
- No more leaking exceptions in the public API;
|
</li>
|
||||||
- Usage of pretty-printers, esp with the [fmt](https://erratique.ch/software/fmt) library `val pp : Format.formatter -> 'a -> unit`, instead of `val to_string : t -> string` functions;
|
<li>Release of <a href="https://erratique.ch/software/rresult">rresult</a>, which includes combinators for computation <code>result</code>s;
|
||||||
- Release of [ptime](https://erratique.ch/software/ptime), a platform-independent POSIX time support;
|
</li>
|
||||||
- Release of [rresult](https://erratique.ch/software/rresult), which includes combinators for computation `result`s;
|
<li>Release of <a href="https://github.com/hannesm/gmap">gmap</a>, a <code>Map</code> whose value types depend on the key, used for X.509 extensions, GeneralName, DistinguishedName, etc.;
|
||||||
- Release of [gmap](https://github.com/hannesm/gmap), a `Map` whose value types depend on the key, used for X.509 extensions, GeneralName, DistinguishedName, etc.;
|
</li>
|
||||||
- Release of [domain-name](https://github.com/hannesm/domain-name), a library for domain name operations (as specified in [RFC 1035](https://tools.ietf.org/html/rfc1035)) - used for name validation;
|
<li>Release of <a href="https://github.com/hannesm/domain-name">domain-name</a>, a library for domain name operations (as specified in <a href="https://tools.ietf.org/html/rfc1035">RFC 1035</a>) - used for name validation;
|
||||||
- Usage of the [alcotest](https://github.com/mirage/alcotest) unit testing framework (instead of oUnit).
|
</li>
|
||||||
|
<li>Usage of the <a href="https://github.com/mirage/alcotest">alcotest</a> unit testing framework (instead of oUnit).
|
||||||
## More use cases for X.509
|
</li>
|
||||||
|
</ul>
|
||||||
Initially, we designed and used ocaml-x509 for providing TLS server endpoints and validation in TLS clients - mostly on the public web, where each operating system ships a set of ~100 trust anchors to validate any web server certificate against. But once you have a X.509 implementation, every authentication problem can be solved by applying it.
|
<h2 id="more-use-cases-for-x.509">More use cases for X.509</h2>
|
||||||
|
<p>Initially, we designed and used ocaml-x509 for providing TLS server endpoints and validation in TLS clients - mostly on the public web, where each operating system ships a set of ~100 trust anchors to validate any web server certificate against. But once you have a X.509 implementation, every authentication problem can be solved by applying it.</p>
|
||||||
### Authentication with path building
|
<h3 id="authentication-with-path-building">Authentication with path building</h3>
|
||||||
|
<p>It turns out that the trust anchor sets are not equal across operating systems and versions, thus some web servers serve sets, instead of chains, of certificates - as described in <a href="https://tools.ietf.org/html/rfc4158">RFC 4158</a>, where the client implementation needs to build valid paths and accept a connection if any path can be validated. The path building was initially in 0.5.2 slightly wrong, but fixed quickly in <a href="https://github.com/mirleft/ocaml-x509/commit/1a1476308d24bdcc49d45c4cd9ef539ca57461d2">0.5.3</a>.</p>
|
||||||
It turns out that the trust anchor sets are not equal across operating systems and versions, thus some web servers serve sets, instead of chains, of certificates - as described in [RFC 4158](https://tools.ietf.org/html/rfc4158), where the client implementation needs to build valid paths and accept a connection if any path can be validated. The path building was initially in 0.5.2 slightly wrong, but fixed quickly in [0.5.3](https://github.com/mirleft/ocaml-x509/commit/1a1476308d24bdcc49d45c4cd9ef539ca57461d2).
|
<h3 id="fingerprint-authentication">Fingerprint authentication</h3>
|
||||||
|
<p>The chain of trust validation is useful for the open web, where you as software developer don't know to which remote endpoint your software will ever connect to - as long as the remote has a certificate signed (via intermediates) by any of the trust anchors. In the early days, before <a href="https://letsencrypt.org/">let's encrypt</a> was launched and embedded as trust anchors (or cross-signed by already deployed trust anchors), operators needed to pay for a certificate - a business model where some CAs did not bother to check the authenticity of a certificate signing request, and thus random people owning valid certificates for microsoft.com or google.com.</p>
|
||||||
### Fingerprint authentication
|
<p>Instead of using the set of trust anchors, the fingerprint of the server certificate, or preferably the fingerprint of the public key of the certificate, can be used for authentication, as optionally done since some years in <a href="https://github.com/hannesm/jackline/commit/a1e6f3159be1e45e6b690845e1b29366c41239a2">jackline</a>, an XMPP client. Support for this certificate / public key pinning was added in x509 0.2.1 / 0.5.0.</p>
|
||||||
|
<h3 id="certificate-signing-requests">Certificate signing requests</h3>
|
||||||
The chain of trust validation is useful for the open web, where you as software developer don't know to which remote endpoint your software will ever connect to - as long as the remote has a certificate signed (via intermediates) by any of the trust anchors. In the early days, before [let's encrypt](https://letsencrypt.org/) was launched and embedded as trust anchors (or cross-signed by already deployed trust anchors), operators needed to pay for a certificate - a business model where some CAs did not bother to check the authenticity of a certificate signing request, and thus random people owning valid certificates for microsoft.com or google.com.
|
<p>Until x509 0.4.0 there was no support for generating certificate signing requests (CSR), as defined in PKCS 10, which are self-signed blobs containing a public key, an identity, and possibly extensions. Such as CSR is sent to the certificate authority, and after validation of ownership of the identity and paying a fee, the certificate is issued. Let's encrypt specified the ACME protocol which automates the proof of ownership: they provide a HTTP API for requesting a challenge, providing the response (the proof of ownership) via HTTP or DNS, and then allow the submission of a CSR and downloading the signed certificate. The ocaml-x509 library provides operations for creating such a CSR, and also for signing a CSR to generate a certificate.</p>
|
||||||
|
<p>Mindy developed the command-line utility <a href="https://github.com/yomimono/ocaml-certify/">certify</a> which uses these operations from the ocaml-x509 library and acts as a swiss-army knife purely in OCaml for these required operations.</p>
|
||||||
Instead of using the set of trust anchors, the fingerprint of the server certificate, or preferably the fingerprint of the public key of the certificate, can be used for authentication, as optionally done since some years in [jackline](https://github.com/hannesm/jackline/commit/a1e6f3159be1e45e6b690845e1b29366c41239a2), an XMPP client. Support for this certificate / public key pinning was added in x509 0.2.1 / 0.5.0.
|
<p>Maker developed a <a href="https://github.com/mmaker/ocaml-letsencrypt">let's encrypt library</a> which implements the above mentioned ACME protocol for provisioning CSR to certificates, also using our ocaml-x509 library.</p>
|
||||||
|
<p>To complete the required certificate authority functionality, in x509 0.6.0 certificate revocation lists, both validation and signing, was implemented.</p>
|
||||||
### Certificate signing requests
|
<h3 id="deploying-unikernels">Deploying unikernels</h3>
|
||||||
|
<p>As <a href="/Posts/VMM">described in another post</a>, I developed <a href="https://github.com/hannesm/albatross">albatross</a>, an orchestration system for MirageOS unikernels. This uses ASN.1 for internal socket communication and allows remote management via a TLS connection which is mutually authenticated with a X.509 client certificate. To encrypt the X.509 client certificate, first a TLS handshake where the server authenticates itself to the client is established, and over that connection another TLS handshake is established where the client certificate is requested. Note that this mechanism can be dropped with TLS 1.3, since there the certificates are transmitted over an already encrypted channel.</p>
|
||||||
Until x509 0.4.0 there was no support for generating certificate signing requests (CSR), as defined in PKCS 10, which are self-signed blobs containing a public key, an identity, and possibly extensions. Such as CSR is sent to the certificate authority, and after validation of ownership of the identity and paying a fee, the certificate is issued. Let's encrypt specified the ACME protocol which automates the proof of ownership: they provide a HTTP API for requesting a challenge, providing the response (the proof of ownership) via HTTP or DNS, and then allow the submission of a CSR and downloading the signed certificate. The ocaml-x509 library provides operations for creating such a CSR, and also for signing a CSR to generate a certificate.
|
<p>The client certificate already contains the command to execute remotely - as a custom extension, being it "show me the console output", or "destroy the unikernel with name = YYY", or "deploy the included unikernel image". The advantage is that the commands are already authenticated, and there is no need for developing an ad-hoc protocol on top of the TLS session. The resource limits, assigned by the authority, are also part of the certificate chain - i.e. the number of unikernels, access to network bridges, available accumulated memory, accumulated size for block devices, are constrained by the certificate chain presented to the server, and currently running unikernels. The names of the chain are used for access control - if Alice and Bob have intermediate certificates from the same CA, neither Alice may manage Bob's unikernels, nor Bob may manage Alice's unikernels. I'm using albatross since 2.5 years in production on two physical machines with ~20 unikernels total (multiple users, multiple administrative domains), and it works stable and is much nicer to deal with than <code>scp</code> and custom hacked shell scripts.</p>
|
||||||
|
<h2 id="why-0.7">Why 0.7?</h2>
|
||||||
Mindy developed the command-line utility [certify](https://github.com/yomimono/ocaml-certify/) which uses these operations from the ocaml-x509 library and acts as a swiss-army knife purely in OCaml for these required operations.
|
<p>There are still some missing pieces in our ocaml-x509 implementation, namely modern ECC certificates (depending on elliptic curve primitives not yet available in OCaml), RSA-PSS signing (should be straightforward), PKCS 12 (there is a <a href="https://github.com/mirleft/ocaml-x509/pull/114">pull request</a>, but this should wait until asn1-combinators supports the <code>ANY defined BY</code> construct to cleanup the code), ...
|
||||||
|
Once these features are supported, the library should likely be named PKCS since it supports more than X.509, and released as 1.0.</p>
|
||||||
Maker developed a [let's encrypt library](https://github.com/mmaker/ocaml-letsencrypt) which implements the above mentioned ACME protocol for provisioning CSR to certificates, also using our ocaml-x509 library.
|
<p>The 0.7 release series moved a lot of modules and function names around, thus it is a major breaking release. By using a map instead of lists for extensions, GeneralName, ..., the API was further revised - invariants that each extension key (an ASN.1 object identifier) may occur at most once are now enforced. By not leaking exceptions through the public interface, the API is easier to use safely - see <a href="https://github.com/mmaker/ocaml-letsencrypt/commit/dc53518f46310f384c9526b1d96a8e8f815a09c7">let's encrypt</a>, <a href="https://git.robur.io/?p=openvpn.git;a=commitdiff;h=929c53116c1438ba1214f53df7506d32da566ccc">openvpn</a>, <a href="https://github.com/yomimono/ocaml-certify/pull/17">certify</a>, <a href="https://github.com/mirleft/ocaml-tls/pull/394">tls</a>, <a href="https://github.com/mirage/capnp-rpc/pull/158">capnp</a>, <a href="https://github.com/hannesm/albatross/commit/50ed6a8d1ead169b3e322aaccb469e870ad72acc">albatross</a>.</p>
|
||||||
|
<p>I intended in 0.7.0 to have much more precise types, esp. for the SubjectAlternativeName (SAN) extension that uses a GeneralName, but it turns out the GeneralName is as well used for NameConstraints (NC) in a different way -- IP in SAN is an IPv4 or IPv6 address, in CN it is the IP/netmask; DNS is a domain name in SAN, in CN it is a name starting with a leading dot (i.e. ".example.com"), which is not a valid domain name. In 0.7.1, based on a bug report, I had to revert these variants and use less precise types.</p>
|
||||||
To complete the required certificate authority functionality, in x509 0.6.0 certificate revocation lists, both validation and signing, was implemented.
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>The work on X.509 was sponsored by <a href="http://ocamllabs.io/">OCaml Labs</a>. You can support our work at robur by a <a href="https://robur.io/Donate">donation</a>, which we will use to work on our OCaml and MirageOS projects. You can also reach out to us to realize commercial products.</p>
|
||||||
### Deploying unikernels
|
<p>I'm interested in feedback, either via <strike><a href="https://twitter.com/h4nnes">twitter</a></strike> <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
As [described in another post](/Posts/VMM), I developed [albatross](https://github.com/hannesm/albatross), an orchestration system for MirageOS unikernels. This uses ASN.1 for internal socket communication and allows remote management via a TLS connection which is mutually authenticated with a X.509 client certificate. To encrypt the X.509 client certificate, first a TLS handshake where the server authenticates itself to the client is established, and over that connection another TLS handshake is established where the client certificate is requested. Note that this mechanism can be dropped with TLS 1.3, since there the certificates are transmitted over an already encrypted channel.
|
|
||||||
|
|
||||||
The client certificate already contains the command to execute remotely - as a custom extension, being it "show me the console output", or "destroy the unikernel with name = YYY", or "deploy the included unikernel image". The advantage is that the commands are already authenticated, and there is no need for developing an ad-hoc protocol on top of the TLS session. The resource limits, assigned by the authority, are also part of the certificate chain - i.e. the number of unikernels, access to network bridges, available accumulated memory, accumulated size for block devices, are constrained by the certificate chain presented to the server, and currently running unikernels. The names of the chain are used for access control - if Alice and Bob have intermediate certificates from the same CA, neither Alice may manage Bob's unikernels, nor Bob may manage Alice's unikernels. I'm using albatross since 2.5 years in production on two physical machines with ~20 unikernels total (multiple users, multiple administrative domains), and it works stable and is much nicer to deal with than `scp` and custom hacked shell scripts.
|
|
||||||
|
|
||||||
## Why 0.7?
|
|
||||||
|
|
||||||
There are still some missing pieces in our ocaml-x509 implementation, namely modern ECC certificates (depending on elliptic curve primitives not yet available in OCaml), RSA-PSS signing (should be straightforward), PKCS 12 (there is a [pull request](https://github.com/mirleft/ocaml-x509/pull/114), but this should wait until asn1-combinators supports the `ANY defined BY` construct to cleanup the code), ...
|
|
||||||
Once these features are supported, the library should likely be named PKCS since it supports more than X.509, and released as 1.0.
|
|
||||||
|
|
||||||
The 0.7 release series moved a lot of modules and function names around, thus it is a major breaking release. By using a map instead of lists for extensions, GeneralName, ..., the API was further revised - invariants that each extension key (an ASN.1 object identifier) may occur at most once are now enforced. By not leaking exceptions through the public interface, the API is easier to use safely - see [let's encrypt](https://github.com/mmaker/ocaml-letsencrypt/commit/dc53518f46310f384c9526b1d96a8e8f815a09c7), [openvpn](https://git.robur.io/?p=openvpn.git;a=commitdiff;h=929c53116c1438ba1214f53df7506d32da566ccc), [certify](https://github.com/yomimono/ocaml-certify/pull/17), [tls](https://github.com/mirleft/ocaml-tls/pull/394), [capnp](https://github.com/mirage/capnp-rpc/pull/158), [albatross](https://github.com/hannesm/albatross/commit/50ed6a8d1ead169b3e322aaccb469e870ad72acc).
|
|
||||||
|
|
||||||
I intended in 0.7.0 to have much more precise types, esp. for the SubjectAlternativeName (SAN) extension that uses a GeneralName, but it turns out the GeneralName is as well used for NameConstraints (NC) in a different way -- IP in SAN is an IPv4 or IPv6 address, in CN it is the IP/netmask; DNS is a domain name in SAN, in CN it is a name starting with a leading dot (i.e. ".example.com"), which is not a valid domain name. In 0.7.1, based on a bug report, I had to revert these variants and use less precise types.
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
The work on X.509 was sponsored by [OCaml Labs](http://ocamllabs.io/). You can support our work at robur by a [donation](https://robur.io/Donate), which we will use to work on our OCaml and MirageOS projects. You can also reach out to us to realize commercial products.
|
|
||||||
|
|
||||||
I'm interested in feedback, either via <strike>[twitter](https://twitter.com/h4nnes)</strike> [hannesm@mastodon.social](https://mastodon.social/@hannesm) or via eMail.
|
|
||||||
30
Posts/index.html
Normal file
30
Posts/index.html
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Retreat2024" class="list-group-item"><h2 class="list-group-item-heading">The MirageOS retreat 2024</h2><span class="author">Written by hannes</span> <time>2024-05-15</time><br/><p class="list-group-item-text abstract"><p>My involvement and experience of the MirageOS retreat 2024</p>
|
||||||
|
</p></a><a href="/Posts/TCP-ns" class="list-group-item"><h2 class="list-group-item-heading">Redeveloping TCP from the ground up</h2><span class="author">Written by hannes</span> <time>2023-11-28</time><br/><p class="list-group-item-text abstract"><p>Core Internet protocols require operational experiments, even if formally specified</p>
|
||||||
|
</p></a><a href="/Posts/Albatross" class="list-group-item"><h2 class="list-group-item-heading">Deploying reproducible unikernels with albatross</h2><span class="author">Written by hannes</span> <time>2022-11-17</time><br/><p class="list-group-item-text abstract"><p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
|
||||||
|
</p></a><a href="/Posts/OpamMirror" class="list-group-item"><h2 class="list-group-item-heading">Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span> <time>2022-09-29</time><br/><p class="list-group-item-text abstract"><p>Re-developing an opam cache from scratch, as a MirageOS unikernel</p>
|
||||||
|
</p></a><a href="/Posts/Monitoring" class="list-group-item"><h2 class="list-group-item-heading">All your metrics belong to influx</h2><span class="author">Written by hannes</span> <time>2022-03-08</time><br/><p class="list-group-item-text abstract"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||||
|
</p></a><a href="/Posts/Deploy" class="list-group-item"><h2 class="list-group-item-heading">Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2021-06-30</time><br/><p class="list-group-item-text abstract"><p>Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder</p>
|
||||||
|
</p></a><a href="/Posts/EC" class="list-group-item"><h2 class="list-group-item-heading">Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span> <time>2021-04-23</time><br/><p class="list-group-item-text abstract"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||||
|
</p></a><a href="/Posts/NGI" class="list-group-item"><h2 class="list-group-item-heading">The road ahead for MirageOS in 2021</h2><span class="author">Written by hannes</span> <time>2021-01-25</time><br/><p class="list-group-item-text abstract"><p>Home office, MirageOS unikernels, 2020 recap, 2021 tbd</p>
|
||||||
|
</p></a><a href="/Posts/Traceroute" class="list-group-item"><h2 class="list-group-item-heading">Traceroute</h2><span class="author">Written by hannes</span> <time>2020-06-24</time><br/><p class="list-group-item-text abstract"><p>A MirageOS unikernel which traces the path between itself and a remote host.</p>
|
||||||
|
</p></a><a href="/Posts/DnsServer" class="list-group-item"><h2 class="list-group-item-heading">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2019-12-23</time><br/><p class="list-group-item-text abstract"><p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
|
||||||
|
</p></a><a href="/Posts/ReproducibleOPAM" class="list-group-item"><h2 class="list-group-item-heading">Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span> <time>2019-12-16</time><br/><p class="list-group-item-text abstract"><p>MirageOS unikernels are reproducible :)</p>
|
||||||
|
</p></a><a href="/Posts/X50907" class="list-group-item"><h2 class="list-group-item-heading">X509 0.7</h2><span class="author">Written by hannes</span> <time>2019-08-15</time><br/><p class="list-group-item-text abstract"><p>Five years since ocaml-x509 initial release, it has been reworked and used more widely</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/Pinata" class="list-group-item"><h2 class="list-group-item-heading">The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span> <time>2018-04-18</time><br/><p class="list-group-item-text abstract"><p>More than three years ago we launched our Bitcoin Piñata as a transparent security bait. It is still up and running!</p>
|
||||||
|
</p></a><a href="/Posts/DNS" class="list-group-item"><h2 class="list-group-item-heading">My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span> <time>2018-01-11</time><br/><p class="list-group-item-text abstract"><p>New year brings new possibilities and a new environment. I've been working on the most Widely deployed key-value store, the domain name system. Primary and secondary name services are available, including dynamic updates, notify, and tsig authentication.</p>
|
||||||
|
</p></a><a href="/Posts/VMM" class="list-group-item"><h2 class="list-group-item-heading">Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span> <time>2017-07-10</time><br/><p class="list-group-item-text abstract"><p>all we need is X.509</p>
|
||||||
|
</p></a><a href="/Posts/Conex" class="list-group-item"><h2 class="list-group-item-heading">Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>Conex is a library to verify and attest package release integrity and authenticity through the use of cryptographic signatures.</p>
|
||||||
|
</p></a><a href="/Posts/Maintainers" class="list-group-item"><h2 class="list-group-item-heading">Who maintains package X?</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>We describe why manual gathering of metadata is out of date, and version control systems are awesome.</p>
|
||||||
|
</p></a><a href="/Posts/Jackline" class="list-group-item"><h2 class="list-group-item-heading">Jackline, a secure terminal-based XMPP client</h2><span class="author">Written by hannes</span> <time>2017-01-30</time><br/><p class="list-group-item-text abstract"><p>implement it once to know you can do it. implement it a second time and you get readable code. implementing it a third time from scratch may lead to useful libraries.</p>
|
||||||
|
</p></a><a href="/Posts/Syslog" class="list-group-item"><h2 class="list-group-item-heading">Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span> <time>2016-11-05</time><br/><p class="list-group-item-text abstract"><p>sometimes preservation of data is useful</p>
|
||||||
|
</p></a><a href="/Posts/ARP" class="list-group-item"><h2 class="list-group-item-heading">Re-engineering ARP</h2><span class="author">Written by hannes</span> <time>2016-07-12</time><br/><p class="list-group-item-text abstract"><p>If you want it as you like, you've to do it yourself</p>
|
||||||
|
</p></a><a href="/Posts/Solo5" class="list-group-item"><h2 class="list-group-item-heading">Minimising the virtual machine monitor</h2><span class="author">Written by hannes</span> <time>2016-07-02</time><br/><p class="list-group-item-text abstract"><p>MirageOS solo5 multiboot native on bhyve</p>
|
||||||
|
</p></a><a href="/Posts/BottomUp" class="list-group-item"><h2 class="list-group-item-heading">Counting Bytes</h2><span class="author">Written by hannes</span> <time>2016-06-11</time><br/><p class="list-group-item-text abstract"><p>looking into dependencies and their sizes</p>
|
||||||
|
</p></a><a href="/Posts/Functoria" class="list-group-item"><h2 class="list-group-item-heading">Configuration DSL step-by-step</h2><span class="author">Written by hannes</span> <time>2016-05-10</time><br/><p class="list-group-item-text abstract"><p>how to actually configure the system</p>
|
||||||
|
</p></a><a href="/Posts/BadRecordMac" class="list-group-item"><h2 class="list-group-item-heading">Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span> <time>2016-05-03</time><br/><p class="list-group-item-text abstract"><p>10BTC could've been yours</p>
|
||||||
|
</p></a><a href="/Posts/nqsbWebsite" class="list-group-item"><h2 class="list-group-item-heading">Fitting the things together</h2><span class="author">Written by hannes</span> <time>2016-04-24</time><br/><p class="list-group-item-text abstract"><p>building a simple website</p>
|
||||||
|
</p></a><a href="/Posts/OCaml" class="list-group-item"><h2 class="list-group-item-heading">Why OCaml</h2><span class="author">Written by hannes</span> <time>2016-04-17</time><br/><p class="list-group-item-text abstract"><p>a gentle introduction into OCaml</p>
|
||||||
|
</p></a><a href="/Posts/OperatingSystem" class="list-group-item"><h2 class="list-group-item-heading">Operating systems</h2><span class="author">Written by hannes</span> <time>2016-04-09</time><br/><p class="list-group-item-text abstract"><p>Operating systems and MirageOS</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
|
|
@ -1,271 +1,194 @@
|
||||||
---
|
<!DOCTYPE html>
|
||||||
title: Fitting the things together
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Fitting the things together</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Fitting the things together" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Fitting the things together</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/http" class="tag">http</a><a href="/tags/tls" class="tag">tls</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2016-04-24 (last updated: 2021-11-19)</span><article><h2 id="task">Task</h2>
|
||||||
author: hannes
|
<p>Our task is to build a small unikernel which provides a project website. On our way we will wade through various layers using code examples. The website itself contains a few paragraphs of text, some link lists, and our published papers in pdf form.</p>
|
||||||
tags: mirageos, http, tls, protocol
|
<p><em>Spoiler alert</em> final result can be seen <a href="https://nqsb.io">here</a>, the full code <a href="https://github.com/mirleft/nqsb.io">here</a>.</p>
|
||||||
abstract: building a simple website
|
<h2 id="a-first-idea">A first idea</h2>
|
||||||
---
|
<p>We could go all the way to use <a href="https://github.com/mirage/ocaml-conduit">conduit</a> for wrapping connections, and <a href="https://github.com/mirage/mirage-http">mirage-http</a> (using <a href="https://github.com/mirage/ocaml-cohttp">cohttp</a>, a very lightweight HTTP server). We'd just need to write routing code which in the end reads from a virtual file system, and some HTML and CSS for the actual site.</p>
|
||||||
|
<p>Turns out, the conduit library is already 1.7 MB in size and depends on 34 libraries, cohttp is another 3.7 MB and 40 dependent libraries.
|
||||||
## Task
|
Both libraries are actively developed, combined there were 25 releases within the last year.</p>
|
||||||
|
<h2 id="plan">Plan</h2>
|
||||||
Our task is to build a small unikernel which provides a project website. On our way we will wade through various layers using code examples. The website itself contains a few paragraphs of text, some link lists, and our published papers in pdf form.
|
<p>Let me state our demands more clearly:</p>
|
||||||
|
<ul>
|
||||||
*Spoiler alert* final result can be seen [here](https://nqsb.io), the full code [here](https://github.com/mirleft/nqsb.io).
|
<li>easy to maintain
|
||||||
|
</li>
|
||||||
## A first idea
|
<li>updates roughly 3 times a year
|
||||||
|
</li>
|
||||||
We could go all the way to use [conduit](https://github.com/mirage/ocaml-conduit) for wrapping connections, and [mirage-http](https://github.com/mirage/mirage-http) (using [cohttp](https://github.com/mirage/ocaml-cohttp), a very lightweight HTTP server). We'd just need to write routing code which in the end reads from a virtual file system, and some HTML and CSS for the actual site.
|
<li>reasonable performance
|
||||||
|
</li>
|
||||||
Turns out, the conduit library is already 1.7 MB in size and depends on 34 libraries, cohttp is another 3.7 MB and 40 dependent libraries.
|
</ul>
|
||||||
Both libraries are actively developed, combined there were 25 releases within the last year.
|
<p>To achieve easy maintenance we keep build and run time dependencies small, use a single virtual machine image to ease deployment. We try to develop only little new code. Our general approach to performance is to do as little work as we can on each request, and precompute at compile time or once at startup as much as we can.</p>
|
||||||
|
<h3 id="html-code">HTML code</h3>
|
||||||
## Plan
|
<p>From the <a href="http://ocsigen.org/tyxml/">tyxml description</a>: "Tyxml provides a set of combinators to build Html5 and Svg documents. These combinators use the OCaml type-system to ensure the validity of the generated Html5 and Svg." A <a href="http://ocsigen.org/tyxml/dev/manual/intro">tutorial</a> is available.</p>
|
||||||
|
<p>You can plug elements (or attributes) inside each other only if the HTML specification allows this (no <code><body></code> inside of a <code><body></code>). An example that can be rendered to a <code>div</code> with <code>pcdata</code> inside.</p>
|
||||||
Let me state our demands more clearly:
|
<p>If you use <code>utop</code> (as interactive read-eval-print-loop), you first need to load tyxml by <code>#require "tyxml"</code>.</p>
|
||||||
- easy to maintain
|
<pre><code class="language-OCaml">open Html5.M
|
||||||
- updates roughly 3 times a year
|
|
||||||
- reasonable performance
|
|
||||||
|
|
||||||
To achieve easy maintenance we keep build and run time dependencies small, use a single virtual machine image to ease deployment. We try to develop only little new code. Our general approach to performance is to do as little work as we can on each request, and precompute at compile time or once at startup as much as we can.
|
|
||||||
|
|
||||||
### HTML code
|
|
||||||
|
|
||||||
From the [tyxml description](http://ocsigen.org/tyxml/): "Tyxml provides a set of combinators to build Html5 and Svg documents. These combinators use the OCaml type-system to ensure the validity of the generated Html5 and Svg." A [tutorial](http://ocsigen.org/tyxml/dev/manual/intro) is available.
|
|
||||||
|
|
||||||
You can plug elements (or attributes) inside each other only if the HTML specification allows this (no `<body>` inside of a `<body>`). An example that can be rendered to a `div` with `pcdata` inside.
|
|
||||||
|
|
||||||
If you use `utop` (as interactive read-eval-print-loop), you first need to load tyxml by `#require "tyxml"`.
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
open Html5.M
|
|
||||||
|
|
||||||
let mycontent =
|
let mycontent =
|
||||||
div ~a:[ a_class ["content"] ]
|
div ~a:[ a_class ["content"] ]
|
||||||
[ pcdata "This is a fabulous content." ]
|
[ pcdata "This is a fabulous content." ]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>In the end, our web server will deliver the page as a string, tyxml provides the function <code>Html5.P.print : output:(string -> unit) -> doc -> unit</code>. We use a temporary <code>Buffer</code> to print the document into.</p>
|
||||||
In the end, our web server will deliver the page as a string, tyxml provides the function `Html5.P.print : output:(string -> unit) -> doc -> unit`. We use a temporary `Buffer` to print the document into.
|
<pre><code class="language-OCaml"># let buf = Buffer.create 100
|
||||||
|
|
||||||
```OCaml
|
|
||||||
# let buf = Buffer.create 100
|
|
||||||
# Html5.P.print ~output:(Buffer.add_string buf) mycontent
|
# Html5.P.print ~output:(Buffer.add_string buf) mycontent
|
||||||
|
|
||||||
Error: This expression has type ([> Html5_types.div ] as 'a) elt but an expression was expected of type doc = [ `Html ] elt Type 'a = [> `Div ] is not compatible with type [ `Html ]
|
Error: This expression has type ([> Html5_types.div ] as 'a) elt but an expression was expected of type doc = [ `Html ] elt Type 'a = [> `Div ] is not compatible with type [ `Html ]
|
||||||
The second variant type does not allow tag(s) `Div
|
The second variant type does not allow tag(s) `Div
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This is pretty nice, we can only print complete HTML5 documents this way (there are printers for standalone elements as well), and will not be able to serve an incomplete page fragment!</p>
|
||||||
This is pretty nice, we can only print complete HTML5 documents this way (there are printers for standalone elements as well), and will not be able to serve an incomplete page fragment!
|
<p>To get it up and running, we wrap it inside of a <code>html</code> which has a <code>header</code> and a <code>body</code>:</p>
|
||||||
|
<pre><code class="language-OCaml"># Html5.P.print ~output:(Buffer.add_string buf) (html (head (title (pcdata "title")) []) (body [ mycontent ]))
|
||||||
To get it up and running, we wrap it inside of a `html` which has a `header` and a `body`:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
# Html5.P.print ~output:(Buffer.add_string buf) (html (head (title (pcdata "title")) []) (body [ mycontent ]))
|
|
||||||
# Buffer.contents buf
|
# Buffer.contents buf
|
||||||
|
|
||||||
"<!DOCTYPE html>\n<html xmlns=\"http://www.w3.org/1999/xhtml\"><head><title>title</title></head><body><div class=\"content\">This is a fabulous content.</div></body></html>"
|
"<!DOCTYPE html>\n<html xmlns=\"http://www.w3.org/1999/xhtml\"><head><title>title</title></head><body><div class=\"content\">This is a fabulous content.</div></body></html>"
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The HTML content is done (in a pure way, no effects!), let's work on the binary pdfs.
|
||||||
The HTML content is done (in a pure way, no effects!), let's work on the binary pdfs.
|
Our full page source (CSS embedding is done via a string, no fancy types there (yet!?)) is <a href="https://github.com/mirleft/nqsb.io/blob/master/page.ml">on GitHub</a>. Below we will use the <code>render</code> function for our content:</p>
|
||||||
Our full page source (CSS embedding is done via a string, no fancy types there (yet!?)) is [on GitHub](https://github.com/mirleft/nqsb.io/blob/master/page.ml). Below we will use the `render` function for our content:
|
<pre><code class="language-OCaml">let render =
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let render =
|
|
||||||
let buf = Buffer.create 500 in
|
let buf = Buffer.create 500 in
|
||||||
Html5.P.print ~output:(Buffer.add_string buf) @@
|
Html5.P.print ~output:(Buffer.add_string buf) @@
|
||||||
html
|
html
|
||||||
(header "not quite so broken")
|
(header "not quite so broken")
|
||||||
(body [ mycontent ]) ;
|
(body [ mycontent ]) ;
|
||||||
Cstruct.of_string @@ Buffer.contents buf
|
Cstruct.of_string @@ Buffer.contents buf
|
||||||
```
|
</code></pre>
|
||||||
|
<h3 id="binary-data">Binary data</h3>
|
||||||
### Binary data
|
<p>There are various ways how to embed binary data into MirageOS:</p>
|
||||||
|
<ul>
|
||||||
There are various ways how to embed binary data into MirageOS:
|
<li>connect an external (FAT) disk image; upside: works for large data, independent, can be shared with other systems; downside: an extra file to distribute onto the production machine, lots of code (block storage and file system access) which can contain directory traversals and other issues
|
||||||
- connect an external (FAT) disk image; upside: works for large data, independent, can be shared with other systems; downside: an extra file to distribute onto the production machine, lots of code (block storage and file system access) which can contain directory traversals and other issues
|
</li>
|
||||||
- embed as a [special ELF section](https://github.com/mirage/mirage/issues/489); downside: not yet implemented
|
<li>embed as a <a href="https://github.com/mirage/mirage/issues/489">special ELF section</a>; downside: not yet implemented
|
||||||
- embed strings in the code; upside: no deployment hassle, works everywhere for small files; downside: need to encode binary to string chunks during build and decode in the MirageOS unikernel, [it breaks with large files](https://github.com/mirage/mirage/issues/396)
|
</li>
|
||||||
- likely others such as use a tar image, read it during build or runtime; wait during bootup on a network socket (or connect somewhere) to receive the static data; use a remote git repository
|
<li>embed strings in the code; upside: no deployment hassle, works everywhere for small files; downside: need to encode binary to string chunks during build and decode in the MirageOS unikernel, <a href="https://github.com/mirage/mirage/issues/396">it breaks with large files</a>
|
||||||
|
</li>
|
||||||
We'll use the embedding. There is support for this built in the mirage tool via [crunch](https://github.com/mirage/ocaml-crunch). If you `crunch "foo"` in `config.ml`, it will create a [read-only key value store](https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L931) (`KV_RO`) named `static1.ml` containing everything in the local `"foo"` directory during `mirage configure`.
|
<li>likely others such as use a tar image, read it during build or runtime; wait during bootup on a network socket (or connect somewhere) to receive the static data; use a remote git repository
|
||||||
Each file is encoded as a list of chunks, each 4096 bytes in size in ASCII (octal escaping `\000` of control characters).
|
</li>
|
||||||
|
</ul>
|
||||||
The API is:
|
<p>We'll use the embedding. There is support for this built in the mirage tool via <a href="https://github.com/mirage/ocaml-crunch">crunch</a>. If you <code>crunch "foo"</code> in <code>config.ml</code>, it will create a <a href="https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L931">read-only key value store</a> (<code>KV_RO</code>) named <code>static1.ml</code> containing everything in the local <code>"foo"</code> directory during <code>mirage configure</code>.
|
||||||
|
Each file is encoded as a list of chunks, each 4096 bytes in size in ASCII (octal escaping <code>\000</code> of control characters).</p>
|
||||||
```OCaml
|
<p>The API is:</p>
|
||||||
val read : t -> string -> int -> int -> [ `Ok of page_aligned_buffer list | `UnknownKey of string ]
|
<pre><code class="language-OCaml">val read : t -> string -> int -> int -> [ `Ok of page_aligned_buffer list | `UnknownKey of string ]
|
||||||
val size : t -> string -> [ `Ok of int64 | `UnknownKey of string ]
|
val size : t -> string -> [ `Ok of int64 | `UnknownKey of string ]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The lookup needs to retrieve first the chunk list for the given filename, and then splice the fragments together and return the requested offset and length as page aligned structures. Since this is not for free, we will read our pdfs only once during startup, and then keep a reference to the full pdfs which we deliver upon request:</p>
|
||||||
The lookup needs to retrieve first the chunk list for the given filename, and then splice the fragments together and return the requested offset and length as page aligned structures. Since this is not for free, we will read our pdfs only once during startup, and then keep a reference to the full pdfs which we deliver upon request:
|
<pre><code class="language-OCaml">let read_full_kv kv name =
|
||||||
|
KV.size kv name >>= function
|
||||||
```OCaml
|
| `Error (KV.Unknown_key e) -> Lwt.fail (Invalid_argument e)
|
||||||
let read_full_kv kv name =
|
| `Ok size ->
|
||||||
KV.size kv name >>= function
|
KV.read kv name 0 (Int64.to_int size) >>= function
|
||||||
| `Error (KV.Unknown_key e) -> Lwt.fail (Invalid_argument e)
|
| `Error (KV.Unknown_key e) -> Lwt.fail (Invalid_argument e)
|
||||||
| `Ok size ->
|
| `Ok bufs -> Lwt.return (Cstruct.concat bufs)
|
||||||
KV.read kv name 0 (Int64.to_int size) >>= function
|
|
||||||
| `Error (KV.Unknown_key e) -> Lwt.fail (Invalid_argument e)
|
|
||||||
| `Ok bufs -> Lwt.return (Cstruct.concat bufs)
|
|
||||||
|
|
||||||
let start kv =
|
let start kv =
|
||||||
let d_nqsb = Page.render in
|
let d_nqsb = Page.render in
|
||||||
read_full_kv kv "nqsbtls-usenix-security15.pdf" >>= fun d_usenix ->
|
read_full_kv kv "nqsbtls-usenix-security15.pdf" >>= fun d_usenix ->
|
||||||
read_full_kv kv "tron.pdf" >>= fun d_tron ->
|
read_full_kv kv "tron.pdf" >>= fun d_tron ->
|
||||||
...
|
...
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The funny <code>>>=</code> syntax notes that something is doing input/output, which might be blocking or interrupted or failing. It composes effects using an imperative style (semicolon in other languages, another term is monadic bind). The <code>Page.render</code> function is described above, and is pure, thus no <code>>>=</code>.</p>
|
||||||
The funny `>>=` syntax notes that something is doing input/output, which might be blocking or interrupted or failing. It composes effects using an imperative style (semicolon in other languages, another term is monadic bind). The `Page.render` function is described above, and is pure, thus no `>>=`.
|
<p>We now have all the resources we wanted available inside our MirageOS unikernel. There is some cost during configuration (converting binary into code), and startup (concatenating lists, lookups, rendering HTML into string representation).</p>
|
||||||
|
<h3 id="building-a-http-response">Building a HTTP response</h3>
|
||||||
We now have all the resources we wanted available inside our MirageOS unikernel. There is some cost during configuration (converting binary into code), and startup (concatenating lists, lookups, rendering HTML into string representation).
|
<p>HTTP consists of headers and data, we already have the data. A HTTP header contains of an initial status line (<code>HTTP/1.1 200 OK</code>), and a list of keys-values, each of the form <code>key + ": " + value + "\r\n"</code> (<code>+</code> is string concatenation). The header is separated with <code>"\r\n\r\n"</code> from the data:</p>
|
||||||
|
<pre><code class="language-OCaml">let http_header ~status xs =
|
||||||
### Building a HTTP response
|
let headers = List.map (fun (k, v) -> k ^ ": " ^ v) xs in
|
||||||
|
let lines = status :: headers @ [ "\r\n" ] in
|
||||||
HTTP consists of headers and data, we already have the data. A HTTP header contains of an initial status line (`HTTP/1.1 200 OK`), and a list of keys-values, each of the form `key + ": " + value + "\r\n"` (`+` is string concatenation). The header is separated with `"\r\n\r\n"` from the data:
|
String.concat "\r\n" lines
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let http_header ~status xs =
|
|
||||||
let headers = List.map (fun (k, v) -> k ^ ": " ^ v) xs in
|
|
||||||
let lines = status :: headers @ [ "\r\n" ] in
|
|
||||||
String.concat "\r\n" lines
|
|
||||||
|
|
||||||
let header content_type =
|
let header content_type =
|
||||||
http_header ~status:"HTTP/1.1 200 OK" [ ("content-type", content_type) ]
|
http_header ~status:"HTTP/1.1 200 OK" [ ("content-type", content_type) ]
|
||||||
```
|
</code></pre>
|
||||||
|
<p>We also know statically (at compile time) which headers to send: <code>content-type</code> should be <code>text/html</code> for our main page, and <code>application/pdf</code> for the pdf files. The status code <code>200</code> is used in HTTP to signal that the request is successful. We can combine the headers and the data during startup, because our single communication channel is HTTP, and thus we don't need access to the data or headers separately (support for HTTP caching etc. is out of scope).</p>
|
||||||
We also know statically (at compile time) which headers to send: `content-type` should be `text/html` for our main page, and `application/pdf` for the pdf files. The status code `200` is used in HTTP to signal that the request is successful. We can combine the headers and the data during startup, because our single communication channel is HTTP, and thus we don't need access to the data or headers separately (support for HTTP caching etc. is out of scope).
|
<p>We are now finished with the response side of HTTP, and can emit three different resources. Now, we need to handle incoming HTTP requests and dispatch them to the resource. Let's first to a brief detour to HTTPS (and thus TLS).</p>
|
||||||
|
<h3 id="security-via-https">Security via HTTPS</h3>
|
||||||
We are now finished with the response side of HTTP, and can emit three different resources. Now, we need to handle incoming HTTP requests and dispatch them to the resource. Let's first to a brief detour to HTTPS (and thus TLS).
|
<p>Transport layer security is a protocol on top of TCP providing an end-to-end encrypted and authenticated channel. In our setting, our web server has a certificate and a private key to authenticate itself to the clients.</p>
|
||||||
|
<p>A certificate is a token containing a public key, a name, a validity period, and a signature from the authority which issued the certificate. The authority is crucial here: this infrastructure only works if the client trusts the public key of the authority (and thus can verify their signature on our certificate). I used <a href="https://letsencrypt.org/">let's encrypt</a> (actually the <a href="https://github.com/lukas2511/letsencrypt.sh/">letsencrypt.sh</a> client (would be great to have one natively in OCaml) to get a signed certificate, which is widely accepted by web browsers.</p>
|
||||||
### Security via HTTPS
|
<p>The MirageOS interface for TLS is that it takes a <a href="https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L108"><code>FLOW</code></a> (byte stream, e.g. TCP) and provides a <code>FLOW</code>. Libraries can be written to be agnostic whether they use a TCP stream or a TLS session to carry data.</p>
|
||||||
|
<p>We need to setup our unikernel that on new connections to the HTTPS port, it should first do a TLS handshake, and afterwards talk the HTTP protocol. For a TLS handshake we need to put the certificate and private key into the unikernel, using yet another key value store (<code>crunch "tls"</code>).</p>
|
||||||
Transport layer security is a protocol on top of TCP providing an end-to-end encrypted and authenticated channel. In our setting, our web server has a certificate and a private key to authenticate itself to the clients.
|
<p>On startup we read the certificate and private key, and use that to create a <a href="https://mirleft.github.io/ocaml-tls/Config.html#VALserver">TLS server config</a>:</p>
|
||||||
|
<pre><code class="language-OCaml">let start stack kv keys =
|
||||||
A certificate is a token containing a public key, a name, a validity period, and a signature from the authority which issued the certificate. The authority is crucial here: this infrastructure only works if the client trusts the public key of the authority (and thus can verify their signature on our certificate). I used [let's encrypt](https://letsencrypt.org/) (actually the [letsencrypt.sh](https://github.com/lukas2511/letsencrypt.sh/) client (would be great to have one natively in OCaml) to get a signed certificate, which is widely accepted by web browsers.
|
read_cert keys "nqsb" >>= fun c_nqsb ->
|
||||||
|
|
||||||
The MirageOS interface for TLS is that it takes a [`FLOW`](https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L108) (byte stream, e.g. TCP) and provides a `FLOW`. Libraries can be written to be agnostic whether they use a TCP stream or a TLS session to carry data.
|
|
||||||
|
|
||||||
We need to setup our unikernel that on new connections to the HTTPS port, it should first do a TLS handshake, and afterwards talk the HTTP protocol. For a TLS handshake we need to put the certificate and private key into the unikernel, using yet another key value store (`crunch "tls"`).
|
|
||||||
|
|
||||||
On startup we read the certificate and private key, and use that to create a [TLS server config](https://mirleft.github.io/ocaml-tls/Config.html#VALserver):
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let start stack kv keys =
|
|
||||||
read_cert keys "nqsb" >>= fun c_nqsb ->
|
|
||||||
let config = Tls.Config.server ~certificates:(`Single c_nqsb) () in
|
let config = Tls.Config.server ~certificates:(`Single c_nqsb) () in
|
||||||
S.listen_tcpv4 stack ~port:443 (tls_accept config)
|
S.listen_tcpv4 stack ~port:443 (tls_accept config)
|
||||||
```
|
</code></pre>
|
||||||
|
<p>The <code>listen_tcpv4</code> is provided by the <a href="https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L754">stackv4 module</a>, and gets a stack, a port number (HTTPS uses 443), and a function which receives a flow instance.</p>
|
||||||
The `listen_tcpv4` is provided by the [stackv4 module](https://github.com/mirage/mirage/blob/54736660606ca06aad1a061ac4276cc45ead1815/types/V1.mli#L754), and gets a stack, a port number (HTTPS uses 443), and a function which receives a flow instance.
|
<pre><code class="language-OCaml">let tls_accept cfg tcp =
|
||||||
|
TLS.server_of_flow cfg tcp >>= function
|
||||||
```OCaml
|
| `Error _ | `Eof -> TCP.close tcp
|
||||||
let tls_accept cfg tcp =
|
| `Ok tls -> ...
|
||||||
TLS.server_of_flow cfg tcp >>= function
|
</code></pre>
|
||||||
| `Error _ | `Eof -> TCP.close tcp
|
<p>The <code>server_of_flow</code> is provided by the <a href="https://mirleft.github.io/ocaml-tls/Tls_mirage.Make.html#VALserver_of_flow">Mirage TLS layer</a>. It can fail, if the client and server do not speak a common protocol suite, ciphersuite, or if one of the sides behaves non-protocol compliant.</p>
|
||||||
| `Ok tls -> ...
|
<p>To wrap up, we managed to listen on the HTTPS port and establish TLS sessions for each incoming request. We have our resources available, and now need to dispatch the request onto the resource.</p>
|
||||||
```
|
<h3 id="http-request-handling">HTTP request handling</h3>
|
||||||
|
<p>HTTP is a string based protocol, the first line of the request contains the method and resource which the client wants to access, <code>GET / HTTP/1.1</code>. We could read a single line from the client, and cut the part between <code>GET</code> and <code>HTTP/1.1</code> to deliver the resource.</p>
|
||||||
The `server_of_flow` is provided by the [Mirage TLS layer](https://mirleft.github.io/ocaml-tls/Tls_mirage.Make.html#VALserver_of_flow). It can fail, if the client and server do not speak a common protocol suite, ciphersuite, or if one of the sides behaves non-protocol compliant.
|
<p>This would either need some (brittle?) string processing, or a full-blown HTTP library on our side. "I'm sorry Dave, I can't do that". There is no way we'll do string processing on data received from the network for this.</p>
|
||||||
|
<p>Looking a bit deeper into TLS, there is a specification for <a href="https://tools.ietf.org/html/rfc3546#section-3.1">server name indication</a> from 2003. The main purpose is to run on a single IP(v4) address multiple TLS services. The client indicates in the TLS handshake what the server name is it wants to talk to. This extension looks <a href="https://en.wikipedia.org/wiki/Server_Name_Indication">in wikipedia</a> widely enough deployed.</p>
|
||||||
To wrap up, we managed to listen on the HTTPS port and establish TLS sessions for each incoming request. We have our resources available, and now need to dispatch the request onto the resource.
|
<p>During the TLS handshake there is already some server name information exposed, and we have a very small set of available resources. Thanks to let's encrypt, generating certificates is easy and free of cost.</p>
|
||||||
|
<p>And, if we're down to a single resource, we can use the same technique used by <a href="https://github.com/pqwy">David</a> in
|
||||||
### HTTP request handling
|
the <a href="http://ownme.ipredator.se">BTC Piñata</a>: just send the resource back without waiting for a request.</p>
|
||||||
|
<h3 id="putting-it-all-together">Putting it all together</h3>
|
||||||
HTTP is a string based protocol, the first line of the request contains the method and resource which the client wants to access, `GET / HTTP/1.1`. We could read a single line from the client, and cut the part between `GET` and `HTTP/1.1` to deliver the resource.
|
<p>What we need is a hostname for each resource, and certificates and private keys for them, or a single certificate with all hostnames as alternative names.</p>
|
||||||
|
<p>Our TLS library supports to select a certificate chain based on the requested name (look <a href="https://mirleft.github.io/ocaml-tls/Config.html#TYPEown_cert">here</a>). The following snippet is a setup to use the <code>nqsb.io</code> certificate chain by default (if no SNI is provided, or none matches), and also have a <code>usenix15</code> and a <code>tron</code> certificate chain.</p>
|
||||||
This would either need some (brittle?) string processing, or a full-blown HTTP library on our side. "I'm sorry Dave, I can't do that". There is no way we'll do string processing on data received from the network for this.
|
<pre><code class="language-OCaml">let start stack keys kv =
|
||||||
|
read_cert keys "nqsb" >>= fun c_nqsb ->
|
||||||
Looking a bit deeper into TLS, there is a specification for [server name indication](https://tools.ietf.org/html/rfc3546#section-3.1) from 2003. The main purpose is to run on a single IP(v4) address multiple TLS services. The client indicates in the TLS handshake what the server name is it wants to talk to. This extension looks [in wikipedia](https://en.wikipedia.org/wiki/Server_Name_Indication) widely enough deployed.
|
read_cert keys "usenix15" >>= fun c_usenix ->
|
||||||
|
read_cert keys "tron" >>= fun c_tron ->
|
||||||
During the TLS handshake there is already some server name information exposed, and we have a very small set of available resources. Thanks to let's encrypt, generating certificates is easy and free of cost.
|
|
||||||
|
|
||||||
And, if we're down to a single resource, we can use the same technique used by [David](https://github.com/pqwy) in
|
|
||||||
the [BTC Piñata](http://ownme.ipredator.se): just send the resource back without waiting for a request.
|
|
||||||
|
|
||||||
### Putting it all together
|
|
||||||
|
|
||||||
What we need is a hostname for each resource, and certificates and private keys for them, or a single certificate with all hostnames as alternative names.
|
|
||||||
|
|
||||||
Our TLS library supports to select a certificate chain based on the requested name (look [here](https://mirleft.github.io/ocaml-tls/Config.html#TYPEown_cert)). The following snippet is a setup to use the `nqsb.io` certificate chain by default (if no SNI is provided, or none matches), and also have a `usenix15` and a `tron` certificate chain.
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let start stack keys kv =
|
|
||||||
read_cert keys "nqsb" >>= fun c_nqsb ->
|
|
||||||
read_cert keys "usenix15" >>= fun c_usenix ->
|
|
||||||
read_cert keys "tron" >>= fun c_tron ->
|
|
||||||
let config = Tls.Config.server ~certificates:(`Multiple_default (c_nqsb, [ c_usenix ; c_tron])) () in
|
let config = Tls.Config.server ~certificates:(`Multiple_default (c_nqsb, [ c_usenix ; c_tron])) () in
|
||||||
S.listen_tcpv4 stack ~port:443 (tls_accept config) ;
|
S.listen_tcpv4 stack ~port:443 (tls_accept config) ;
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Back to dispatching code. We can extract the hostname information from an opaque <code>tls</code> value (the epoch data is fully described <a href="https://mirleft.github.io/ocaml-tls/Core.html#TYPEepoch_data">here</a>:</p>
|
||||||
Back to dispatching code. We can extract the hostname information from an opaque `tls` value (the epoch data is fully described [here](https://mirleft.github.io/ocaml-tls/Core.html#TYPEepoch_data):
|
<pre><code class="language-OCaml">let extract_name tls =
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let extract_name tls =
|
|
||||||
match TLS.epoch tls with
|
match TLS.epoch tls with
|
||||||
| `Error -> None
|
| `Error -> None
|
||||||
| `Ok e -> e.Tls.Core.own_name
|
| `Ok e -> e.Tls.Core.own_name
|
||||||
```
|
</code></pre>
|
||||||
|
<p>Since this TLS extension is optional, the return type will be a string option.</p>
|
||||||
Since this TLS extension is optional, the return type will be a string option.
|
<p>Now, putting the dispatch together we need a function that gets all resources and the tls state value, and returns the data to send out:</p>
|
||||||
|
<pre><code class="language-OCaml">let dispatch nqsb usenix tron tls =
|
||||||
Now, putting the dispatch together we need a function that gets all resources and the tls state value, and returns the data to send out:
|
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let dispatch nqsb usenix tron tls =
|
|
||||||
match extract_name tls with
|
match extract_name tls with
|
||||||
| Some "usenix15.nqsb.io" -> usenix
|
| Some "usenix15.nqsb.io" -> usenix
|
||||||
| Some "tron.nqsb.io" -> tron
|
| Some "tron.nqsb.io" -> tron
|
||||||
| Some "nqsb.io" -> nqsb
|
| Some "nqsb.io" -> nqsb
|
||||||
| _ -> nqsb
|
| _ -> nqsb
|
||||||
```
|
</code></pre>
|
||||||
|
<p>This is again pure code, we need to put it now into the handler, our <code>tls_accept</code> calls the provided function with the tls flow:</p>
|
||||||
This is again pure code, we need to put it now into the handler, our `tls_accept` calls the provided function with the tls flow:
|
<pre><code class="language-OCaml">let tls_accept f cfg tcp =
|
||||||
|
TLS.server_of_flow cfg tcp >>= function
|
||||||
```OCaml
|
| `Error _ | `Eof -> TCP.close tcp
|
||||||
let tls_accept f cfg tcp =
|
| `Ok tls -> Tls.writev tls (f tls) >>= fun _ -> TLS.close tls
|
||||||
TLS.server_of_flow cfg tcp >>= function
|
</code></pre>
|
||||||
| `Error _ | `Eof -> TCP.close tcp
|
<p>And our full startup code:</p>
|
||||||
| `Ok tls -> Tls.writev tls (f tls) >>= fun _ -> TLS.close tls
|
<pre><code class="language-OCaml">let start stack keys kv =
|
||||||
```
|
let d_nqsb = [ header "text/html;charset=utf-8" ; Page.render ] in
|
||||||
|
read_pdf kv "nqsbtls-usenix-security15.pdf" >>= fun d_usenix ->
|
||||||
And our full startup code:
|
read_pdf kv "tron.pdf" >>= fun d_tron ->
|
||||||
|
|
||||||
```OCaml
|
|
||||||
let start stack keys kv =
|
|
||||||
let d_nqsb = [ header "text/html;charset=utf-8" ; Page.render ] in
|
|
||||||
read_pdf kv "nqsbtls-usenix-security15.pdf" >>= fun d_usenix ->
|
|
||||||
read_pdf kv "tron.pdf" >>= fun d_tron ->
|
|
||||||
let f = dispatch d_nqsb d_usenix d_tron in
|
let f = dispatch d_nqsb d_usenix d_tron in
|
||||||
|
|
||||||
read_cert keys "nqsb" >>= fun c_nqsb ->
|
read_cert keys "nqsb" >>= fun c_nqsb ->
|
||||||
read_cert keys "usenix15" >>= fun c_usenix ->
|
read_cert keys "usenix15" >>= fun c_usenix ->
|
||||||
read_cert keys "tron" >>= fun c_tron ->
|
read_cert keys "tron" >>= fun c_tron ->
|
||||||
let config = Tls.Config.server ~certificates:(`Multiple_default (c_nqsb, [ c_usenix ; c_tron])) () in
|
let config = Tls.Config.server ~certificates:(`Multiple_default (c_nqsb, [ c_usenix ; c_tron])) () in
|
||||||
|
|
||||||
S.listen_tcpv4 stack ~port:443 (tls_accept f config) ;
|
S.listen_tcpv4 stack ~port:443 (tls_accept f config) ;
|
||||||
S.listen stack
|
S.listen stack
|
||||||
```
|
</code></pre>
|
||||||
|
<p>That's it, the <a href="https://nqsb.io">nqsb.io</a> contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a <code>301 Moved permanently</code> HTTP status code).</p>
|
||||||
That's it, the [nqsb.io](https://nqsb.io) contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a `301 Moved permanently` HTTP status code).
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>A comparison using Firefox builtin network diagnostics shows that the waiting before receiving data is minimal (3ms, even spotted 0ms).</p>
|
||||||
## Conclusion
|
<p><a href="/static/img/performance-nqsbio.png"><img src="/static/img/performance-nqsbio.png" title="latency of our website" width="50%" /></a></p>
|
||||||
|
<p>We do not render HTML for each request, we do not splice data together, we <em>don't even read the client request</em>. And I'm sure we can improve the performance even more by profiling.</p>
|
||||||
A comparison using Firefox builtin network diagnostics shows that the waiting before receiving data is minimal (3ms, even spotted 0ms).
|
<p>We saw a journey from typed XML over key value stores, HTTP, TLS, and HTTPS. The actual application code of our unikernel serving <a href="https://nqsb.io">nqsb,io</a> is less than 100 lines of OCaml. We used MirageOS for our minimal HTTPS website, serving a <em>single resource per hostname</em>. We depend (directly) on the tyxml library, the mirage tool and network stack, and the tls library. That's it.</p>
|
||||||
|
<p>There is a long list of potential features, such as full HTTP protocol compliance (caching, favicon, ...), logging, natively getting let's encrypt certificates -- but in the web out there it is sufficient to get picked up by search engines, and the maintenance is marginal.</p>
|
||||||
[<img src="/static/img/performance-nqsbio.png" title="latency of our website" width="50%" />](/static/img/performance-nqsbio.png)
|
<p>For a start in MirageOS unikernels, look into our <a href="https://github.com/mirage/mirage-skeleton">mirage-skeleton</a> project, and into the <a href="https://github.com/mattgray/devwinter2016/">/dev/winter</a> presentation by Matt Gray.</p>
|
||||||
|
<p>I'm interested in feedback, either via
|
||||||
We do not render HTML for each request, we do not splice data together, we *don't even read the client request*. And I'm sure we can improve the performance even more by profiling.
|
<a href="https://twitter.com/h4nnes">twitter</a> or via eMail.</p>
|
||||||
|
<h2 id="other-updates-in-the-mirageos-ecosystem">Other updates in the MirageOS ecosystem</h2>
|
||||||
We saw a journey from typed XML over key value stores, HTTP, TLS, and HTTPS. The actual application code of our unikernel serving [nqsb,io](https://nqsb.io) is less than 100 lines of OCaml. We used MirageOS for our minimal HTTPS website, serving a *single resource per hostname*. We depend (directly) on the tyxml library, the mirage tool and network stack, and the tls library. That's it.
|
<ul>
|
||||||
|
<li><a href="https://github.com/Engil/Canopy">Canopy</a> improvements: <a href="https://github.com/Engil/Canopy/pull/26">no bower anymore</a>, <a href="https://github.com/Engil/Canopy/pull/27">HTTP caching support (via etags)</a>, <a href="https://github.com/Engil/Canopy/pull/31">listings now include dates</a>, <a href="https://github.com/Engil/Canopy/pull/32">dates are now in big-endian (y-m-d)</a>
|
||||||
There is a long list of potential features, such as full HTTP protocol compliance (caching, favicon, ...), logging, natively getting let's encrypt certificates -- but in the web out there it is sufficient to get picked up by search engines, and the maintenance is marginal.
|
</li>
|
||||||
|
<li><a href="http://canopy.mirage.io/irclogs/20-04-2016">MirageOS call irclog from 2014-04-20</a>
|
||||||
For a start in MirageOS unikernels, look into our [mirage-skeleton](https://github.com/mirage/mirage-skeleton) project, and into the [/dev/winter](https://github.com/mattgray/devwinter2016/) presentation by Matt Gray.
|
</li>
|
||||||
|
<li>blog article about <a href="https://abailly.github.io/posts/mirage-os-newbie.html">naive authentication service using MirageOS</a>
|
||||||
I'm interested in feedback, either via
|
</li>
|
||||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
<li><a href="https://lwn.net/SubscriberLink/684128/1436601f401c1f09/">OCaml 4.03 post</a>
|
||||||
|
</li>
|
||||||
## Other updates in the MirageOS ecosystem
|
</ul>
|
||||||
|
</article></div></div></main></body></html>
|
||||||
- [Canopy](https://github.com/Engil/Canopy) improvements: [no bower anymore](https://github.com/Engil/Canopy/pull/26), [HTTP caching support (via etags)](https://github.com/Engil/Canopy/pull/27), [listings now include dates](https://github.com/Engil/Canopy/pull/31), [dates are now in big-endian (y-m-d)](https://github.com/Engil/Canopy/pull/32)
|
|
||||||
- [MirageOS call irclog from 2014-04-20](http://canopy.mirage.io/irclogs/20-04-2016)
|
|
||||||
- blog article about [naive authentication service using MirageOS](https://abailly.github.io/posts/mirage-os-newbie.html)
|
|
||||||
- [OCaml 4.03 post](https://lwn.net/SubscriberLink/684128/1436601f401c1f09/)
|
|
||||||
30
index.html
Normal file
30
index.html
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Retreat2024" class="list-group-item"><h2 class="list-group-item-heading">The MirageOS retreat 2024</h2><span class="author">Written by hannes</span> <time>2024-05-15</time><br/><p class="list-group-item-text abstract"><p>My involvement and experience of the MirageOS retreat 2024</p>
|
||||||
|
</p></a><a href="/Posts/TCP-ns" class="list-group-item"><h2 class="list-group-item-heading">Redeveloping TCP from the ground up</h2><span class="author">Written by hannes</span> <time>2023-11-28</time><br/><p class="list-group-item-text abstract"><p>Core Internet protocols require operational experiments, even if formally specified</p>
|
||||||
|
</p></a><a href="/Posts/Albatross" class="list-group-item"><h2 class="list-group-item-heading">Deploying reproducible unikernels with albatross</h2><span class="author">Written by hannes</span> <time>2022-11-17</time><br/><p class="list-group-item-text abstract"><p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
|
||||||
|
</p></a><a href="/Posts/OpamMirror" class="list-group-item"><h2 class="list-group-item-heading">Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span> <time>2022-09-29</time><br/><p class="list-group-item-text abstract"><p>Re-developing an opam cache from scratch, as a MirageOS unikernel</p>
|
||||||
|
</p></a><a href="/Posts/Monitoring" class="list-group-item"><h2 class="list-group-item-heading">All your metrics belong to influx</h2><span class="author">Written by hannes</span> <time>2022-03-08</time><br/><p class="list-group-item-text abstract"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||||
|
</p></a><a href="/Posts/Deploy" class="list-group-item"><h2 class="list-group-item-heading">Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2021-06-30</time><br/><p class="list-group-item-text abstract"><p>Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder</p>
|
||||||
|
</p></a><a href="/Posts/EC" class="list-group-item"><h2 class="list-group-item-heading">Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span> <time>2021-04-23</time><br/><p class="list-group-item-text abstract"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||||
|
</p></a><a href="/Posts/NGI" class="list-group-item"><h2 class="list-group-item-heading">The road ahead for MirageOS in 2021</h2><span class="author">Written by hannes</span> <time>2021-01-25</time><br/><p class="list-group-item-text abstract"><p>Home office, MirageOS unikernels, 2020 recap, 2021 tbd</p>
|
||||||
|
</p></a><a href="/Posts/Traceroute" class="list-group-item"><h2 class="list-group-item-heading">Traceroute</h2><span class="author">Written by hannes</span> <time>2020-06-24</time><br/><p class="list-group-item-text abstract"><p>A MirageOS unikernel which traces the path between itself and a remote host.</p>
|
||||||
|
</p></a><a href="/Posts/DnsServer" class="list-group-item"><h2 class="list-group-item-heading">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2019-12-23</time><br/><p class="list-group-item-text abstract"><p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
|
||||||
|
</p></a><a href="/Posts/ReproducibleOPAM" class="list-group-item"><h2 class="list-group-item-heading">Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span> <time>2019-12-16</time><br/><p class="list-group-item-text abstract"><p>MirageOS unikernels are reproducible :)</p>
|
||||||
|
</p></a><a href="/Posts/X50907" class="list-group-item"><h2 class="list-group-item-heading">X509 0.7</h2><span class="author">Written by hannes</span> <time>2019-08-15</time><br/><p class="list-group-item-text abstract"><p>Five years since ocaml-x509 initial release, it has been reworked and used more widely</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/Pinata" class="list-group-item"><h2 class="list-group-item-heading">The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span> <time>2018-04-18</time><br/><p class="list-group-item-text abstract"><p>More than three years ago we launched our Bitcoin Piñata as a transparent security bait. It is still up and running!</p>
|
||||||
|
</p></a><a href="/Posts/DNS" class="list-group-item"><h2 class="list-group-item-heading">My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span> <time>2018-01-11</time><br/><p class="list-group-item-text abstract"><p>New year brings new possibilities and a new environment. I've been working on the most Widely deployed key-value store, the domain name system. Primary and secondary name services are available, including dynamic updates, notify, and tsig authentication.</p>
|
||||||
|
</p></a><a href="/Posts/VMM" class="list-group-item"><h2 class="list-group-item-heading">Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span> <time>2017-07-10</time><br/><p class="list-group-item-text abstract"><p>all we need is X.509</p>
|
||||||
|
</p></a><a href="/Posts/Conex" class="list-group-item"><h2 class="list-group-item-heading">Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>Conex is a library to verify and attest package release integrity and authenticity through the use of cryptographic signatures.</p>
|
||||||
|
</p></a><a href="/Posts/Maintainers" class="list-group-item"><h2 class="list-group-item-heading">Who maintains package X?</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>We describe why manual gathering of metadata is out of date, and version control systems are awesome.</p>
|
||||||
|
</p></a><a href="/Posts/Jackline" class="list-group-item"><h2 class="list-group-item-heading">Jackline, a secure terminal-based XMPP client</h2><span class="author">Written by hannes</span> <time>2017-01-30</time><br/><p class="list-group-item-text abstract"><p>implement it once to know you can do it. implement it a second time and you get readable code. implementing it a third time from scratch may lead to useful libraries.</p>
|
||||||
|
</p></a><a href="/Posts/Syslog" class="list-group-item"><h2 class="list-group-item-heading">Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span> <time>2016-11-05</time><br/><p class="list-group-item-text abstract"><p>sometimes preservation of data is useful</p>
|
||||||
|
</p></a><a href="/Posts/ARP" class="list-group-item"><h2 class="list-group-item-heading">Re-engineering ARP</h2><span class="author">Written by hannes</span> <time>2016-07-12</time><br/><p class="list-group-item-text abstract"><p>If you want it as you like, you've to do it yourself</p>
|
||||||
|
</p></a><a href="/Posts/Solo5" class="list-group-item"><h2 class="list-group-item-heading">Minimising the virtual machine monitor</h2><span class="author">Written by hannes</span> <time>2016-07-02</time><br/><p class="list-group-item-text abstract"><p>MirageOS solo5 multiboot native on bhyve</p>
|
||||||
|
</p></a><a href="/Posts/BottomUp" class="list-group-item"><h2 class="list-group-item-heading">Counting Bytes</h2><span class="author">Written by hannes</span> <time>2016-06-11</time><br/><p class="list-group-item-text abstract"><p>looking into dependencies and their sizes</p>
|
||||||
|
</p></a><a href="/Posts/Functoria" class="list-group-item"><h2 class="list-group-item-heading">Configuration DSL step-by-step</h2><span class="author">Written by hannes</span> <time>2016-05-10</time><br/><p class="list-group-item-text abstract"><p>how to actually configure the system</p>
|
||||||
|
</p></a><a href="/Posts/BadRecordMac" class="list-group-item"><h2 class="list-group-item-heading">Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span> <time>2016-05-03</time><br/><p class="list-group-item-text abstract"><p>10BTC could've been yours</p>
|
||||||
|
</p></a><a href="/Posts/nqsbWebsite" class="list-group-item"><h2 class="list-group-item-heading">Fitting the things together</h2><span class="author">Written by hannes</span> <time>2016-04-24</time><br/><p class="list-group-item-text abstract"><p>building a simple website</p>
|
||||||
|
</p></a><a href="/Posts/OCaml" class="list-group-item"><h2 class="list-group-item-heading">Why OCaml</h2><span class="author">Written by hannes</span> <time>2016-04-17</time><br/><p class="list-group-item-text abstract"><p>a gentle introduction into OCaml</p>
|
||||||
|
</p></a><a href="/Posts/OperatingSystem" class="list-group-item"><h2 class="list-group-item-heading">Operating systems</h2><span class="author">Written by hannes</span> <time>2016-04-09</time><br/><p class="list-group-item-text abstract"><p>Operating systems and MirageOS</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/UI
Normal file
3
tags/UI
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Jackline" class="list-group-item"><h2 class="list-group-item-heading">Jackline, a secure terminal-based XMPP client</h2><span class="author">Written by hannes</span> <time>2017-01-30</time><br/><p class="list-group-item-text abstract"><p>implement it once to know you can do it. implement it a second time and you get readable code. implementing it a third time from scratch may lead to useful libraries.</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
6
tags/background
Normal file
6
tags/background
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/BottomUp" class="list-group-item"><h2 class="list-group-item-heading">Counting Bytes</h2><span class="author">Written by hannes</span> <time>2016-06-11</time><br/><p class="list-group-item-text abstract"><p>looking into dependencies and their sizes</p>
|
||||||
|
</p></a><a href="/Posts/Functoria" class="list-group-item"><h2 class="list-group-item-heading">Configuration DSL step-by-step</h2><span class="author">Written by hannes</span> <time>2016-05-10</time><br/><p class="list-group-item-text abstract"><p>how to actually configure the system</p>
|
||||||
|
</p></a><a href="/Posts/OCaml" class="list-group-item"><h2 class="list-group-item-heading">Why OCaml</h2><span class="author">Written by hannes</span> <time>2016-04-17</time><br/><p class="list-group-item-text abstract"><p>a gentle introduction into OCaml</p>
|
||||||
|
</p></a><a href="/About" class="list-group-item"><h2 class="list-group-item-heading">About</h2><span class="author">Written by hannes</span> <time>2016-04-01</time><br/><p class="list-group-item-text abstract"><p>introduction (myself, this site)</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/bitcoin
Normal file
3
tags/bitcoin
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Pinata" class="list-group-item"><h2 class="list-group-item-heading">The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span> <time>2018-04-18</time><br/><p class="list-group-item-text abstract"><p>More than three years ago we launched our Bitcoin Piñata as a transparent security bait. It is still up and running!</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/community
Normal file
3
tags/community
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Retreat2024" class="list-group-item"><h2 class="list-group-item-heading">The MirageOS retreat 2024</h2><span class="author">Written by hannes</span> <time>2024-05-15</time><br/><p class="list-group-item-text abstract"><p>My involvement and experience of the MirageOS retreat 2024</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
9
tags/deployment
Normal file
9
tags/deployment
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Albatross" class="list-group-item"><h2 class="list-group-item-heading">Deploying reproducible unikernels with albatross</h2><span class="author">Written by hannes</span> <time>2022-11-17</time><br/><p class="list-group-item-text abstract"><p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
|
||||||
|
</p></a><a href="/Posts/OpamMirror" class="list-group-item"><h2 class="list-group-item-heading">Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span> <time>2022-09-29</time><br/><p class="list-group-item-text abstract"><p>Re-developing an opam cache from scratch, as a MirageOS unikernel</p>
|
||||||
|
</p></a><a href="/Posts/Monitoring" class="list-group-item"><h2 class="list-group-item-heading">All your metrics belong to influx</h2><span class="author">Written by hannes</span> <time>2022-03-08</time><br/><p class="list-group-item-text abstract"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||||
|
</p></a><a href="/Posts/Deploy" class="list-group-item"><h2 class="list-group-item-heading">Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2021-06-30</time><br/><p class="list-group-item-text abstract"><p>Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder</p>
|
||||||
|
</p></a><a href="/Posts/DnsServer" class="list-group-item"><h2 class="list-group-item-heading">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2019-12-23</time><br/><p class="list-group-item-text abstract"><p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/VMM" class="list-group-item"><h2 class="list-group-item-heading">Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span> <time>2017-07-10</time><br/><p class="list-group-item-text abstract"><p>all we need is X.509</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/future
Normal file
3
tags/future
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Solo5" class="list-group-item"><h2 class="list-group-item-heading">Minimising the virtual machine monitor</h2><span class="author">Written by hannes</span> <time>2016-07-02</time><br/><p class="list-group-item-text abstract"><p>MirageOS solo5 multiboot native on bhyve</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/http
Normal file
3
tags/http
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/nqsbWebsite" class="list-group-item"><h2 class="list-group-item-heading">Fitting the things together</h2><span class="author">Written by hannes</span> <time>2016-04-24</time><br/><p class="list-group-item-text abstract"><p>building a simple website</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/logging
Normal file
3
tags/logging
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Syslog" class="list-group-item"><h2 class="list-group-item-heading">Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span> <time>2016-11-05</time><br/><p class="list-group-item-text abstract"><p>sometimes preservation of data is useful</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
26
tags/mirageos
Normal file
26
tags/mirageos
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Retreat2024" class="list-group-item"><h2 class="list-group-item-heading">The MirageOS retreat 2024</h2><span class="author">Written by hannes</span> <time>2024-05-15</time><br/><p class="list-group-item-text abstract"><p>My involvement and experience of the MirageOS retreat 2024</p>
|
||||||
|
</p></a><a href="/Posts/TCP-ns" class="list-group-item"><h2 class="list-group-item-heading">Redeveloping TCP from the ground up</h2><span class="author">Written by hannes</span> <time>2023-11-28</time><br/><p class="list-group-item-text abstract"><p>Core Internet protocols require operational experiments, even if formally specified</p>
|
||||||
|
</p></a><a href="/Posts/Albatross" class="list-group-item"><h2 class="list-group-item-heading">Deploying reproducible unikernels with albatross</h2><span class="author">Written by hannes</span> <time>2022-11-17</time><br/><p class="list-group-item-text abstract"><p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
|
||||||
|
</p></a><a href="/Posts/OpamMirror" class="list-group-item"><h2 class="list-group-item-heading">Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span> <time>2022-09-29</time><br/><p class="list-group-item-text abstract"><p>Re-developing an opam cache from scratch, as a MirageOS unikernel</p>
|
||||||
|
</p></a><a href="/Posts/Monitoring" class="list-group-item"><h2 class="list-group-item-heading">All your metrics belong to influx</h2><span class="author">Written by hannes</span> <time>2022-03-08</time><br/><p class="list-group-item-text abstract"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||||
|
</p></a><a href="/Posts/Deploy" class="list-group-item"><h2 class="list-group-item-heading">Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2021-06-30</time><br/><p class="list-group-item-text abstract"><p>Finally, we provide reproducible binary MirageOS unikernels together with packages to reproduce them and setup your own builder</p>
|
||||||
|
</p></a><a href="/Posts/EC" class="list-group-item"><h2 class="list-group-item-heading">Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span> <time>2021-04-23</time><br/><p class="list-group-item-text abstract"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||||
|
</p></a><a href="/Posts/NGI" class="list-group-item"><h2 class="list-group-item-heading">The road ahead for MirageOS in 2021</h2><span class="author">Written by hannes</span> <time>2021-01-25</time><br/><p class="list-group-item-text abstract"><p>Home office, MirageOS unikernels, 2020 recap, 2021 tbd</p>
|
||||||
|
</p></a><a href="/Posts/Traceroute" class="list-group-item"><h2 class="list-group-item-heading">Traceroute</h2><span class="author">Written by hannes</span> <time>2020-06-24</time><br/><p class="list-group-item-text abstract"><p>A MirageOS unikernel which traces the path between itself and a remote host.</p>
|
||||||
|
</p></a><a href="/Posts/DnsServer" class="list-group-item"><h2 class="list-group-item-heading">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2019-12-23</time><br/><p class="list-group-item-text abstract"><p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
|
||||||
|
</p></a><a href="/Posts/ReproducibleOPAM" class="list-group-item"><h2 class="list-group-item-heading">Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span> <time>2019-12-16</time><br/><p class="list-group-item-text abstract"><p>MirageOS unikernels are reproducible :)</p>
|
||||||
|
</p></a><a href="/Posts/X50907" class="list-group-item"><h2 class="list-group-item-heading">X509 0.7</h2><span class="author">Written by hannes</span> <time>2019-08-15</time><br/><p class="list-group-item-text abstract"><p>Five years since ocaml-x509 initial release, it has been reworked and used more widely</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/Pinata" class="list-group-item"><h2 class="list-group-item-heading">The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span> <time>2018-04-18</time><br/><p class="list-group-item-text abstract"><p>More than three years ago we launched our Bitcoin Piñata as a transparent security bait. It is still up and running!</p>
|
||||||
|
</p></a><a href="/Posts/DNS" class="list-group-item"><h2 class="list-group-item-heading">My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span> <time>2018-01-11</time><br/><p class="list-group-item-text abstract"><p>New year brings new possibilities and a new environment. I've been working on the most Widely deployed key-value store, the domain name system. Primary and secondary name services are available, including dynamic updates, notify, and tsig authentication.</p>
|
||||||
|
</p></a><a href="/Posts/VMM" class="list-group-item"><h2 class="list-group-item-heading">Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span> <time>2017-07-10</time><br/><p class="list-group-item-text abstract"><p>all we need is X.509</p>
|
||||||
|
</p></a><a href="/Posts/Syslog" class="list-group-item"><h2 class="list-group-item-heading">Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span> <time>2016-11-05</time><br/><p class="list-group-item-text abstract"><p>sometimes preservation of data is useful</p>
|
||||||
|
</p></a><a href="/Posts/ARP" class="list-group-item"><h2 class="list-group-item-heading">Re-engineering ARP</h2><span class="author">Written by hannes</span> <time>2016-07-12</time><br/><p class="list-group-item-text abstract"><p>If you want it as you like, you've to do it yourself</p>
|
||||||
|
</p></a><a href="/Posts/Solo5" class="list-group-item"><h2 class="list-group-item-heading">Minimising the virtual machine monitor</h2><span class="author">Written by hannes</span> <time>2016-07-02</time><br/><p class="list-group-item-text abstract"><p>MirageOS solo5 multiboot native on bhyve</p>
|
||||||
|
</p></a><a href="/Posts/BottomUp" class="list-group-item"><h2 class="list-group-item-heading">Counting Bytes</h2><span class="author">Written by hannes</span> <time>2016-06-11</time><br/><p class="list-group-item-text abstract"><p>looking into dependencies and their sizes</p>
|
||||||
|
</p></a><a href="/Posts/Functoria" class="list-group-item"><h2 class="list-group-item-heading">Configuration DSL step-by-step</h2><span class="author">Written by hannes</span> <time>2016-05-10</time><br/><p class="list-group-item-text abstract"><p>how to actually configure the system</p>
|
||||||
|
</p></a><a href="/Posts/BadRecordMac" class="list-group-item"><h2 class="list-group-item-heading">Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span> <time>2016-05-03</time><br/><p class="list-group-item-text abstract"><p>10BTC could've been yours</p>
|
||||||
|
</p></a><a href="/Posts/nqsbWebsite" class="list-group-item"><h2 class="list-group-item-heading">Fitting the things together</h2><span class="author">Written by hannes</span> <time>2016-04-24</time><br/><p class="list-group-item-text abstract"><p>building a simple website</p>
|
||||||
|
</p></a><a href="/Posts/OperatingSystem" class="list-group-item"><h2 class="list-group-item-heading">Operating systems</h2><span class="author">Written by hannes</span> <time>2016-04-09</time><br/><p class="list-group-item-text abstract"><p>Operating systems and MirageOS</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
4
tags/monitoring
Normal file
4
tags/monitoring
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Monitoring" class="list-group-item"><h2 class="list-group-item-heading">All your metrics belong to influx</h2><span class="author">Written by hannes</span> <time>2022-03-08</time><br/><p class="list-group-item-text abstract"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/myself
Normal file
3
tags/myself
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/About" class="list-group-item"><h2 class="list-group-item-heading">About</h2><span class="author">Written by hannes</span> <time>2016-04-01</time><br/><p class="list-group-item-text abstract"><p>introduction (myself, this site)</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/opam
Normal file
3
tags/opam
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/OpamMirror" class="list-group-item"><h2 class="list-group-item-heading">Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span> <time>2022-09-29</time><br/><p class="list-group-item-text abstract"><p>Re-developing an opam cache from scratch, as a MirageOS unikernel</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/operating system
Normal file
3
tags/operating system
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/OperatingSystem" class="list-group-item"><h2 class="list-group-item-heading">Operating systems</h2><span class="author">Written by hannes</span> <time>2016-04-09</time><br/><p class="list-group-item-text abstract"><p>Operating systems and MirageOS</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
6
tags/overview
Normal file
6
tags/overview
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/Conex" class="list-group-item"><h2 class="list-group-item-heading">Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>Conex is a library to verify and attest package release integrity and authenticity through the use of cryptographic signatures.</p>
|
||||||
|
</p></a><a href="/Posts/OCaml" class="list-group-item"><h2 class="list-group-item-heading">Why OCaml</h2><span class="author">Written by hannes</span> <time>2016-04-17</time><br/><p class="list-group-item-text abstract"><p>a gentle introduction into OCaml</p>
|
||||||
|
</p></a><a href="/Posts/OperatingSystem" class="list-group-item"><h2 class="list-group-item-heading">Operating systems</h2><span class="author">Written by hannes</span> <time>2016-04-09</time><br/><p class="list-group-item-text abstract"><p>Operating systems and MirageOS</p>
|
||||||
|
</p></a><a href="/About" class="list-group-item"><h2 class="list-group-item-heading">About</h2><span class="author">Written by hannes</span> <time>2016-04-01</time><br/><p class="list-group-item-text abstract"><p>introduction (myself, this site)</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
6
tags/package signing
Normal file
6
tags/package signing
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/ReproducibleOPAM" class="list-group-item"><h2 class="list-group-item-heading">Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span> <time>2019-12-16</time><br/><p class="list-group-item-text abstract"><p>MirageOS unikernels are reproducible :)</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/Conex" class="list-group-item"><h2 class="list-group-item-heading">Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>Conex is a library to verify and attest package release integrity and authenticity through the use of cryptographic signatures.</p>
|
||||||
|
</p></a><a href="/Posts/Maintainers" class="list-group-item"><h2 class="list-group-item-heading">Who maintains package X?</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>We describe why manual gathering of metadata is out of date, and version control systems are awesome.</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
9
tags/protocol
Normal file
9
tags/protocol
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/TCP-ns" class="list-group-item"><h2 class="list-group-item-heading">Redeveloping TCP from the ground up</h2><span class="author">Written by hannes</span> <time>2023-11-28</time><br/><p class="list-group-item-text abstract"><p>Core Internet protocols require operational experiments, even if formally specified</p>
|
||||||
|
</p></a><a href="/Posts/Traceroute" class="list-group-item"><h2 class="list-group-item-heading">Traceroute</h2><span class="author">Written by hannes</span> <time>2020-06-24</time><br/><p class="list-group-item-text abstract"><p>A MirageOS unikernel which traces the path between itself and a remote host.</p>
|
||||||
|
</p></a><a href="/Posts/DnsServer" class="list-group-item"><h2 class="list-group-item-heading">Deploying authoritative OCaml-DNS servers as MirageOS unikernels</h2><span class="author">Written by hannes</span> <time>2019-12-23</time><br/><p class="list-group-item-text abstract"><p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
|
||||||
|
</p></a><a href="/Posts/DNS" class="list-group-item"><h2 class="list-group-item-heading">My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span> <time>2018-01-11</time><br/><p class="list-group-item-text abstract"><p>New year brings new possibilities and a new environment. I've been working on the most Widely deployed key-value store, the domain name system. Primary and secondary name services are available, including dynamic updates, notify, and tsig authentication.</p>
|
||||||
|
</p></a><a href="/Posts/Syslog" class="list-group-item"><h2 class="list-group-item-heading">Exfiltrating log data using syslog</h2><span class="author">Written by hannes</span> <time>2016-11-05</time><br/><p class="list-group-item-text abstract"><p>sometimes preservation of data is useful</p>
|
||||||
|
</p></a><a href="/Posts/ARP" class="list-group-item"><h2 class="list-group-item-heading">Re-engineering ARP</h2><span class="author">Written by hannes</span> <time>2016-07-12</time><br/><p class="list-group-item-text abstract"><p>If you want it as you like, you've to do it yourself</p>
|
||||||
|
</p></a><a href="/Posts/nqsbWebsite" class="list-group-item"><h2 class="list-group-item-heading">Fitting the things together</h2><span class="author">Written by hannes</span> <time>2016-04-24</time><br/><p class="list-group-item-text abstract"><p>building a simple website</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/provisioning
Normal file
3
tags/provisioning
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/VMM" class="list-group-item"><h2 class="list-group-item-heading">Albatross - provisioning, deploying, managing, and monitoring virtual machines</h2><span class="author">Written by hannes</span> <time>2017-07-10</time><br/><p class="list-group-item-text abstract"><p>all we need is X.509</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
12
tags/security
Normal file
12
tags/security
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/EC" class="list-group-item"><h2 class="list-group-item-heading">Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span> <time>2021-04-23</time><br/><p class="list-group-item-text abstract"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||||
|
</p></a><a href="/Posts/ReproducibleOPAM" class="list-group-item"><h2 class="list-group-item-heading">Reproducible MirageOS unikernel builds</h2><span class="author">Written by hannes</span> <time>2019-12-16</time><br/><p class="list-group-item-text abstract"><p>MirageOS unikernels are reproducible :)</p>
|
||||||
|
</p></a><a href="/Posts/X50907" class="list-group-item"><h2 class="list-group-item-heading">X509 0.7</h2><span class="author">Written by hannes</span> <time>2019-08-15</time><br/><p class="list-group-item-text abstract"><p>Five years since ocaml-x509 initial release, it has been reworked and used more widely</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/Pinata" class="list-group-item"><h2 class="list-group-item-heading">The Bitcoin Piñata - no candy for you</h2><span class="author">Written by hannes</span> <time>2018-04-18</time><br/><p class="list-group-item-text abstract"><p>More than three years ago we launched our Bitcoin Piñata as a transparent security bait. It is still up and running!</p>
|
||||||
|
</p></a><a href="/Posts/Conex" class="list-group-item"><h2 class="list-group-item-heading">Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>Conex is a library to verify and attest package release integrity and authenticity through the use of cryptographic signatures.</p>
|
||||||
|
</p></a><a href="/Posts/Maintainers" class="list-group-item"><h2 class="list-group-item-heading">Who maintains package X?</h2><span class="author">Written by hannes</span> <time>2017-02-16</time><br/><p class="list-group-item-text abstract"><p>We describe why manual gathering of metadata is out of date, and version control systems are awesome.</p>
|
||||||
|
</p></a><a href="/Posts/Jackline" class="list-group-item"><h2 class="list-group-item-heading">Jackline, a secure terminal-based XMPP client</h2><span class="author">Written by hannes</span> <time>2017-01-30</time><br/><p class="list-group-item-text abstract"><p>implement it once to know you can do it. implement it a second time and you get readable code. implementing it a third time from scratch may lead to useful libraries.</p>
|
||||||
|
</p></a><a href="/Posts/Solo5" class="list-group-item"><h2 class="list-group-item-heading">Minimising the virtual machine monitor</h2><span class="author">Written by hannes</span> <time>2016-07-02</time><br/><p class="list-group-item-text abstract"><p>MirageOS solo5 multiboot native on bhyve</p>
|
||||||
|
</p></a><a href="/Posts/BadRecordMac" class="list-group-item"><h2 class="list-group-item-heading">Catch the bug, walking through the stack</h2><span class="author">Written by hannes</span> <time>2016-05-03</time><br/><p class="list-group-item-text abstract"><p>10BTC could've been yours</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
3
tags/tcp
Normal file
3
tags/tcp
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/TCP-ns" class="list-group-item"><h2 class="list-group-item-heading">Redeveloping TCP from the ground up</h2><span class="author">Written by hannes</span> <time>2023-11-28</time><br/><p class="list-group-item-text abstract"><p>Core Internet protocols require operational experiments, even if formally specified</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
6
tags/tls
Normal file
6
tags/tls
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>full stack engineer</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="full stack engineer" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="flex-container"><div class="list-group listing"><a href="/Posts/EC" class="list-group-item"><h2 class="list-group-item-heading">Cryptography updates in OCaml and MirageOS</h2><span class="author">Written by hannes</span> <time>2021-04-23</time><br/><p class="list-group-item-text abstract"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||||
|
</p></a><a href="/Posts/X50907" class="list-group-item"><h2 class="list-group-item-heading">X509 0.7</h2><span class="author">Written by hannes</span> <time>2019-08-15</time><br/><p class="list-group-item-text abstract"><p>Five years since ocaml-x509 initial release, it has been reworked and used more widely</p>
|
||||||
|
</p></a><a href="/Posts/Summer2019" class="list-group-item"><h2 class="list-group-item-heading">Summer 2019</h2><span class="author">Written by hannes</span> <time>2019-07-08</time><br/><p class="list-group-item-text abstract"><p>Bringing MirageOS into production, take IV monitoring, CalDAV, DNS</p>
|
||||||
|
</p></a><a href="/Posts/nqsbWebsite" class="list-group-item"><h2 class="list-group-item-heading">Fitting the things together</h2><span class="author">Written by hannes</span> <time>2016-04-24</time><br/><p class="list-group-item-text abstract"><p>building a simple website</p>
|
||||||
|
</p></a></div></div></div></main></body></html>
|
||||||
Loading…
Reference in a new issue