About
Published: 2016-04-01 (last updated: 2023-11-20)
What is a "full stack engineer"?
+About
Published: 2016-04-01 (last updated: 2024-10-11)
What is a "full stack engineer"?
Analysing the word literally, we should start with silicon and some electrons, maybe a soldering iron, and build everything all the way up to our favourite communication system.
@@ -68,7 +68,7 @@ researched (I would not consider it production ready yet, needs more work on a faster runtime, and libraries).After I finished my PhD, I decided to slack off for some time to make decent espresso. I ended up spending the winter (beginning of 2014) in Mirleft, -Morocco. A good friend of mine pointed me to MirageOS, a +Morocco. A good friend of mine pointed me to MirageOS, a clean-slate operating system written in the high-level language OCaml. I got hooked pretty fast, after some experience with LISP machines I imagined a modern OS written in a single functional programming language.
diff --git a/Posts/ARP b/Posts/ARP index 198f10d..e82fad9 100644 --- a/Posts/ARP +++ b/Posts/ARP @@ -1,5 +1,5 @@ -Re-engineering ARP
Published: 2016-07-12 (last updated: 2021-11-19)
What is ARP?
+Re-engineering ARP
Published: 2016-07-12 (last updated: 2024-10-11)
What is ARP?
ARP is the Address Resolution Protocol, widely used in legacy IP networks (which support only IPv4). It is responsible to translate an IPv4 address to an Ethernet address. It is strictly more general, abstracting over protocol and hardware addresses. It is basically DNS (the domain name system) on a different layer.
ARP is link-local: ARP frames are not routed into other networks, all stay in the same broadcast domain. Thus there is no need for a hop limit (time-to-live). A reverse lookup mechanism (hardware address to protocol) is also available, named reverse ARP ;).
I will focus on ARP in this article, as used widely to translate IPv4 addresses into Ethernet addresses. There are two operations in ARP: request and response. A request is usually broadcasted to all hosts (by setting the destination to the broadcast Ethernet address, ff:ff:ff:ff:ff:ff), while a reply is send via unicast (to the host which requested that information).
That's it. Pretty straightforward.
Design
Back in 2008, together with Andreas Bogk, we just used a hash table and installed expiration and retransmission timers when needed. Certainly timers sometimes needed to be cancelled, and testing the code was cumbersome. It were only 250 lines of Dylan code plus some wire format definition.
-Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated. The code should not contain surprises, but straightforward to understand. The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should. Once this is achieved, testing is straightforward. If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused. It can also be repurposed to run as a test oracle. You can read more marketing of this style in our Usenix security paper.
+Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated. The code should not contain surprises, but straightforward to understand. The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should. Once this is achieved, testing is straightforward. If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused. It can also be repurposed to run as a test oracle. You can read more marketing of this style in our Usenix security paper.
My proposed style and hash tables are not good friends, since hash tables in OCaml are imperative structures. Instead, a Map (documentation) is a functional data structure for associating keys with values. Its underlying data structure is a balanced binary tree.
Our ARP handler certainly has some state, at least its IPv4 address, its Ethernet address, and the map containing entries.
We have to deal with the various effects mentioned earlier:
diff --git a/Posts/BadRecordMac b/Posts/BadRecordMac index ef9cb98..eceb50f 100644 --- a/Posts/BadRecordMac +++ b/Posts/BadRecordMac @@ -1,5 +1,5 @@ -Catch the bug, walking through the stack
Published: 2016-05-03 (last updated: 2021-11-19)
BAD RECORD MAC
+Catch the bug, walking through the stack
Published: 2016-05-03 (last updated: 2024-10-11)
BAD RECORD MAC
Roughly 2 weeks ago, Engil informed me that a TLS alert pops up in his browser sometimes when he reads this website. His browser reported that the message authentication code was wrong. From RFC 5246: This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
I tried hard, but could not reproduce, but was very worried and was eager to find the root cause (some little fear remained that it was in our TLS stack). I setup this website with some TLS-level tracing (extending the code from our TLS handshake server). We tried to reproduce the issue with traces and packet captures (both on client and server side) in place from our computer labs office with no success. Later, Engil tried from his home and after 45MB of wire data, ran into this issue. Finally, evidence! Isolating the TCP flow with the alert resulted in just about 200KB of packet capture data (TLS ASCII trace around 650KB).
My 2018 contains robur and starts with re-engineering DNS
Published: 2018-01-11 (last updated: 2023-11-20)
2018
+My 2018 contains robur and starts with re-engineering DNS
Published: 2018-01-11 (last updated: 2024-10-11)
2018
At the end of 2017, I resigned from my PostDoc position at University of Cambridge (in the rems project). Early -December 2017 I organised the 4th MirageOS hack +December 2017 I organised the 4th MirageOS hack retreat, with which I'm -very satisfied. In March 2018 the 5th retreat will +very satisfied. In March 2018 the 5th retreat will happen (please sign up!).
In 2018 I moved to Berlin and started to work for the (non-profit) Center for the cultivation of technology with our diff --git a/Posts/DnsServer b/Posts/DnsServer index ae578cd..85eec38 100644 --- a/Posts/DnsServer +++ b/Posts/DnsServer @@ -1,5 +1,5 @@ -
Deploying authoritative OCaml-DNS servers as MirageOS unikernels
Published: 2019-12-23 (last updated: 2023-03-02)
Goal
+Deploying authoritative OCaml-DNS servers as MirageOS unikernels
Published: 2019-12-23 (last updated: 2024-10-11)
Goal
Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as nqsb.io and robur.coop. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as µDNS.
Prerequisites
You need to own a domain, and be able to delegate the name service to your own servers. @@ -173,7 +173,7 @@ personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv 10.0.42.2.10.0.42.3._transfer.mirage. DNSKEY 0 3 163 cDK6sKyvlt8UBerZlmxuD84ih2KookJGDagJlLVNo20= git-repo> git commit -m "updates" . && git push -
Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see other targets for further information).
+Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see other targets for further information).
# back to primary
$ cd ../dns-primary-git
$ mirage configure -t hvt # or e.g. -t spt (and solo5-spt below)
diff --git a/Posts/EC b/Posts/EC
index 60f21e6..90f0aad 100644
--- a/Posts/EC
+++ b/Posts/EC
@@ -1,8 +1,8 @@
-Cryptography updates in OCaml and MirageOS Cryptography updates in OCaml and MirageOS
Published: 2021-04-23 (last updated: 2021-11-19)Introduction
+Cryptography updates in OCaml and MirageOS Cryptography updates in OCaml and MirageOS
Published: 2021-04-23 (last updated: 2024-10-11)Introduction
Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified fiat-crypto stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.
Elliptic curve cryptography
-Since May 2020, our OCaml-TLS stack supports TLS 1.3 (since tls version 0.12.0 on opam).
+Since May 2020, our OCaml-TLS stack supports TLS 1.3 (since tls version 0.12.0 on opam).
TLS 1.3 requires elliptic curve cryptography - which was not available in mirage-crypto (the maintained fork of nocrypto).
There are two major uses of elliptic curves: key exchange (ECDH) for establishing a shared secret over an insecure channel, and digital signature (ECDSA) for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)
Elliptic curve cryptoraphy is vulnerable to various timing attacks - have a read of the overview article on ECDSA. When implementing elliptic curve cryptography, it is best to avoid these known attacks. Gladly, there are some projects which address these issues by construction.
diff --git a/Posts/Functoria b/Posts/Functoria
index 174706e..870c5c1 100644
--- a/Posts/Functoria
+++ b/Posts/Functoria
@@ -1,11 +1,11 @@
-Configuration DSL step-by-step Configuration DSL step-by-step
Published: 2016-05-10 (last updated: 2021-11-19)Sorry for being late again with this article, I had other ones planned, but am not yet satisfied with content and code, will have to wait another week.
+Configuration DSL step-by-step Configuration DSL step-by-step
Published: 2016-05-10 (last updated: 2024-10-11)Sorry for being late again with this article, I had other ones planned, but am not yet satisfied with content and code, will have to wait another week.
MirageOS configuration
As described in an earlier post, MirageOS is a library operating system which generates single address space custom kernels (so called unikernels) for each application. The application code is (mostly) independent on the used backend. To achieve this, the language which expresses the configuration of a MirageOS unikernel is rather complex, and has to deal with package dependencies, setup of layers (network stack starting at the (virtual) ethernet device, or sockets), logging, tracing.
The abstraction over concrete implementation of e.g. the network stack is done by providing a module signature in the mirage-types package. The socket-based network stack, the tap device based network stack, and the Xen virtual network device based network stack implement this signature (depending on other module signatures). The unikernel contains code which applies those dependent modules to instantiate a custom-tailored network stack for the specific configuration. A developer should only describe what their requirements are, the user who wants to deploy it should provide the concrete configuration. And the developer should not need to manually instantiate the network stack for all possible configurations, this is what the mirage tool should embed.
-Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code. This turned out to be error prone. In 2015 Drup developed Functoria, a domain-specific language (DSL) to organize functor applications, primarily for MirageOS. It has been introduced in a blog post. It is not limited to MirageOS (although this is the primary user right now).
+Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code. This turned out to be error prone. In 2015 Drup developed Functoria, a domain-specific language (DSL) to organize functor applications, primarily for MirageOS. It has been introduced in a blog post. It is not limited to MirageOS (although this is the primary user right now).
Functoria has been included in MirageOS since its 2.7.0 release at the end of February 2016. Functoria provides support for command line arguments which can then either be passed at configuration time or at boot time to the unikernel (such as IP address configuration) using the cmdliner library underneath (and includes dynamic man pages, help, sensible command line parsing, and even visualisation (mirage describe) of the configuration and data dependencies).
-I won't go into details about command line arguments in here, please have a look at the functoria blog post in case you're interested. Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel (running here, source). Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown. Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).
+I won't go into details about command line arguments in here, please have a look at the functoria blog post in case you're interested. Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel (running here, source). Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown. Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).
Unikernel
Similar to my nqsb.io website post, this unikernel only has a single resource and thus does not need to do any parsing (or even call read). The main function is start:
let start stack _ =
@@ -111,7 +111,7 @@ end
mirage-platform now has 4.03 support and strtod (finally :)
-blog posts about retreat in marrakech
+ blog posts about retreat in marrakech
syndic 1.5.0 release now using ptime instead of calendar
diff --git a/Posts/Jackline b/Posts/Jackline
index 580e6f7..534051c 100644
--- a/Posts/Jackline
+++ b/Posts/Jackline
@@ -1,6 +1,6 @@
-Jackline, a secure terminal-based XMPP client Jackline, a secure terminal-based XMPP client
Published: 2017-01-30 (last updated: 2021-09-08)
-Back in 2014, when we implemented TLS in OCaml, at some point
+
Jackline, a secure terminal-based XMPP client Jackline, a secure terminal-based XMPP client
Published: 2017-01-30 (last updated: 2024-10-11)
+Back in 2014, when we implemented TLS in OCaml, at some point
I was bored with TLS. I usually need at least two projects (but not more than 5) at the same time to
procrastinate the one I should do with the other one - it is always more fun to
do what you're not supposed to do. I started to implement another security
@@ -139,7 +139,7 @@ friends logged in multiple times with wrongly set priorities - and end-to-end
encryption. I don't need inline HTML, avatar images, my currently running
music, leaking timezone information, etc. I explicitly don't want to import any
private key material from other clients and libraries, because I want to ensure
-that the key was generated by a good random number generator (read David's blog article on randomness and entropy).
+that the key was generated by a good random number generator (read David's blog article on randomness and entropy).
The security story is crucial: always do strict certificate validation, fail
hard, make it noticable by the user if they're doing insecure communication.
Only few people are into reading out loud their OTR public key fingerprint, and
@@ -244,7 +244,7 @@ things I'd like to have as well).
The XMPP library makes heavy use of
functors (to abstract over the concrete IO, etc.), and embeds IO deep inside it.
-I do prefer (see e.g. our TLS paper, or my ARP post) these days to have a pure interface for
+I do prefer (see e.g. our TLS paper, or my ARP post) these days to have a pure interface for
the protocol implementation, providing explicit input (state, event, data), and
output (state, action, potentially data to send on network, potentially data to
process by the application). The sasl implementation
diff --git a/Posts/NGI b/Posts/NGI
index 939b2c7..5b56ca1 100644
--- a/Posts/NGI
+++ b/Posts/NGI
@@ -1,5 +1,5 @@
-
The road ahead for MirageOS in 2021 The road ahead for MirageOS in 2021
Published: 2021-01-25 (last updated: 2021-11-19)Introduction
+The road ahead for MirageOS in 2021 The road ahead for MirageOS in 2021
Published: 2021-01-25 (last updated: 2024-10-11)Introduction
2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations, and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)
I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.
For 2021, we (at robur) got a grant from the EU (via NGI pointer) for "Deploying MirageOS" (more details below), and another grant from OCaml software foundation for securing the opam supply chain (using conex). Some long-awaited releases for MirageOS libraries, namely a ssh implementation and a rewrite of our git implementation have already been published.
@@ -13,9 +13,9 @@
Mirage-crypto provides the underlying cryptographic primitives, initially released in March 2020 as a fork of nocrypto -- huge thanks to pqwy for his great work. Mirage-crypto detects CPU features at runtime (thanks to Julow) (bugfix for bswap), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports compilation on Windows (thanks to avsm), async entropy harvesting (thanks to seliopou), 32 bit support, chacha20/poly1305 (thanks to abeaumont), cross-compilation (thanks to EduardoRFS) and various bug fixes, even memory leak (thanks to talex5 for reporting several of these issues), and RSA interoperability (thanks to psafont for investigation and mattjbray for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.
Qubes Firewall
The MirageOS based Qubes firewall is the most widely used MirageOS unikernel. And it got major updates: in May Steffi announced her and Mindy's work on improving it for Qubes 4.0 - including dynamic firewall rules via QubesDB. Thanks to prototypefund for sponsoring.
-In October 2020, we released Mirage 3.9 with PVH virtualization mode (thanks to mato). There's still a memory leak to be investigated and fixed.
+In October 2020, we released Mirage 3.9 with PVH virtualization mode (thanks to mato). There's still a memory leak to be investigated and fixed.
IPv6
-In December, with Mirage 3.10 we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to nojb for the initial code and MagnusS.
+In December, with Mirage 3.10 we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to nojb for the initial code and MagnusS.
Turns out this blog, but also robur services, are now available via IPv6 :)
Albatross
Also in December, I pushed an initial release of albatross, a unikernel orchestration system with remote access. Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.
@@ -23,7 +23,7 @@
CA certs
For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to emillon for initial code, there are now ca-certs and ca-certs-nss opam packages (see release announcement) which fills this gap.
Unikernels
-I developed several useful unikernels in 2020, and also pushed a unikernel gallery to the Mirage website:
+I developed several useful unikernels in 2020, and also pushed a unikernel gallery to the Mirage website:
Traceroute in MirageOS
I already wrote about traceroute which traces the routing to a given remote host.
Unipi - static website hosting
diff --git a/Posts/OCaml b/Posts/OCaml
index a7cdf0c..6b09323 100644
--- a/Posts/OCaml
+++ b/Posts/OCaml
@@ -1,5 +1,5 @@
-Why OCaml Why OCaml
Published: 2016-04-17 (last updated: 2021-11-19)Programming
+Why OCaml Why OCaml
Published: 2016-04-17 (last updated: 2024-10-11)Programming
For me, programming is fun. I enjoy doing it, every single second. All the way
from designing over experimenting to debugging why it does not do what I want.
In the end, the computer is dumb and executes only what you (or code from
@@ -72,7 +72,7 @@ on a regular schedule. Daniel wrote opam repository contains over 1000
libraries. The quality varies, I personally like the small libraries done by
Daniel Bünzli, as well as our
-nqsb libraries (see mirleft org),
+not quite so broken libraries (see mirleft org),
notty.
A concise library (not much code),
including tests, documentation, etc. is
@@ -101,7 +101,7 @@ developed within MirageOS. Because I'm impatient, I setup an Not-quite-so-broken TLS: lessons in re-engineering a security protocol
+Not-quite-so-broken TLS: lessons in re-engineering a security protocol
specification and
implementation. I'm interested in feedback, either via
twitter or via eMail.
diff --git a/Posts/OpamMirror b/Posts/OpamMirror
index 07e43cd..576e911 100644
--- a/Posts/OpamMirror
+++ b/Posts/OpamMirror
@@ -1,5 +1,5 @@
-Mirroring the opam repository and all tarballs Mirroring the opam repository and all tarballs
Published: 2022-09-29 (last updated: 2023-11-20)We at robur developed opam-mirror in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).
+Mirroring the opam repository and all tarballs Mirroring the opam repository and all tarballs
Published: 2022-09-29 (last updated: 2024-10-11)We at robur developed opam-mirror in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).
What is opam and why should I care?
Opam is the OCaml package manager (also used by other projects such as coq). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as ocaml/opam-repository, where authors of OCaml software can contribute (as pull request) their latest releases.
When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state "my package will not work with cmdliner in version 1.2.0").
@@ -10,9 +10,9 @@
Opam, after initialisation, downloads the index.tar.gz from https://opam.ocaml.org/index.tar.gz, and uses this as the local opam universe. An opam install cmdliner will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e.
How does the opam repository work?
According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses opam admin index periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was a reported downtime.
-Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS hack retreats in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).
+Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS hack retreats in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).
Re-developing "opam admin create" as MirageOS unikernel
-The need for a local opam cache at our reproducible build infrastructure and the retreats, we decided to develop opam-mirror as a MirageOS unikernel. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the opam admin index is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.
+The need for a local opam cache at our reproducible build infrastructure and the retreats, we decided to develop opam-mirror as a MirageOS unikernel. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the opam admin index is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.
Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.
In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with crunch, or using a remote git repository), but recent development, e.g. of chamelon sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.
Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.
diff --git a/Posts/OperatingSystem b/Posts/OperatingSystem
index 72c064d..3dd93bc 100644
--- a/Posts/OperatingSystem
+++ b/Posts/OperatingSystem
@@ -1,5 +1,5 @@
-Operating systems Operating systems
Published: 2016-04-09 (last updated: 2021-11-19)Sorry to be late with this entry, but I had to fix some issues.
+Operating systems Operating systems
Published: 2016-04-09 (last updated: 2024-10-11)Sorry to be late with this entry, but I had to fix some issues.
What is an operating system?
Wikipedia says: "An operating system (OS) is system software that manages
computer hardware and software resources and provides common services for
@@ -57,7 +57,7 @@ Dylan), and as mentioned earlier I went into formal methods and mechanised
proofs of full functional correctness properties.
MirageOS
At the end of 2013, David pointed me to
-MirageOS, an operating system developed from scratch in the
+MirageOS, an operating system developed from scratch in the
functional and statically typed language OCaml. I've not
used much OCaml before, but some other functional programming languages.
Since then, I spend nearly every day on developing OCaml libraries (with varying success on being happy
@@ -153,7 +153,7 @@ how MirageOS fits into the picture. I'm interested in feedback, either via
I was travelling
-good news: it now works on Xen, and there is an atom feed
+ good news: it now works on Xen, and there is an atom feed
life of an "eat your own dogfood" full stack engineer ;)
diff --git a/Posts/Pinata b/Posts/Pinata
index 9ac8b7e..12af5aa 100644
--- a/Posts/Pinata
+++ b/Posts/Pinata
@@ -1,21 +1,21 @@
-The Bitcoin Piñata - no candy for you The Bitcoin Piñata - no candy for you
Published: 2018-04-18 (last updated: 2021-11-19)History
+The Bitcoin Piñata - no candy for you The Bitcoin Piñata - no candy for you
Published: 2018-04-18 (last updated: 2024-10-11)History
On February 10th 2015 David Kaloper-Meršinjak and Hannes Mehnert
-launched (read also Amir's
+launched (read also Amir's
description) our bug bounty
program in the form of our
Bitcoin Piñata MirageOS unikernel. Thanks again to
IPredator for both hosting our services and lending us
-the 10 Bitcoins! We analysed a
+the 10 Bitcoins! We analysed a
bit more in depth after running it for five months. Mindy recently wrote about
whacking the Bitcoin
Piñata.
On March 18th 2018, after more than three years, IPredator, the lender of the Bitcoins, repurposed the 10 Bitcoins for other projects. Initially, we thought that the Piñata would maybe run for a month or two, but IPredator, David, and I decided to keep it running. The update of the Piñata's bounty is a good opportunity to reflect on the project.
The 10 Bitcoin in the Piñata were fluctuating in price over time, at peak worth 165000€.
From the start of the Piñata project, we published the source code, the virtual machine image, and the versions of the used libraries in a git repository. Everybody could develop their exploits locally before launching them against our Piñata. The Piñata provides TLS endpoints, which require private keys and certificates. These are generated by the Piñata at startup, and the secret for the Bitcoin wallet is provided as a command line argument.
-Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with solo5, and in December 2017 it was migrated to native BHyve (using ukvm-bin and solo5). We also changed the Piñata code to accomodate for updates, such as the MirageOS 3.0 release, and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).
+Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with solo5, and in December 2017 it was migrated to native BHyve (using ukvm-bin and solo5). We also changed the Piñata code to accomodate for updates, such as the MirageOS 3.0 release, and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).
Motivation
-We built the Piñata for many purposes: to attract security professionals to evaluate our from-scratch developed TLS stack, to gather empirical data for our Usenix Security 15 paper, and as an improvement to current bug bounty programs.
+We built the Piñata for many purposes: to attract security professionals to evaluate our from-scratch developed TLS stack, to gather empirical data for our Usenix Security 15 paper, and as an improvement to current bug bounty programs.
Most bug bounty programs require communication via forms and long wait times for
human experts to evaluate the potential bug. This evaluation is subjective,
intransparent, and often requires signing of non-disclosure agreements (NDA),
@@ -32,7 +32,7 @@ that the access to the wallet is actually inside. But trust us, it is!
In April 2016 we stumbled upon an information disclosure in the virtual network
device driver for Xen in MirageOS. Given enough
bandwidth, this could have been used to access the private wallet key. We
-upgraded the Piñata and released the MirageOS Security Advisory
+upgraded the Piñata and released the MirageOS Security Advisory
00.
We analysed the Piñata's access logs to the and bucketed them into website traffic and bounty connections. We are still wondering what happened in July 2015 and July 2017 where the graph shows spikes. Could it be a presentation mentioning the Piñata, or a new automated tool which tests for TLS vulnerabilities, or an increase in market price for Bitcoins?
diff --git a/Posts/ReproducibleOPAM b/Posts/ReproducibleOPAM
index 3269b4b..fc61200 100644
--- a/Posts/ReproducibleOPAM
+++ b/Posts/ReproducibleOPAM
@@ -1,11 +1,11 @@
-Reproducible MirageOS unikernel builds Reproducible MirageOS unikernel builds
Published: 2019-12-16 (last updated: 2021-11-19)Reproducible builds summit
-I'm just back from the Reproducible builds summit 2019. In 2018, several people developing OCaml and opam and MirageOS, attended the Reproducible builds summit in Paris. The notes from last year on opam reproducibility and MirageOS reproducibility are online. After last years workshop, Raja started developing the opam reproducibilty builder orb, which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.
+Reproducible MirageOS unikernel builds Reproducible MirageOS unikernel builds
Published: 2019-12-16 (last updated: 2024-10-11)Reproducible builds summit
+I'm just back from the Reproducible builds summit 2019. In 2018, several people developing OCaml and opam and MirageOS, attended the Reproducible builds summit in Paris. The notes from last year on opam reproducibility and MirageOS reproducibility are online. After last years workshop, Raja started developing the opam reproducibilty builder orb, which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.
Goal
Stepping back a bit, first look on the goal of reproducible builds: when compiling source code multiple times, the produced binaries should be identical. It should be sufficient if the binaries are behaviourally equal, but this is pretty hard to check. It is much easier to check bit-wise identity of binaries, and relaxes the burden on the checker -- checking for reproducibility is reduced to computing the hash of the binaries. Let's stick to the bit-wise identical binary definition, which also means software developers have to avoid non-determinism during compilation in their toolchains, dependent libraries, and developed code.
A checklist of potential things leading to non-determinism has been written up by the reproducible builds project. Examples include recording the build timestamp into the binary, ordering of code and embedded data. The reproducible builds project also developed disorderfs for testing reproducibility and diffoscope for comparing binaries with file-dependent readers, falling back to objdump and hexdump. A giant test infrastructure with lots of variations between the builds, mostly using Debian, has been setup over the years.
Reproducibility is a precondition for trustworthy binaries. See why does it matter. If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. I will only ever publish binaries if they are reproducible.
-My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of MirageOS unikernels -- a unikernel is a statically linked ELF binary to be run as Unix process or virtual machine. MirageOS heavily uses OCaml and opam, the OCaml package manager, and is an opam package itself. Thus, checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package.
+My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of MirageOS unikernels -- a unikernel is a statically linked ELF binary to be run as Unix process or virtual machine. MirageOS heavily uses OCaml and opam, the OCaml package manager, and is an opam package itself. Thus, checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package.
Reproducible builds with opam
Testing for reproducibility is achieved by taking the sources and compile them twice independently. Afterwards the equality of the resulting binaries can be checked. In trivial projects, the sources is just a single file, or originate from a single tarball. In OCaml, opam uses a community repository where OCaml developers publish their package releases to, but can also use custom repositores, and in addition pin packages to git remotes (url including branch or commit), or a directory on the local filesystem. Manually tracking and updating all dependent packages of a MirageOS unikernel is not feasible: our hello-world compiled for hvt (kvm/BHyve) already has 79 opam dependencies, including the OCaml compiler which is distribued as opam package. The unikernel serving this website depends on 175 opam packages.
Conceptually there should be two tools, the initial builder, which takes the latest opam packages which do not conflict, and exports exact package versions used during the build, as well as hashes of binaries. The other tool is a rebuilder, which imports the export, conducts a build, and outputs the hashes of the produced binaries.
diff --git a/Posts/Retreat2024 b/Posts/Retreat2024
index 9133bd8..f5fb8ab 100644
--- a/Posts/Retreat2024
+++ b/Posts/Retreat2024
@@ -1,7 +1,7 @@
-The MirageOS retreat 2024 The MirageOS retreat 2024
Published: 2024-05-15 (last updated: 2024-05-16)End of April I spent a week in Marrakech to work on MirageOS at our annual retreat. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
+The MirageOS retreat 2024 The MirageOS retreat 2024
Published: 2024-05-15 (last updated: 2024-10-11)End of April I spent a week in Marrakech to work on MirageOS at our annual retreat. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.
-Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our website only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
+Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our website only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.
Operator handbook
I sat down with some people to start off a MirageOS operator handbook - we talked about the scope and the early chapters (and what is not in scope). This is still very much work in progress, issues and pull requests are welcome. The target audience is someone who wants to execute a MirageOS unikernel with hvt on a laptop or server. It will not be necessary to install OCaml, but the focus is really on the operational aspects with binaries that we provide from our reproducible build infrastructure.
diff --git a/Posts/Summer2019 b/Posts/Summer2019
index 52eefaa..a46f628 100644
--- a/Posts/Summer2019
+++ b/Posts/Summer2019
@@ -1,9 +1,9 @@
-Summer 2019 Summer 2019
Published: 2019-07-08 (last updated: 2021-11-19)Working at robur
+Summer 2019 Summer 2019
Published: 2019-07-08 (last updated: 2024-10-11)Working at robur
As announced previously, I started to work at robur early 2018. We're a collective of five people, distributed around Europe and the US, with the goal to deploy MirageOS unikernels. We do this by developing bespoke MirageOS unikernels which provide useful services, and deploy them for ourselves. We also develop new libraries and enhance existing ones and other components of MirageOS. Example unikernels include our website which uses Canopy, a CalDAV server that stores entries in a git remote, and DNS servers (the latter two are further described below).
Robur is part of the non-profit company Center for the Cultivation of Technology, who are managing the legal and administrative sides for us. We're ourselves responsible to acquire funding to pay ourselves reasonable salaries. We received funding for CalDAV from prototypefund and further funding from Tarides, for TLS 1.3 from OCaml Labs; security-audited an OCaml codebase, and received donations, also in the form of Bitcoins. We're looking for further funded collaborations and also contracting, mail us at team@robur.io. Please donate (tax-deductible in EU), so we can accomplish our goal of putting robust and sustainable MirageOS unikernels into production, replacing insecure legacy system that emit tons of CO2.
Deploying MirageOS unikernels
-While several examples are running since years (the MirageOS website, Bitcoin Piñata, TLS demo server, etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.
+While several examples are running since years (the MirageOS website, Bitcoin Piñata, TLS demo server, etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.
Service deployment is complex: you have to consider its configuration, exfiltration of logs and metrics, provisioning with valid key material (TLS certificate, hmac shared secret) and authenticators (CA certificate, ssh key fingerprint). Instead of requiring millions lines of code during orchestration (such as Kubernetes), creating the images (docker), or provisioning (ansible), why not minimise the required configuration and dependencies?
Earlier in this blog I introduced Albatross, which serves in an enhanced version as our deployment platform on a physical machine (running 15 unikernels at the moment), I won't discuss more detail thereof in this article.
CalDAV
diff --git a/Posts/Syslog b/Posts/Syslog
index 317e83d..d6d77f1 100644
--- a/Posts/Syslog
+++ b/Posts/Syslog
@@ -1,5 +1,5 @@
-Exfiltrating log data using syslog Exfiltrating log data using syslog
Published: 2016-11-05 (last updated: 2021-11-19)It has been a while since my last entry... I've been busy working on too many
+
Exfiltrating log data using syslog Exfiltrating log data using syslog
Published: 2016-11-05 (last updated: 2024-10-11)It has been a while since my last entry... I've been busy working on too many
projects in parallel, and was also travelling on several continents. I hope to
get back to a biweekly cycle.
What is syslog?
@@ -61,7 +61,7 @@ MirageOS, and a more reliable log stream (what about authentication and
encryption?). I'll cover both topics in the rest of this article.
MirageOS integration
Since Mirage3, syslog is integrated (see
-documentation).
+documentation).
Some additions to your config.ml are needed, see ns
example or
marrakech
diff --git a/Posts/TCP-ns b/Posts/TCP-ns
index 64015b2..2e22951 100644
--- a/Posts/TCP-ns
+++ b/Posts/TCP-ns
@@ -1,15 +1,15 @@
-Redeveloping TCP from the ground up Redeveloping TCP from the ground up
Published: 2023-11-28 (last updated: 2023-11-29)The Transmission Control Protocol (TCP) is one of the main Internet protocols. Usually spoken on top of the Internet Protocol (legacy version 4 or version 6), it provides a reliable, ordered, and error-checked stream of octets. When an application uses TCP, they get these properties for free (in contrast to UDP).
+Redeveloping TCP from the ground up Redeveloping TCP from the ground up
Published: 2023-11-28 (last updated: 2024-10-11)The Transmission Control Protocol (TCP) is one of the main Internet protocols. Usually spoken on top of the Internet Protocol (legacy version 4 or version 6), it provides a reliable, ordered, and error-checked stream of octets. When an application uses TCP, they get these properties for free (in contrast to UDP).
As common for Internet protocols, TCP is specified in a series of so-called requests for comments (RFC). The latest revised version from August 2022 is RFC 9293; the initial one was RFC 793 from September 1981.
My brief personal TCP story
My interest in TCP started back in 2006 when we worked on a network stack in Dylan (these days abandoned). Ever since, I wanted to understand the implementation tradeoffs in more detail, including attacks and how to prevent a TCP stack from being vulnerable.
In 2012, I attended ICFP in Copenhagen while a PhD student at ITU Copenhagen. There, Peter Sewell gave an invited talk "Tales from the jungle" about rigorous methods for real-world infrastructure (C semantics, hardware (concurrency) behaviour of CPUs, TCP/IP, and likely more). Working on formal specifications myself in (my dissertation), and having a strong interest in real systems, I was immediately hooked by his perspective.
To dive a bit more into network semantics, the work done on TCP by Peter Sewell, et al., is a formal specification (or a model) of TCP/IP and the Unix sockets API developed in HOL4. It is a label transition system with nondeterministic choices, and the model itself is executable. It has been validated with the real world by collecting thousands of traces on Linux, Windows, and FreeBSD, which have been checked by the model for validity. This copes with the different implementations of the English prose of the RFCs. The network semantics research found several issues in existing TCP stacks and reported them upstream to have them fixed (though, there still is some special treatment, e.g., for the "BSD listen bug").
-In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the journal paper (author's copy) published. At the same time, the MirageOS melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.
+In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the journal paper (author's copy) published. At the same time, the MirageOS melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.
My intention was to understand TCP better and use the specification as a basis for a TCP stack for MirageOS. The existing one (which is still used) has technical debt: a high issue to number of lines ratio. The Lwt monad is ubiquitous, which makes testing and debugging pretty hard, and also utilising multiple cores with OCaml Multicore won't be easy. Plus it has various resource leaks, and there is no active maintainer. But honestly, it works fine on a local network, and with well-behaved traffic. It doesn't work that well on the wild Internet with a variety of broken implementations. Apart from resource leakage, which made me implement things such as restart-on-failure in Albatross, there are certain connection states which will never be exited.
The rise of µTCP
Back in Cambridge, I didn't manage to write a TCP stack based on the model, but in 2019, I restarted that work and got µTCP (the formal model manually translated to OCaml) to compile and do TCP session setup and teardown. Since it was a model that uses nondeterminism, this couldn't be translated one-to-one into an executable program, but there are places where decisions have to be made. Due to other projects, I worked only briefly in 2021 and 2022 on µTCP, but finally in the Summer of 2023, I motivated myself to push µTCP into a usable state. So far I've spend 25 days in 2023 on µTCP. Thanks to Tarides for supporting my work.
-Since late August, we have been running some unikernels using µTCP, e.g., the retreat website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.
+Since late August, we have been running some unikernels using µTCP, e.g., the retreat website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.
The secondary nameserver issue
One of our secondary nameservers attempts to receive zones (via AXFR using TCP) from another nameserver that is currently not running. Thus it replies to each SYN packet a corresponding RST. Below I graphed the network utilisation (send data/packets is positive y-axis, receive part on the negative) over time (on the x-axis) on the left and memory usage (bytes on y-axis) over time (x-axis) on the right of our nameserver. You can observe that both increases over time, and roughly every 3 hours, the unikernel hits its configured memory limit (64 MB), crashes with out of memory, and is restarted. The graph below is using the mirage-tcpip stack.
diff --git a/Posts/Traceroute b/Posts/Traceroute
index 495464d..f9cd7b5 100644
--- a/Posts/Traceroute
+++ b/Posts/Traceroute
@@ -1,5 +1,5 @@
-Traceroute Traceroute
Published: 2020-06-24 (last updated: 2021-11-19)Traceroute
+Traceroute Traceroute
Published: 2020-06-24 (last updated: 2024-10-11)Traceroute
Is a diagnostic utility which displays the route and measures transit delays of
packets across an Internet protocol (IP) network.
$ doas solo5-hvt --net:service=tap0 -- traceroute.hvt --ipv4=10.0.42.2/24 --ipv4-gateway=10.0.42.1 --host=198.167.222.207
@@ -331,7 +331,7 @@ let () =
And voila, that's all the code. If you copy it together (or download the two
files from the GitHub repository),
-and have OCaml, opam, and mirage (>= 3.8.0) installed,
+and have OCaml, opam, and mirage (>= 3.8.0) installed,
you should be able to:
$ mirage configure -t hvt
$ make depend
diff --git a/Posts/X50907 b/Posts/X50907
index b7d3faf..620a108 100644
--- a/Posts/X50907
+++ b/Posts/X50907
@@ -1,8 +1,8 @@
-X509 0.7 X509 0.7
Published: 2019-08-15 (last updated: 2021-11-19)Cryptographic material
+X509 0.7 X509 0.7
Published: 2019-08-15 (last updated: 2024-10-11)Cryptographic material
Once a private and public key pair is generated (doesn't matter whether it is plain RSA, DSA, ECC on any curve), this is fine from a scientific point of view, and can already be used for authenticating and encrypting. From a practical point of view, the public parts need to be exchanged and verified (usually a fingerprint or hash thereof). This leads to the struggle how to encode this cryptographic material, and how to embed an identity (or multiple), capabilities, and other information into it. X.509 is a standard to solve this encoding and embedding, and provides more functionality, such as establishing chains of trust and revocation of invalidated or compromised material. X.509 uses certificates, which contain the public key, and additional information (in a extensible key-value store), and are signed by an issuer, either the private key corresponding to the public key - a so-called self-signed certificate - or by a different private key, an authority one step up the chain. A rather long, but very good introduction to certificates by Mike Malone is available here.
OCaml ecosystem evolving
-More than 5 years ago David Kaloper and I released the initial ocaml-x509 package as part of our TLS stack, which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in RFC 5280). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's asn1-combinators library changed much over time.
+More than 5 years ago David Kaloper and I released the initial ocaml-x509 package as part of our TLS stack, which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in RFC 5280). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's asn1-combinators library changed much over time.
The OCaml ecosystem evolved over the years, which lead to some changes:
- Camlp4 deprecation - we used camlp4 for stream parsers of PEM-encoded certificates, and sexplib.syntax to derive s-expression decoders and encoders;
diff --git a/Posts/nqsbWebsite b/Posts/nqsbWebsite
index f9bca02..abfed0d 100644
--- a/Posts/nqsbWebsite
+++ b/Posts/nqsbWebsite
@@ -1,7 +1,7 @@
-
Fitting the things together Fitting the things together
Published: 2016-04-24 (last updated: 2021-11-19)Task
+Fitting the things together Fitting the things together
Published: 2016-04-24 (last updated: 2024-10-11)Task
Our task is to build a small unikernel which provides a project website. On our way we will wade through various layers using code examples. The website itself contains a few paragraphs of text, some link lists, and our published papers in pdf form.
-Spoiler alert final result can be seen here, the full code here.
+Spoiler alert final result can be seen (no longer available), the full code here.
A first idea
We could go all the way to use conduit for wrapping connections, and mirage-http (using cohttp, a very lightweight HTTP server). We'd just need to write routing code which in the end reads from a virtual file system, and some HTML and CSS for the actual site.
Turns out, the conduit library is already 1.7 MB in size and depends on 34 libraries, cohttp is another 3.7 MB and 40 dependent libraries.
@@ -170,25 +170,14 @@ the BTC Piñata: just send the resource
S.listen_tcpv4 stack ~port:443 (tls_accept f config) ;
S.listen stack
-
That's it, the nqsb.io contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a 301 Moved permanently HTTP status code).
+That's it, it contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a 301 Moved permanently HTTP status code).
Conclusion
A comparison using Firefox builtin network diagnostics shows that the waiting before receiving data is minimal (3ms, even spotted 0ms).
We do not render HTML for each request, we do not splice data together, we don't even read the client request. And I'm sure we can improve the performance even more by profiling.
-We saw a journey from typed XML over key value stores, HTTP, TLS, and HTTPS. The actual application code of our unikernel serving nqsb,io is less than 100 lines of OCaml. We used MirageOS for our minimal HTTPS website, serving a single resource per hostname. We depend (directly) on the tyxml library, the mirage tool and network stack, and the tls library. That's it.
+We saw a journey from typed XML over key value stores, HTTP, TLS, and HTTPS. The actual application code of our unikernel serving nqsb.io is less than 100 lines of OCaml. We used MirageOS for our minimal HTTPS website, serving a single resource per hostname. We depend (directly) on the tyxml library, the mirage tool and network stack, and the tls library. That's it.
There is a long list of potential features, such as full HTTP protocol compliance (caching, favicon, ...), logging, natively getting let's encrypt certificates -- but in the web out there it is sufficient to get picked up by search engines, and the maintenance is marginal.
For a start in MirageOS unikernels, look into our mirage-skeleton project, and into the /dev/winter presentation by Matt Gray.
I'm interested in feedback, either via
twitter or via eMail.
-Other updates in the MirageOS ecosystem
-
-- Canopy improvements: no bower anymore, HTTP caching support (via etags), listings now include dates, dates are now in big-endian (y-m-d)
-
-- MirageOS call irclog from 2014-04-20
-
-- blog article about naive authentication service using MirageOS
-
-- OCaml 4.03 post
-
-
urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156 full stack engineer 2024-05-16T14:19:12-00:00 <p>My involvement and experience of the MirageOS retreat 2024</p>
- 2024-05-15T14:38:51-00:00 <p>End of April I spent a week in Marrakech to work on MirageOS at our annual <a href="https://retreat.mirage.io">retreat</a>. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.</p>
+urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156 full stack engineer 2024-10-11T09:43:26-00:00 <p>My involvement and experience of the MirageOS retreat 2024</p>
+ 2024-05-15T14:38:51-00:00 <p>End of April I spent a week in Marrakech to work on MirageOS at our annual <a href="https://retreat.mirageos.org">retreat</a>. This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.</p>
<p>Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.</p>
-<p>Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our <a href="https://mirage.io">website</a> only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).</p>
+<p>Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our <a href="https://mirageos.org">website</a> only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).</p>
<p>I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.</p>
<h2 id="operator-handbook">Operator handbook</h2>
<p>I sat down with some people to start off a <a href="https://mirage.github.io/operator-handbook/index.html">MirageOS operator handbook</a> - we talked about the scope and the early chapters (and what is not in scope). This is still very much work in progress, <a href="https://github.com/mirage/operator-handbook">issues</a> and pull requests are welcome. The target audience is someone who wants to execute a MirageOS unikernel with hvt on a laptop or server. It will not be necessary to install OCaml, but the focus is really on the operational aspects with binaries that we provide from our <a href="https://builds.robur.coop">reproducible build infrastructure</a>.</p>
@@ -21,18 +21,18 @@
<p>I'm looking forward to other writeups and future retreats, or other events where I can say hi to attendees.</p>
<p>We at <a href="https://robur.coop">Robur</a> are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by <a href="https://aenderwerk.de/donate/">donating</a> (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).</p>
<p>If you have any questions, reach us best via eMail to team AT robur DOT coop.</p>
- urn:uuid:2c9e2634-78c5-5957-b6f2-d0706f80d7eb The MirageOS retreat 2024 2024-05-16T14:19:12-00:00 hannes <p>Core Internet protocols require operational experiments, even if formally specified</p>
+ urn:uuid:2c9e2634-78c5-5957-b6f2-d0706f80d7eb The MirageOS retreat 2024 2024-10-11T09:43:26-00:00 hannes <p>Core Internet protocols require operational experiments, even if formally specified</p>
2023-11-28T21:17:01-00:00 <p>The <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">Transmission Control Protocol (TCP)</a> is one of the main Internet protocols. Usually spoken on top of the Internet Protocol (legacy version 4 or version 6), it provides a reliable, ordered, and error-checked stream of octets. When an application uses TCP, they get these properties for free (in contrast to UDP).</p>
<p>As common for Internet protocols, TCP is specified in a series of so-called requests for comments (RFC). The latest revised version from August 2022 is <a href="https://datatracker.ietf.org/doc/html/rfc9293">RFC 9293</a>; the initial one was <a href="https://datatracker.ietf.org/doc/html/rfc793">RFC 793</a> from September 1981.</p>
<h1 id="my-brief-personal-tcp-story">My brief personal TCP story</h1>
<p>My interest in TCP started back in 2006 when we worked on a <a href="https://github.com/dylan-hackers/network-night-vision">network stack in Dylan</a> (these days abandoned). Ever since, I wanted to understand the implementation tradeoffs in more detail, including attacks and how to prevent a TCP stack from being vulnerable.</p>
<p>In 2012, I attended ICFP in Copenhagen while a PhD student at ITU Copenhagen. There, <a href="https://www.cl.cam.ac.uk/~pes20/">Peter Sewell</a> gave an invited talk "Tales from the jungle" about rigorous methods for real-world infrastructure (C semantics, hardware (concurrency) behaviour of CPUs, TCP/IP, and likely more). Working on formal specifications myself in (<a href="https://en.itu.dk/-/media/EN/Research/PhD-Programme/PhD-defences/2013/130731-Hannes-Mehnert-PhD-dissertation-finalpdf.pdf">my dissertation</a>), and having a strong interest in real systems, I was immediately hooked by his perspective.</p>
<p>To dive a bit more into <a href="https://www.cl.cam.ac.uk/~pes20/Netsem/">network semantics</a>, the work done on TCP by Peter Sewell, et al., is a formal specification (or a model) of TCP/IP and the Unix sockets API developed in HOL4. It is a label transition system with nondeterministic choices, and the model itself is executable. It has been validated with the real world by collecting thousands of traces on Linux, Windows, and FreeBSD, which have been checked by the model for validity. This copes with the different implementations of the English prose of the RFCs. The network semantics research found several issues in existing TCP stacks and reported them upstream to have them fixed (though, there still is some special treatment, e.g., for the "BSD listen bug").</p>
-<p>In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the <a href="https://dl.acm.org/doi/10.1145/3243650">journal paper</a> (<a href="http://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf">author's copy</a>) published. At the same time, the <a href="https://mirage.io">MirageOS</a> melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.</p>
+<p>In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the <a href="https://dl.acm.org/doi/10.1145/3243650">journal paper</a> (<a href="http://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf">author's copy</a>) published. At the same time, the <a href="https://mirageos.org">MirageOS</a> melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.</p>
<p>My intention was to understand TCP better and use the specification as a basis for a TCP stack for MirageOS. The <a href="https://github.com/mirage/mirage-tcpip">existing one</a> (which is still used) has technical debt: a high issue to number of lines ratio. The Lwt monad is ubiquitous, which makes testing and debugging pretty hard, and also utilising multiple cores with OCaml Multicore won't be easy. Plus it has various resource leaks, and there is no active maintainer. But honestly, it works fine on a local network, and with well-behaved traffic. It doesn't work that well on the wild Internet with a variety of broken implementations. Apart from resource leakage, which made me implement things such as restart-on-failure in <a href="https://github.com/robur-coop/albatross">Albatross</a>, there are certain connection states which will never be exited.</p>
<h1 id="the-rise-of-µtcp">The rise of <a href="https://github.com/robur-coop/utcp">µTCP</a></h1>
<p>Back in Cambridge, I didn't manage to write a TCP stack based on the model, but in 2019, I restarted that work and got µTCP (the formal model manually translated to OCaml) to compile and do TCP session setup and teardown. Since it was a model that uses nondeterminism, this couldn't be translated one-to-one into an executable program, but there are places where decisions have to be made. Due to other projects, I worked only briefly in 2021 and 2022 on µTCP, but finally in the Summer of 2023, I motivated myself to push µTCP into a usable state. So far I've spend 25 days in 2023 on µTCP. Thanks to <a href="https://tarides.com">Tarides</a> for supporting my work.</p>
-<p>Since late August, we have been running some unikernels using µTCP, e.g., the <a href="https://retreat.mirage.io">retreat</a> website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.</p>
+<p>Since late August, we have been running some unikernels using µTCP, e.g., the <a href="https://retreat.mirageos.org">retreat</a> website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.</p>
<h1 id="the-secondary-nameserver-issue">The secondary nameserver issue</h1>
<p>One of our secondary nameservers attempts to receive zones (via AXFR using TCP) from another nameserver that is currently not running. Thus it replies to each SYN packet a corresponding RST. Below I graphed the network utilisation (send data/packets is positive y-axis, receive part on the negative) over time (on the x-axis) on the left and memory usage (bytes on y-axis) over time (x-axis) on the right of our nameserver. You can observe that both increases over time, and roughly every 3 hours, the unikernel hits its configured memory limit (64 MB), crashes with <em>out of memory</em>, and is restarted. The graph below is using the <code>mirage-tcpip</code> stack.</p>
<p><a href="/static/img/a.ns.mtcp.png"><img src="/static/img/a.ns.mtcp.png" width="750" /></a></p>
@@ -58,7 +58,7 @@
<p>We'll take more time to investigate issues of µTCP in production, plan to write further documentation and blog posts, and hopefully soon will be ready for an initial public release. In the meantime, you can follow our development repository.</p>
<p>We at <a href="https://robur.coop">Robur</a> are working as a collective since 2018 on public funding, commercial contracts, and donations. Our mission is to get sustainable, robust, and secure MirageOS unikernels developed and deployed. Running your own digital communication infrastructure should be easy, including trustworthy binaries and smooth upgrades. You can help us continue our work by <a href="https://aenderwerk.de/donate/">donating</a> (select Robur from the drop-down or put "donation Robur" in the purpose of the bank transfer).</p>
<p>If you have any questions, reach us best via eMail to team AT robur DOT coop.</p>
- urn:uuid:96688956-0808-5d44-b795-1d64cbb4f947 Redeveloping TCP from the ground up 2023-11-29T13:31:13-00:00 hannes <p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
+urn:uuid:96688956-0808-5d44-b795-1d64cbb4f947 Redeveloping TCP from the ground up 2024-10-11T09:43:26-00:00 hannes <p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
2022-11-17T12:41:11-00:00 <p>EDIT (2023-05-16): Updated with albatross release version 2.0.0.</p>
<h2 id="deploying-mirageos-unikernels">Deploying MirageOS unikernels</h2>
<p>More than five years ago, I posted <a href="/Posts/VMM">how to deploy MirageOS unikernels</a>. My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described <a href="/Posts/Deploy">how to deploy reproducible unikernels</a>.</p>
@@ -152,9 +152,9 @@ $ pkg install solo5 albatross
<p>Opam, after initialisation, downloads the <code>index.tar.gz</code> from <code>https://opam.ocaml.org/index.tar.gz</code>, and uses this as the local opam universe. An <code>opam install cmdliner</code> will resolve the dependencies, and download all required tarballs. The download is first tried from the cache, and if that failed, the URL in the package file is used. The download from the cache uses the base url, appends the archive-mirror, followed by the hash algorithm, the first two characters of the has of the tarball, and the hex encoded hash of the archive, i.e. for cmdliner 1.1.1 which specifies its sha512: <code>https://opam.ocaml.org/cache/sha512/54/5478ad833da254b5587b3746e3a8493e66e867a081ac0f653a901cc8a7d944f66e4387592215ce25d939be76f281c4785702f54d4a74b1700bc8838a62255c9e</code>.</p>
<h1 id="how-does-the-opam-repository-work">How does the opam repository work?</h1>
<p>According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses <code>opam admin index</code> periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was <a href="https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/">a reported downtime</a>.</p>
-<p>Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS <a href="http://retreat.mirage.io">hack retreats</a> in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).</p>
+<p>Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS <a href="http://retreat.mirageos.org">hack retreats</a> in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).</p>
<h1 id="re-developing-opam-admin-create-as-mirageos-unikernel">Re-developing "opam admin create" as MirageOS unikernel</h1>
-<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
+<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirageos.org">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
<p>Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.</p>
<p>In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with <a href="https://github.com/mirage/ocaml-crunch">crunch</a>, or using a remote git repository), but recent development, e.g. of <a href="https://somerandomidiot.com/blog/2022/03/04/chamelon/">chamelon</a> sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.</p>
<p>Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.</p>
@@ -171,7 +171,7 @@ $ pkg install solo5 albatross
<h1 id="conclusion">Conclusion</h1>
<p>To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide <a href="https://builds.robur.coop/job/opam-mirror">reproducible binaries</a> for solo5-hvt. You can use our opam mirror with <code>opam repository set-url default https://opam.robur.coop</code> (revert to the other with <code>opam repository set-url default https://opam.ocaml.org</code>) or use it as a backup with <code>opam repository add robur --rank 2 https://opam.robur.coop</code>.</p>
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
- urn:uuid:0dbd251f-32c7-57bd-8e8f-7392c0833a09 Mirroring the opam repository and all tarballs 2023-11-20T16:58:35-00:00 hannes <p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
+urn:uuid:0dbd251f-32c7-57bd-8e8f-7392c0833a09 Mirroring the opam repository and all tarballs 2024-10-11T09:43:26-00:00 hannes <p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
2022-03-08T11:26:31-00:00 <h1 id="introduction-to-monitoring">Introduction to monitoring</h1>
<p>At <a href="https://robur.coop">robur</a> we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our <a href="https://builds.robur.coop">reproducible builds infrastructure</a>. Another part is, once deployed we want to observe what is going on.</p>
<p>I first got into touch with monitoring - collecting and graphing metrics - with <a href="https://oss.oetiker.ch/mrtg/">MRTG</a> and <a href="https://munin-monitoring.org/">munin</a> - and the simple network management protocol <a href="https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol">SNMP</a>. From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.</p>
@@ -347,7 +347,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
2021-04-23T13:33:06-00:00 <h2 id="introduction">Introduction</h2>
<p>Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a> stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.</p>
<h2 id="elliptic-curve-cryptography">Elliptic curve cryptography</h2>
-<p><a href="https://mirage.io/blog/tls-1-3-mirageos">Since May 2020</a>, our <a href="https://usenix15.nqsb.io">OCaml-TLS</a> stack supports TLS 1.3 (since tls version 0.12.0 on opam).</p>
+<p><a href="https://mirageos.org/blog/tls-1-3-mirageos">Since May 2020</a>, our <a href="https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak">OCaml-TLS</a> stack supports TLS 1.3 (since tls version 0.12.0 on opam).</p>
<p>TLS 1.3 requires elliptic curve cryptography - which was not available in <a href="https://github.com/mirage/mirage-crypto">mirage-crypto</a> (the maintained fork of <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a>).</p>
<p>There are two major uses of elliptic curves: <a href="https://en.wikipedia.org/wiki/Elliptic-curve_Diffie%E2%80%93Hellman">key exchange (ECDH)</a> for establishing a shared secret over an insecure channel, and <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm">digital signature (ECDSA)</a> for authentication, integrity, and non-repudiation. (Please note that the construction of digital signatures on Edwards curves (Curve25519, Ed448) is called EdDSA instead of ECDSA.)</p>
<p>Elliptic curve cryptoraphy is <a href="https://eprint.iacr.org/2020/615">vulnerable</a> <a href="https://raccoon-attack.com/">to</a> <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-5407">various</a> <a href="https://github.com/mimoo/timing_attack_ecdsa_tls">timing</a> <a href="https://minerva.crocs.fi.muni.cz/">attacks</a> - have a read of the <a href="https://blog.trailofbits.com/2020/06/11/ecdsa-handle-with-care/">overview article on ECDSA</a>. When implementing elliptic curve cryptography, it is best to avoid these known attacks. Gladly, there are some projects which address these issues by construction.</p>
@@ -387,7 +387,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
<p>Another potential project that is very close now is a certificate authority developed in MirageOS - now that EC keys, PKCS 12, revocation lists, ... are implemented.</p>
<h2 id="footer">Footer</h2>
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
- urn:uuid:16427713-5da1-50cd-b17c-ca5b5cca431d Cryptography updates in OCaml and MirageOS 2021-11-19T18:04:52-00:00 hannes <p>Home office, MirageOS unikernels, 2020 recap, 2021 tbd</p>
+urn:uuid:16427713-5da1-50cd-b17c-ca5b5cca431d Cryptography updates in OCaml and MirageOS 2024-10-11T09:43:26-00:00 hannes <p>Home office, MirageOS unikernels, 2020 recap, 2021 tbd</p>
2021-01-25T12:45:54-00:00 <h2 id="introduction">Introduction</h2>
<p>2020 was an intense year. I hope you're healthy and keep being healthy. I am privileged (as lots of software engineers and academics are) to be able to work from home during the pandemic. Let's not forget people in less privileged situations, and let’s try to give them as much practical, psychological and financial support as we can these days. And as much joy as possible to everyone around :)</p>
<p>I cancelled the autumn MirageOS retreat due to the pandemic. Instead I collected donations for our hosts in Marrakech - they were very happy to receive our financial support, since they had a difficult year, since their income is based on tourism. I hope that in autumn 2021 we'll have an on-site retreat again.</p>
@@ -402,9 +402,9 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
<p><a href="https://github.com/mirage/mirage-crypto">Mirage-crypto</a> provides the underlying cryptographic primitives, initially released in March 2020 as a fork of <a href="https://github.com/mirleft/ocaml-nocrypto">nocrypto</a> -- huge thanks to <a href="https://github.com/pqwy">pqwy</a> for his great work. Mirage-crypto detects <a href="https://github.com/mirage/mirage-crypto/pull/53">CPU features at runtime</a> (thanks to <a href="https://github.com/Julow">Julow</a>) (<a href="https://github.com/mirage/mirage-crypto/pull/96">bugfix for bswap</a>), using constant time modular exponentation (powm_sec) and hardens against Lenstra's CRT attack, supports <a href="https://github.com/mirage/mirage-crypto/pull/39">compilation on Windows</a> (thanks to <a href="https://github.com/avsm">avsm</a>), <a href="https://github.com/mirage/mirage-crypto/pull/90">async entropy harvesting</a> (thanks to <a href="https://github.com/seliopou">seliopou</a>), <a href="https://github.com/mirage/mirage-crypto/pull/65">32 bit support</a>, <a href="https://github.com/mirage/mirage-crypto/pull/72">chacha20/poly1305</a> (thanks to <a href="https://github.com/abeaumont">abeaumont</a>), <a href="https://github.com/mirage/mirage-crypto/pull/84">cross-compilation</a> (thanks to <a href="https://github.com/EduardoRFS">EduardoRFS</a>) and <a href="https://github.com/mirage/mirage-crypto/pull/78">various</a> <a href="https://github.com/mirage/mirage-crypto/pull/81">bug</a> <a href="https://github.com/mirage/mirage-crypto/pull/83">fixes</a>, even <a href="https://github.com/mirage/mirage-crypto/pull/95">memory leak</a> (thanks to <a href="https://github.com/talex5">talex5</a> for reporting several of these issues), and <a href="https://github.com/mirage/mirage-crypto/pull/99">RSA</a> <a href="https://github.com/mirage/mirage-crypto/pull/100">interoperability</a> (thanks to <a href="https://github.com/psafont">psafont</a> for investigation and <a href="https://github.com/mattjbray">mattjbray</a> for reporting). This library feels very mature now - being used by multiple stakeholders, and lots of issues have been fixed in 2020.</p>
<h3 id="qubes-firewall">Qubes Firewall</h3>
<p>The <a href="https://github.com/mirage/qubes-mirage-firewall/">MirageOS based Qubes firewall</a> is the most widely used MirageOS unikernel. And it got major updates: in May <a href="https://github.com/linse">Steffi</a> <a href="https://groups.google.com/g/qubes-users/c/Xzplmkjwa5Y">announced</a> her and <a href="https://github.com/yomimono">Mindy's</a> work on improving it for Qubes 4.0 - including <a href="https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x">dynamic firewall rules via QubesDB</a>. Thanks to <a href="https://prototypefund.de/project/portable-firewall-fuer-qubesos/">prototypefund</a> for sponsoring.</p>
-<p>In October 2020, we released <a href="https://mirage.io/blog/announcing-mirage-39-release">Mirage 3.9</a> with PVH virtualization mode (thanks to <a href="https://github.com/mato">mato</a>). There's still a <a href="https://github.com/mirage/qubes-mirage-firewall/issues/120">memory leak</a> to be investigated and fixed.</p>
+<p>In October 2020, we released <a href="https://mirageos.org/blog/announcing-mirage-39-release">Mirage 3.9</a> with PVH virtualization mode (thanks to <a href="https://github.com/mato">mato</a>). There's still a <a href="https://github.com/mirage/qubes-mirage-firewall/issues/120">memory leak</a> to be investigated and fixed.</p>
<h3 id="ipv6">IPv6</h3>
-<p>In December, with <a href="https://mirage.io/blog/announcing-mirage-310-release">Mirage 3.10</a> we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to <a href="https://github.com/nojb">nojb</a> for the initial code and <a href="https://github.com/MagnusS">MagnusS</a>.</p>
+<p>In December, with <a href="https://mirageos.org/blog/announcing-mirage-310-release">Mirage 3.10</a> we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to <a href="https://github.com/nojb">nojb</a> for the initial code and <a href="https://github.com/MagnusS">MagnusS</a>.</p>
<p>Turns out this blog, but also robur services, are now available via IPv6 :)</p>
<h3 id="albatross">Albatross</h3>
<p>Also in December, I pushed an initial release of <a href="https://github.com/roburio/albatross">albatross</a>, a unikernel orchestration system with remote access. <em>Deploy your unikernel via a TLS handshake -- the unikernel image is embedded in the TLS client certificates.</em></p>
@@ -412,7 +412,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
<h3 id="ca-certs">CA certs</h3>
<p>For several years I postponed the problem of how to actually use the operating system trust anchors for OCaml-TLS connections. Thanks to <a href="https://github.com/emillon">emillon</a> for initial code, there are now <a href="https://github.com/mirage/ca-certs">ca-certs</a> and <a href="https://github.com/mirage/ca-certs-nss">ca-certs-nss</a> opam packages (see <a href="https://discuss.ocaml.org/t/ann-ca-certs-and-ca-certs-nss">release announcement</a>) which fills this gap.</p>
<h2 id="unikernels">Unikernels</h2>
-<p>I developed several useful unikernels in 2020, and also pushed <a href="https://mirage.io/wiki/gallery">a unikernel gallery</a> to the Mirage website:</p>
+<p>I developed several useful unikernels in 2020, and also pushed <a href="https://mirageos.org/wiki/gallery">a unikernel gallery</a> to the Mirage website:</p>
<h3 id="traceroute-in-mirageos">Traceroute in MirageOS</h3>
<p>I already wrote about <a href="/Posts/Traceroute">traceroute</a> which traces the routing to a given remote host.</p>
<h3 id="unipi---static-website-hosting">Unipi - static website hosting</h3>
@@ -453,7 +453,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
<p>Mirage so far still uses ocamlbuild and ocamlfind for compiling the virtual machine binary. But the switch to dune is <a href="https://github.com/mirage/mirage/issues/1195">close</a>, a lot of effort has been done. This will make the developer experience of MirageOS much more smooth, with a per-unikernel monorepo workflow where you can push your changes to the individual libraries.</p>
<h2 id="footer">Footer</h2>
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
- urn:uuid:bc7675a5-47d0-5ce1-970c-01ed07fdf404 The road ahead for MirageOS in 2021 2021-11-19T18:04:52-00:00 hannes <p>A MirageOS unikernel which traces the path between itself and a remote host.</p>
+urn:uuid:bc7675a5-47d0-5ce1-970c-01ed07fdf404 The road ahead for MirageOS in 2021 2024-10-11T09:43:26-00:00 hannes <p>A MirageOS unikernel which traces the path between itself and a remote host.</p>
2020-06-24T10:38:10-00:00 <h2 id="traceroute">Traceroute</h2>
<p>Is a diagnostic utility which displays the route and measures transit delays of
packets across an Internet protocol (IP) network.</p>
@@ -786,7 +786,7 @@ let () =
</code></pre>
<p>And voila, that's all the code. If you copy it together (or download the two
files from <a href="https://github.com/roburio/traceroute">the GitHub repository</a>),
-and have OCaml, opam, and <a href="https://mirage.io/wiki/install">mirage (>= 3.8.0)</a> installed,
+and have OCaml, opam, and <a href="https://mirageos.org/wiki/install">mirage (>= 3.8.0)</a> installed,
you should be able to:</p>
<pre><code class="language-bash">$ mirage configure -t hvt
$ make depend
@@ -798,7 +798,7 @@ $ solo5-hvt --net:service=tap0 -- traceroute.hvt ...
<p>If you develop enhancements you'd like to share, please sent a pull request to the git repository.</p>
<p>Motivation for this traceroute unikernel was while talking with <a href="https://twitter.com/networkservice">Aaron</a> and <a href="https://github.com/phaer">Paul</a>, who contributed several patches to the IP stack which pass the ttl through.</p>
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
- urn:uuid:ed3036f6-83d2-5e80-b3da-4ccbedb5ae9e Traceroute 2021-11-19T18:04:52-00:00 hannes <p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
+urn:uuid:ed3036f6-83d2-5e80-b3da-4ccbedb5ae9e Traceroute 2024-10-11T09:43:26-00:00 hannes <p>A tutorial how to deploy authoritative name servers, let's encrypt, and updating entries from unix services.</p>
2019-12-23T21:30:53-00:00 <h2 id="goal">Goal</h2>
<p>Have your domain served by OCaml-DNS authoritative name servers. Data is stored in a git remote, and let's encrypt certificates can be requested to DNS. This software is deployed since more than two years for several domains such as <code>nqsb.io</code> and <code>robur.coop</code>. This present the authoritative server side, and certificate library of the OCaml-DNS implementation formerly known as <a href="/Posts/DNS">µDNS</a>.</p>
<h2 id="prerequisites">Prerequisites</h2>
@@ -973,7 +973,7 @@ personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv
10.0.42.2.10.0.42.3._transfer.mirage. DNSKEY 0 3 163 cDK6sKyvlt8UBerZlmxuD84ih2KookJGDagJlLVNo20=
git-repo> git commit -m "updates" . && git push
</code></pre>
-<p>Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see <a href="https://mirage.io/wiki/hello-world#Building-for-Another-Backend">other targets</a> for further information).</p>
+<p>Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see <a href="https://mirageos.org/wiki/hello-world#Building-for-Another-Backend">other targets</a> for further information).</p>
<pre><code class="language-shell"># back to primary
$ cd ../dns-primary-git
$ mirage configure -t hvt # or e.g. -t spt (and solo5-spt below)
@@ -1073,4 +1073,4 @@ $ ocertify 10.0.42.2 foo.mirage
<p>There are further steps to take, such as monitoring (<code>mirage configure --monitoring</code>), which use a second network interface for reporting syslog and metrics to telegraf / influx / grafana. Some DNS features are still missing, most prominently DNSSec.</p>
<p>I'd like to thank all people involved in this software stack, without other key components, including <a href="https://github.com/mirage/ocaml-git">git</a>, <a href="https://github.com/mirage/mirage-crypto">mirage-crypto</a>, <a href="https://github.com/mirage/awa-ssh">awa-ssh</a>, <a href="https://github.com/solo5/sol5">solo5</a>, <a href="https://github.com/mirage/mirage">mirage</a>, <a href="https://github.com/mmaker/ocaml-letsencrypt">ocaml-letsencrypt</a>, and more.</p>
<p>If you want to support our work on MirageOS unikernels, please <a href="https://robur.coop/Donate">donate to robur</a>. I'm interested in feedback, either via <a href="https://twitter.com/h4nnes">twitter</a>, <a href="https://mastodon.social/@hannesm">hannesm@mastodon.social</a> or via eMail.</p>
- urn:uuid:e3d4fd9e-e379-5c86-838e-46034ddd435d Deploying authoritative OCaml-DNS servers as MirageOS unikernels 2023-03-02T17:20:44-00:00 hannes urn:uuid:e3d4fd9e-e379-5c86-838e-46034ddd435d Deploying authoritative OCaml-DNS servers as MirageOS unikernels 2024-10-11T09:43:26-00:00 hannes