update links
This commit is contained in:
parent
1ecc61c2d0
commit
c59a208400
21 changed files with 40 additions and 47 deletions
2
About
2
About
|
|
@ -88,7 +88,7 @@ faster runtime, and libraries).
|
|||
|
||||
After I finished my PhD, I decided to slack off for some time to make decent
|
||||
espresso. I ended up spending the winter (beginning of 2014) in Mirleft,
|
||||
Morocco. A good friend of mine pointed me to [MirageOS](https://mirage.io), a
|
||||
Morocco. A good friend of mine pointed me to [MirageOS](https://mirageos.org), a
|
||||
clean-slate operating system written in the high-level language [OCaml](https://ocaml.org). I got
|
||||
hooked pretty fast, after some experience with LISP machines I imagined a modern
|
||||
OS written in a single functional programming language.
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ That's it. Pretty straightforward.
|
|||
|
||||
Back in 2008, together with Andreas Bogk, we just used a hash table and installed expiration and retransmission timers when needed. Certainly timers sometimes needed to be cancelled, and testing the code was cumbersome. It were only [250 lines of Dylan code](https://github.com/dylan-hackers/network-night-vision/blob/master/network/ip-stack/layers/network/arp/arp.dylan) plus some [wire format definition](https://github.com/dylan-hackers/network-night-vision/blob/master/protocols/ipv4.dylan).
|
||||
|
||||
Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated. The code should not contain surprises, but straightforward to understand. The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should. Once this is achieved, testing is straightforward. If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused. It can also be repurposed to run as a test oracle. You can read more marketing of this style in our [Usenix security paper](https://usenix15.nqsb.io).
|
||||
Nowadays, after some years of doing formal verification and typed functional programming, I try to have effects, including mutable state, isolated and explicitly annotated. The code should not contain surprises, but straightforward to understand. The core protocol logic should not be convoluted with side effects, rather a small wrapper around it should. Once this is achieved, testing is straightforward. If the fashion of the asynchronous task library changes (likely with OCaml multicore), the core logic can be reused. It can also be repurposed to run as a test oracle. You can read more marketing of this style in our [Usenix security paper](https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak).
|
||||
|
||||
My proposed style and hash tables are not good friends, since hash tables in OCaml are imperative structures. Instead, a *Map* ([documentation](http://caml.inria.fr/pub/docs/manual-ocaml/libref/Map.html)) is a functional data structure for associating keys with values. Its underlying data structure is a balanced binary tree.
|
||||
|
||||
|
|
|
|||
|
|
@ -75,5 +75,5 @@ I'm interested in feedback, either via
|
|||
- Canopy uses a [map instead of a hashtable](https://github.com/Engil/Canopy/issues/30#issuecomment-215010365), [tags](https://hannes.nqsb.io/tags) now contains a list of tags ([PR here](https://github.com/Engil/Canopy/pull/39)), both thanks to voila! I also use the [new CSS](https://github.com/Engil/Canopy/pull/38) from Engil
|
||||
- There is a [CVE for OCaml <=4.03](http://www.openwall.com/lists/oss-security/2016/04/29/1)
|
||||
- [Mirage 2.9.0](https://github.com/mirage/mirage/pull/534) was released, which integrates support of the logs library (now already used in [mirage-net-xen](https://github.com/mirage/mirage-net-xen/pull/43) and [mirage-tcpip](https://github.com/mirage/mirage-tcpip/pull/199))
|
||||
- This blog post has an accompanied [MirageOS security advisory](https://mirage.io/blog/MSA00)
|
||||
- This blog post has an accompanied [MirageOS security advisory](https://mirageos.org/blog/MSA00)
|
||||
- cfcs documented some [basic unikernels](https://github.com/cfcs/mirage-examples)
|
||||
|
|
|
|||
|
|
@ -10,8 +10,8 @@ abstract: New year brings new possibilities and a new environment. I've been wo
|
|||
At the end of 2017, I resigned from my PostDoc position at University of
|
||||
Cambridge (in the [rems](https://www.cl.cam.ac.uk/~pes20/rems/) project). Early
|
||||
December 2017 I organised the [4th MirageOS hack
|
||||
retreat](https://mirage.io/blog/2017-winter-hackathon-roundup), with which I'm
|
||||
very satisfied. In March 2018 the [5th retreat](http://retreat.mirage.io) will
|
||||
retreat](https://mirageos.org/blog/2017-winter-hackathon-roundup), with which I'm
|
||||
very satisfied. In March 2018 the [5th retreat](http://retreat.mirageos.org) will
|
||||
happen (please sign up!).
|
||||
|
||||
In 2018 I moved to Berlin and started to work for the (non-profit) [Center for
|
||||
|
|
|
|||
|
|
@ -244,7 +244,7 @@ personal._update.mirage. DNSKEY 0 3 163 kJJqipaQHQWqZL31Raar6uPnepGFIdtpjkXot9rv
|
|||
git-repo> git commit -m "updates" . && git push
|
||||
```
|
||||
|
||||
Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see [other targets](https://mirage.io/wiki/hello-world#Building-for-Another-Backend) for further information).
|
||||
Ok, the git repository is ready, now we need to compile the unikernels for the virtualisation target (see [other targets](https://mirageos.org/wiki/hello-world#Building-for-Another-Backend) for further information).
|
||||
|
||||
```shell
|
||||
# back to primary
|
||||
|
|
|
|||
2
Posts/EC
2
Posts/EC
|
|
@ -11,7 +11,7 @@ Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secu
|
|||
|
||||
## Elliptic curve cryptography
|
||||
|
||||
[Since May 2020](https://mirage.io/blog/tls-1-3-mirageos), our [OCaml-TLS](https://usenix15.nqsb.io) stack supports TLS 1.3 (since tls version 0.12.0 on opam).
|
||||
[Since May 2020](https://mirageos.org/blog/tls-1-3-mirageos), our [OCaml-TLS](https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak) stack supports TLS 1.3 (since tls version 0.12.0 on opam).
|
||||
|
||||
TLS 1.3 requires elliptic curve cryptography - which was not available in [mirage-crypto](https://github.com/mirage/mirage-crypto) (the maintained fork of [nocrypto](https://github.com/mirleft/ocaml-nocrypto)).
|
||||
|
||||
|
|
|
|||
|
|
@ -13,11 +13,11 @@ As described in an [earlier post](/Posts/OperatingSystem), MirageOS is a library
|
|||
|
||||
The abstraction over concrete implementation of e.g. the network stack is done by providing a module signature in the [mirage-types](https://github.com/mirage/mirage/tree/master/types) package. The socket-based network stack, the tap device based network stack, and the Xen virtual network device based network stack implement this signature (depending on other module signatures). The unikernel contains code which applies those dependent modules to instantiate a custom-tailored network stack for the specific configuration. A developer should only describe what their requirements are, the user who wants to deploy it should provide the concrete configuration. And the developer should not need to manually instantiate the network stack for all possible configurations, this is what the mirage tool should embed.
|
||||
|
||||
Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code. This turned out to be error prone. In 2015 [Drup](https://github.com/Drup) developed [Functoria](https://github.com/mirage/functoria), a domain-specific language (DSL) to organize functor applications, primarily for MirageOS. It has been introduced in [a blog post](https://mirage.io/blog/introducing-functoria). It is not limited to MirageOS (although this is the primary user right now).
|
||||
Initially, MirageOS contained an adhoc system which relied on concatenation of strings representing OCaml code. This turned out to be error prone. In 2015 [Drup](https://github.com/Drup) developed [Functoria](https://github.com/mirage/functoria), a domain-specific language (DSL) to organize functor applications, primarily for MirageOS. It has been introduced in [a blog post](https://mirageos.org/blog/introducing-functoria). It is not limited to MirageOS (although this is the primary user right now).
|
||||
|
||||
Functoria has been included in MirageOS since its [2.7.0 release](https://github.com/mirage/mirage/releases/tag/v2.7.0) at the end of February 2016. Functoria provides support for command line arguments which can then either be passed at configuration time or at boot time to the unikernel (such as IP address configuration) using the [cmdliner library](http://erratique.ch/software/cmdliner) underneath (and includes dynamic man pages, help, sensible command line parsing, and even visualisation (`mirage describe`) of the configuration and data dependencies).
|
||||
|
||||
I won't go into details about command line arguments in here, please have a look at the [functoria blog post](https://mirage.io/blog/introducing-functoria) in case you're interested. Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel ([running here](http://marrakech2016.mirage.io), [source](https://github.com/mirage/marrakech2016)). Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown. Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).
|
||||
I won't go into details about command line arguments in here, please have a look at the [functoria blog post](https://mirageos.org/blog/introducing-functoria) in case you're interested. Instead, I'll describe how to define a Functoria device which inserts content as code at configuration time into a MirageOS unikernel ([running here](http://retreat.mirageos.org), [source](https://github.com/mirage/retreat.mirageos.org)). Using this approach, no external data (using crunch or a file system image) is needed, while the content can still be modified using markdown. Also, no markdown to HTML converter is needed at runtime, but this step is completely done at compile time (the result is a small (still too large) unikernel, 4.6MB).
|
||||
|
||||
### Unikernel
|
||||
|
||||
|
|
@ -157,5 +157,5 @@ I'm interested in feedback, either via
|
|||
- Canopy updated and created timestamps (for [irmin-0.10](https://github.com/Engil/Canopy/pull/48) and [irmin-0.11](https://github.com/Engil/Canopy/pull/43))
|
||||
- another [resource leak in mirage-http](https://github.com/mirage/mirage-http/pull/24)
|
||||
- [mirage-platform now has 4.03 support](https://github.com/mirage/mirage-platform/pull/165) and [strtod](https://github.com/mirage/mirage-platform/issues/118) (finally :)
|
||||
- [blog posts about retreat in marrakech](https://mirage.io/blog/2016-spring-hackathon)
|
||||
- [blog posts about retreat in marrakech](https://mirageos.org/blog/2016-spring-hackathon)
|
||||
- [syndic 1.5.0 release](https://github.com/Cumulus/Syndic) now using ptime instead of calendar
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ abstract: implement it once to know you can do it. implement it a second time a
|
|||
|
||||
|
||||
|
||||
Back in 2014, when we implemented [TLS](https://nqsb.io) in OCaml, at some point
|
||||
Back in 2014, when we implemented [TLS](https://github.com/mirleft/ocaml-tls) in OCaml, at some point
|
||||
I was bored with TLS. I usually need at least two projects (but not more than 5) at the same time to
|
||||
procrastinate the one I should do with the other one - it is always more fun to
|
||||
do what you're not supposed to do. I started to implement another security
|
||||
|
|
@ -169,7 +169,7 @@ friends logged in multiple times with wrongly set priorities - and end-to-end
|
|||
encryption. I don't need inline HTML, avatar images, my currently running
|
||||
music, leaking timezone information, etc. I explicitly don't want to import any
|
||||
private key material from other clients and libraries, because I want to ensure
|
||||
that the key was generated by a good random number generator (read [David's blog article](https://mirage.io/blog/mirage-entropy) on randomness and entropy).
|
||||
that the key was generated by a good random number generator (read [David's blog article](https://mirageos.org/blog/mirage-entropy) on randomness and entropy).
|
||||
|
||||
The security story is crucial: always do strict certificate validation, fail
|
||||
hard, make it noticable by the user if they're doing insecure communication.
|
||||
|
|
@ -297,7 +297,7 @@ But there's some mess to clean up:
|
|||
|
||||
1. The [XMPP library](https://github.com/ermine/xmpp) makes heavy use of
|
||||
functors (to abstract over the concrete IO, etc.), and embeds IO deep inside it.
|
||||
I do prefer (see e.g. [our TLS paper](https://usenix15.nqsb.io), or [my ARP post](/Posts/ARP)) these days to have a pure interface for
|
||||
I do prefer (see e.g. [our TLS paper](https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak), or [my ARP post](/Posts/ARP)) these days to have a pure interface for
|
||||
the protocol implementation, providing explicit input (state, event, data), and
|
||||
output (state, action, potentially data to send on network, potentially data to
|
||||
process by the application). The [sasl implementation](https://github.com/hannesm/xmpp/blob/eee18bd3dd343550169969c0b45548eafd51cfe1/src/sasl.ml)
|
||||
|
|
|
|||
|
|
@ -33,11 +33,11 @@ Dating back to 2016, at the [TRON](https://www.ndss-symposium.org/ndss2016/tron-
|
|||
|
||||
The [MirageOS based Qubes firewall](https://github.com/mirage/qubes-mirage-firewall/) is the most widely used MirageOS unikernel. And it got major updates: in May [Steffi](https://github.com/linse) [announced](https://groups.google.com/g/qubes-users/c/Xzplmkjwa5Y) her and [Mindy's](https://github.com/yomimono) work on improving it for Qubes 4.0 - including [dynamic firewall rules via QubesDB](https://www.qubes-os.org/doc/vm-interface/#firewall-rules-in-4x). Thanks to [prototypefund](https://prototypefund.de/project/portable-firewall-fuer-qubesos/) for sponsoring.
|
||||
|
||||
In October 2020, we released [Mirage 3.9](https://mirage.io/blog/announcing-mirage-39-release) with PVH virtualization mode (thanks to [mato](https://github.com/mato)). There's still a [memory leak](https://github.com/mirage/qubes-mirage-firewall/issues/120) to be investigated and fixed.
|
||||
In October 2020, we released [Mirage 3.9](https://mirageos.org/blog/announcing-mirage-39-release) with PVH virtualization mode (thanks to [mato](https://github.com/mato)). There's still a [memory leak](https://github.com/mirage/qubes-mirage-firewall/issues/120) to be investigated and fixed.
|
||||
|
||||
### IPv6
|
||||
|
||||
In December, with [Mirage 3.10](https://mirage.io/blog/announcing-mirage-310-release) we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to [nojb](https://github.com/nojb) for the initial code and [MagnusS](https://github.com/MagnusS).
|
||||
In December, with [Mirage 3.10](https://mirageos.org/blog/announcing-mirage-310-release) we got the IPv6 code up and running. Now MirageOS unikernels have a dual stack available, besides IPv4-only and IPv6-only network stacks. Thanks to [nojb](https://github.com/nojb) for the initial code and [MagnusS](https://github.com/MagnusS).
|
||||
|
||||
Turns out this blog, but also robur services, are now available via IPv6 :)
|
||||
|
||||
|
|
@ -53,7 +53,7 @@ For several years I postponed the problem of how to actually use the operating s
|
|||
|
||||
## Unikernels
|
||||
|
||||
I developed several useful unikernels in 2020, and also pushed [a unikernel gallery](https://mirage.io/wiki/gallery) to the Mirage website:
|
||||
I developed several useful unikernels in 2020, and also pushed [a unikernel gallery](https://mirageos.org/wiki/gallery) to the Mirage website:
|
||||
|
||||
### Traceroute in MirageOS
|
||||
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ on a regular schedule. Daniel wrote [guidelines](http://erratique.ch/software/r
|
|||
The [opam repository](https://opam.ocaml.org/packages/) contains over 1000
|
||||
libraries. The quality varies, I personally like the small libraries done by
|
||||
[Daniel Bünzli](http://erratique.ch/software), as well as our
|
||||
[nqsb](https://nqsb.io) libraries (see [mirleft org](https://github.com/mirleft)),
|
||||
not quite so broken libraries (see [mirleft org](https://github.com/mirleft)),
|
||||
[notty](https://github.com/pqwy/notty).
|
||||
A concise library (not much code),
|
||||
including tests, documentation, etc. is
|
||||
|
|
@ -130,7 +130,7 @@ which watches lots of MirageOS-related repositories.
|
|||
I hope I gave some insight into OCaml, and why I currently enjoy it. A longer read on applicability of OCaml is our Usenix 2015 paper
|
||||
[Not-quite-so-broken TLS: lessons in re-engineering a security protocol
|
||||
specification and
|
||||
implementation](https://nqsb.io/nqsbtls-usenix-security15.pdf). I'm interested in feedback, either via
|
||||
implementation](https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak). I'm interested in feedback, either via
|
||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
||||
|
||||
## Other updates in the MirageOS ecosystem
|
||||
|
|
|
|||
|
|
@ -27,11 +27,11 @@ Opam, after initialisation, downloads the `index.tar.gz` from `https://opam.ocam
|
|||
|
||||
According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses `opam admin index` periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was [a reported downtime](https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/).
|
||||
|
||||
Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS [hack retreats](http://retreat.mirage.io) in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).
|
||||
Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS [hack retreats](http://retreat.mirageos.org) in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).
|
||||
|
||||
# Re-developing "opam admin create" as MirageOS unikernel
|
||||
|
||||
The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.coop/robur/opam-mirror) as a [MirageOS unikernel](https://mirage.io). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.
|
||||
The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.coop/robur/opam-mirror) as a [MirageOS unikernel](https://mirageos.org). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.
|
||||
|
||||
Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.
|
||||
|
||||
|
|
|
|||
|
|
@ -77,7 +77,7 @@ proofs of full functional correctness properties.
|
|||
### MirageOS
|
||||
|
||||
At the end of 2013, David pointed me to
|
||||
[MirageOS](https://mirage.io), an operating system developed from scratch in the
|
||||
[MirageOS](https://mirageos.org), an operating system developed from scratch in the
|
||||
functional and statically typed language [OCaml](https://ocaml.org). I've not
|
||||
used much OCaml before, but some other functional programming languages.
|
||||
Since then, I spend nearly every day on developing OCaml libraries (with varying success on being happy
|
||||
|
|
@ -176,5 +176,5 @@ how MirageOS fits into the picture. I'm interested in feedback, either via
|
|||
- I wrote a clean patch to [serve via TLS](https://github.com/Engil/Canopy/pull/15) (including [HSTS header](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security) and redirecting HTTP (moved permanently) to HTTPS)
|
||||
- I found a memory leak in the [mirage-http](https://github.com/mirage/mirage-http/pull/23) library
|
||||
- I was travelling
|
||||
- good news: it now works on Xen, and there is [an atom feed](https://hannes.nqsb.io/atom)
|
||||
- good news: it now works on Xen, and there is [an atom feed](https://hannes.robur.coop/atom)
|
||||
- life of an "eat your own dogfood" full stack engineer ;)
|
||||
|
|
|
|||
10
Posts/Pinata
10
Posts/Pinata
|
|
@ -8,12 +8,12 @@ abstract: More than three years ago we launched our Bitcoin Piñata as a transpa
|
|||
## History
|
||||
|
||||
On February 10th 2015 David Kaloper-Meršinjak and Hannes Mehnert
|
||||
[launched](https://mirage.io/announcing-bitcoin-pinata) (read also [Amir's
|
||||
[launched](https://mirageos.org/announcing-bitcoin-pinata) (read also [Amir's
|
||||
description](http://amirchaudhry.com/bitcoin-pinata)) our [bug bounty
|
||||
program](https://en.wikipedia.org/wiki/Bug_bounty_program) in the form of our
|
||||
[Bitcoin Piñata](http://ownme.ipredator.se) MirageOS unikernel. Thanks again to
|
||||
[IPredator](https://ipredator.se) for both hosting our services and lending us
|
||||
the 10 Bitcoins! We [analysed](https://mirage.io/blog/bitcoin-pinata-results) a
|
||||
the 10 Bitcoins! We [analysed](https://mirageos.org/blog/bitcoin-pinata-results) a
|
||||
bit more in depth after running it for five months. Mindy recently wrote about
|
||||
[whacking the Bitcoin
|
||||
Piñata](https://somerandomidiot.com/blog/2018/04/17/whacking-the-bitcoin-pinata/).
|
||||
|
|
@ -24,11 +24,11 @@ The 10 Bitcoin in the Piñata were fluctuating in price over time, at peak worth
|
|||
|
||||
From the start of the Piñata project, we published the [source code](https://github.com/mirleft/btc-pinata), the virtual machine image, and the versions of the used libraries in a git repository. Everybody could develop their exploits locally before launching them against our Piñata. The Piñata provides TLS endpoints, which require private keys and certificates. These are generated by the Piñata at startup, and the secret for the Bitcoin wallet is provided as a command line argument.
|
||||
|
||||
Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with [solo5](https://github.com/solo5/solo5), and in December 2017 it was migrated to native BHyve ([using `ukvm-bin` and solo5](/Posts/Solo5)). We also changed the Piñata code to accomodate for updates, such as the [MirageOS 3.0 release](https://mirage.io/blog/announcing-mirage-30-release), and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).
|
||||
Initially the Piñata was deployed on a Linux/Xen machine, later it was migrated to a FreeBSD host using BHyve and VirtIO with [solo5](https://github.com/solo5/solo5), and in December 2017 it was migrated to native BHyve ([using `ukvm-bin` and solo5](/Posts/Solo5)). We also changed the Piñata code to accomodate for updates, such as the [MirageOS 3.0 release](https://mirageos.org/blog/announcing-mirage-30-release), and the discontinuation of floating point numbers for timestamps (asn1-combinators 0.2.0, x509 0.6.0, tls 0.9.0).
|
||||
|
||||
## Motivation
|
||||
|
||||
We built the Piñata for many purposes: to attract security professionals to evaluate our [from-scratch developed TLS stack](https://mirage.io/blog/introducing-ocaml-tls), to gather empirical data for our [Usenix Security 15 paper](https://usenix15.nqsb.io), and as an improvement to current bug bounty programs.
|
||||
We built the Piñata for many purposes: to attract security professionals to evaluate our [from-scratch developed TLS stack](https://mirageos.org/blog/introducing-ocaml-tls), to gather empirical data for our [Usenix Security 15 paper](https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak), and as an improvement to current bug bounty programs.
|
||||
|
||||
Most bug bounty programs require communication via forms and long wait times for
|
||||
human experts to evaluate the potential bug. This evaluation is subjective,
|
||||
|
|
@ -52,7 +52,7 @@ In April 2016 we stumbled upon an [information disclosure in the virtual network
|
|||
device driver for Xen in MirageOS](/Posts/BadRecordMac). Given enough
|
||||
bandwidth, this could have been used to access the private wallet key. We
|
||||
upgraded the Piñata and released the [MirageOS Security Advisory
|
||||
00](https://mirage.io/blog/MSA00).
|
||||
00](https://mirageos.org/blog/MSA00).
|
||||
|
||||
We analysed the Piñata's access logs to the and bucketed them into website traffic and bounty connections. We are still wondering what happened in July 2015 and July 2017 where the graph shows spikes. Could it be a presentation mentioning the Piñata, or a new automated tool which tests for TLS vulnerabilities, or an increase in market price for Bitcoins?
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ abstract: MirageOS unikernels are reproducible :)
|
|||
|
||||
## Reproducible builds summit
|
||||
|
||||
I'm just back from the [Reproducible builds summit 2019](https://reproducible-builds.org/events/Marrakesh2019/). In 2018, several people developing [OCaml](https://ocaml.org) and [opam](https://opam.ocaml.org) and [MirageOS](https://mirage.io), attended [the Reproducible builds summit in Paris](https://reproducible-builds.org/events/paris2018/). The notes from last year on [opam reproducibility](https://reproducible-builds.org/events/paris2018/report/#Toc11410_331763073) and [MirageOS reproducibility](https://reproducible-builds.org/events/paris2018/report/#Toc11681_331763073) are online. After last years workshop, Raja started developing the opam reproducibilty builder [orb](https://github.com/rjbou/orb), which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.
|
||||
I'm just back from the [Reproducible builds summit 2019](https://reproducible-builds.org/events/Marrakesh2019/). In 2018, several people developing [OCaml](https://ocaml.org) and [opam](https://opam.ocaml.org) and [MirageOS](https://mirageos.org), attended [the Reproducible builds summit in Paris](https://reproducible-builds.org/events/paris2018/). The notes from last year on [opam reproducibility](https://reproducible-builds.org/events/paris2018/report/#Toc11410_331763073) and [MirageOS reproducibility](https://reproducible-builds.org/events/paris2018/report/#Toc11681_331763073) are online. After last years workshop, Raja started developing the opam reproducibilty builder [orb](https://github.com/rjbou/orb), which I extended at and after this years summit. This year before and after the facilitated summit there were hacking days, which allowed further interaction with participants, writing some code and conduct experiments. I had this year again an exciting time at the summit and hacking days, thanks to our hosts, organisers, and all participants.
|
||||
|
||||
## Goal
|
||||
|
||||
|
|
@ -17,7 +17,7 @@ A [checklist](https://reproducible-builds.org/docs/test-bench/) of potential thi
|
|||
|
||||
Reproducibility is a precondition for trustworthy binaries. See [why does it matter](https://reproducible-builds.org/#why-does-it-matter). If there are no instructions how to get from the published sources to the exact binary, why should anyone trust and use the binary which claims to be the result of the sources? It may as well contain different code, including a backdoor, bitcoin mining code, outputting the wrong results for specific inputs, etc. Reproducibility does not imply the software is free of security issues or backdoors, but instead of a audit of the binary - which is tedious and rarely done - the source code can be audited - but the toolchain (compiler, linker, ..) used for compilation needs to be taken into account, i.e. trusted or audited to not be malicious. **I will only ever publish binaries if they are reproducible**.
|
||||
|
||||
My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of [MirageOS unikernels](https://mirage.io) -- a unikernel is a statically linked ELF binary to be run as Unix process or [virtual machine](https://github.com/solo5/solo5). MirageOS heavily uses [OCaml](https://ocaml.org) and [opam](https://opam.ocaml.org), the OCaml package manager, and is an opam package itself. Thus, *checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package*.
|
||||
My main interest at the summit was to enhance existing tooling and conduct some experiments about the reproducibility of [MirageOS unikernels](https://mirageos.org) -- a unikernel is a statically linked ELF binary to be run as Unix process or [virtual machine](https://github.com/solo5/solo5). MirageOS heavily uses [OCaml](https://ocaml.org) and [opam](https://opam.ocaml.org), the OCaml package manager, and is an opam package itself. Thus, *checking reproducibility of a MirageOS unikernel is the same problem as checking reproducibility of an opam package*.
|
||||
|
||||
## Reproducible builds with opam
|
||||
|
||||
|
|
|
|||
|
|
@ -5,11 +5,11 @@ tags: mirageos, community
|
|||
abstract: My involvement and experience of the MirageOS retreat 2024
|
||||
---
|
||||
|
||||
End of April I spent a week in Marrakech to work on MirageOS at our annual [retreat](https://retreat.mirage.io). This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
|
||||
End of April I spent a week in Marrakech to work on MirageOS at our annual [retreat](https://retreat.mirageos.org). This was the 12th time when it actually took place, and it was amazing. We were a total of 17 people attending, of which 5 people used ground transportation. I have respect for them, to take the 3 or more day trip from Berlin or Glasgow or Paris or whereever to Marrakech - with a ferry in between (which availability depends on the wind etc.). This time, I didn't take ground transportation since I had appointments just before and just after the retreat nearby Berlin where I had to be in person. The food an weather was really nice and motivating.
|
||||
|
||||
Of the 17 people there were lots of new faces. It is really nice to see an active community which is welcoming to new people, and also that there's change: a lot of fresh ideas are brought by these people and we rethink what we are doing and why.
|
||||
|
||||
Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our [website](https://mirage.io) only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
|
||||
Our daily routine included breakfast, circle, lunch, dinner, (sometimes) presentations. There wasn't more formal structure, and there was no need for it since people kept themselves busy working on various projects. We had a bigger discussion about "deploying MirageOS", which is a reoccurring theme. Even running a MirageOS unikernel on a laptop is for people who are not very familiar with networking pretty hard. In addition, our [website](https://mirageos.org) only briefly described network setup, and included DHCP very early (which is overwhelming to setup and use on your laptop).
|
||||
|
||||
I went to the retreat with some ideas in my head what I want to achieve (for example, replace all the bigarray allocations by bytes & string), but the discussion about deployment made me prioritise this more.
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ Robur is part of the non-profit company [Center for the Cultivation of Technolog
|
|||
|
||||
## Deploying MirageOS unikernels
|
||||
|
||||
While several examples are running since years (the [MirageOS website](https://mirage.io), [Bitcoin Piñata](http://ownme.ipredator.se), [TLS demo server](https://tls.nqsb.io), etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.
|
||||
While several examples are running since years (the [MirageOS website](https://mirageos.org), [Bitcoin Piñata](http://ownme.ipredator.se), TLS demo server, etc.), and some shell-scripts for cloud providers are floating around, it is not (yet) streamlined.
|
||||
|
||||
Service deployment is complex: you have to consider its configuration, exfiltration of logs and metrics, provisioning with valid key material (TLS certificate, hmac shared secret) and authenticators (CA certificate, ssh key fingerprint). Instead of requiring millions lines of code during orchestration (such as Kubernetes), creating the images (docker), or provisioning (ansible), why not minimise the required configuration and dependencies?
|
||||
|
||||
|
|
|
|||
|
|
@ -83,7 +83,7 @@ encryption?). I'll cover both topics in the rest of this article.
|
|||
### MirageOS integration
|
||||
|
||||
Since Mirage3, syslog is integrated (see
|
||||
[documentation](http://docs.mirage.io/mirage/Mirage/index.html#type-syslog_config)).
|
||||
[documentation](http://docs.mirageos.org/mirage/Mirage/index.html#type-syslog_config)).
|
||||
Some additions to your `config.ml` are needed, see [ns
|
||||
example](https://github.com/hannesm/ns.nqsb.io/blob/master/config.ml) or
|
||||
[marrakech
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ In 2012, I attended ICFP in Copenhagen while a PhD student at ITU Copenhagen. Th
|
|||
|
||||
To dive a bit more into [network semantics](https://www.cl.cam.ac.uk/~pes20/Netsem/), the work done on TCP by Peter Sewell, et al., is a formal specification (or a model) of TCP/IP and the Unix sockets API developed in HOL4. It is a label transition system with nondeterministic choices, and the model itself is executable. It has been validated with the real world by collecting thousands of traces on Linux, Windows, and FreeBSD, which have been checked by the model for validity. This copes with the different implementations of the English prose of the RFCs. The network semantics research found several issues in existing TCP stacks and reported them upstream to have them fixed (though, there still is some special treatment, e.g., for the "BSD listen bug").
|
||||
|
||||
In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the [journal paper](https://dl.acm.org/doi/10.1145/3243650) ([author's copy](http://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf)) published. At the same time, the [MirageOS](https://mirage.io) melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.
|
||||
In 2014, I joined Peter's research group in Cambridge to continue their work on the model: updating to more recent versions of HOL4 and PolyML, revising the test system to use DTrace, updating to a more recent FreeBSD network stack (from FreeBSD 4.6 to FreeBSD 10), and finally getting the [journal paper](https://dl.acm.org/doi/10.1145/3243650) ([author's copy](http://www.cl.cam.ac.uk/~pes20/Netsem/paper3.pdf)) published. At the same time, the [MirageOS](https://mirageos.org) melting pot was happening at University of Cambridge, where I contributed with David OCaml-TLS and other things.
|
||||
|
||||
My intention was to understand TCP better and use the specification as a basis for a TCP stack for MirageOS. The [existing one](https://github.com/mirage/mirage-tcpip) (which is still used) has technical debt: a high issue to number of lines ratio. The Lwt monad is ubiquitous, which makes testing and debugging pretty hard, and also utilising multiple cores with OCaml Multicore won't be easy. Plus it has various resource leaks, and there is no active maintainer. But honestly, it works fine on a local network, and with well-behaved traffic. It doesn't work that well on the wild Internet with a variety of broken implementations. Apart from resource leakage, which made me implement things such as restart-on-failure in [Albatross](https://github.com/robur-coop/albatross), there are certain connection states which will never be exited.
|
||||
|
||||
|
|
@ -25,7 +25,7 @@ My intention was to understand TCP better and use the specification as a basis f
|
|||
|
||||
Back in Cambridge, I didn't manage to write a TCP stack based on the model, but in 2019, I restarted that work and got µTCP (the formal model manually translated to OCaml) to compile and do TCP session setup and teardown. Since it was a model that uses nondeterminism, this couldn't be translated one-to-one into an executable program, but there are places where decisions have to be made. Due to other projects, I worked only briefly in 2021 and 2022 on µTCP, but finally in the Summer of 2023, I motivated myself to push µTCP into a usable state. So far I've spend 25 days in 2023 on µTCP. Thanks to [Tarides](https://tarides.com) for supporting my work.
|
||||
|
||||
Since late August, we have been running some unikernels using µTCP, e.g., the [retreat](https://retreat.mirage.io) website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.
|
||||
Since late August, we have been running some unikernels using µTCP, e.g., the [retreat](https://retreat.mirageos.org) website. This allows us to observe µTCP and find and solve issues that occur in the real world. It turned out that the model is not always correct (i.e., there is no retransmit timer in the close wait state, which avoids proper session teardowns). We report statistics about how many TCP connections are in which state to an Influx time series database and view graphs rendered by Grafana. If there are connections that are stuck for multiple hours, this indicates a resource leak that should be addressed. Grafana was tremendously helpful to find out where to look for resource leaks. Still, there's work to understand the behaviour, look at what the model does, what µTCP does, what the RFC says, and eventually what existing deployed TCP stacks do.
|
||||
|
||||
# The secondary nameserver issue
|
||||
|
||||
|
|
|
|||
|
|
@ -362,7 +362,7 @@ let () =
|
|||
|
||||
And voila, that's all the code. If you copy it together (or download the two
|
||||
files from [the GitHub repository](https://github.com/roburio/traceroute)),
|
||||
and have OCaml, opam, and [mirage (>= 3.8.0)](https://mirage.io/wiki/install) installed,
|
||||
and have OCaml, opam, and [mirage (>= 3.8.0)](https://mirageos.org/wiki/install) installed,
|
||||
you should be able to:
|
||||
```bash
|
||||
$ mirage configure -t hvt
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ Once a private and public key pair is generated (doesn't matter whether it is pl
|
|||
|
||||
## OCaml ecosystem evolving
|
||||
|
||||
More than 5 years ago David Kaloper and I [released the initial ocaml-x509](https://mirage.io/blog/introducing-x509) package as part of our [TLS stack](https://nqsb.io), which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in [RFC 5280](https://tools.ietf.org/html/rfc6125)). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's [asn1-combinators](https://github.com/mirleft/ocaml-asn1-combinators) library changed much over time.
|
||||
More than 5 years ago David Kaloper and I [released the initial ocaml-x509](https://mirageos.org/blog/introducing-x509) package as part of our [TLS stack](https://github.com/mirleft/ocaml-tls), which contained code for decoding and encoding certificates, and path validation of a certificate chain (as described in [RFC 5280](https://tools.ietf.org/html/rfc6125)). The validation logic and the decoder/encoder, based on the ASN.1 grammar specified in the RFC, implemented using David's [asn1-combinators](https://github.com/mirleft/ocaml-asn1-combinators) library changed much over time.
|
||||
|
||||
The OCaml ecosystem evolved over the years, which lead to some changes:
|
||||
- Camlp4 deprecation - we used camlp4 for stream parsers of PEM-encoded certificates, and sexplib.syntax to derive s-expression decoders and encoders;
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ abstract: building a simple website
|
|||
|
||||
Our task is to build a small unikernel which provides a project website. On our way we will wade through various layers using code examples. The website itself contains a few paragraphs of text, some link lists, and our published papers in pdf form.
|
||||
|
||||
*Spoiler alert* final result can be seen [here](https://nqsb.io), the full code [here](https://github.com/mirleft/nqsb.io).
|
||||
*Spoiler alert* final result can be seen (no longer available), the full code [here](https://github.com/mirleft/nqsb.io).
|
||||
|
||||
## A first idea
|
||||
|
||||
|
|
@ -244,7 +244,7 @@ let start stack keys kv =
|
|||
S.listen stack
|
||||
```
|
||||
|
||||
That's it, the [nqsb.io](https://nqsb.io) contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a `301 Moved permanently` HTTP status code).
|
||||
That's it, it contains slightly more code to log onto a console, and to redirect requests on port 80 (HTTP) to port 443 (by signaling a `301 Moved permanently` HTTP status code).
|
||||
|
||||
## Conclusion
|
||||
|
||||
|
|
@ -254,7 +254,7 @@ A comparison using Firefox builtin network diagnostics shows that the waiting be
|
|||
|
||||
We do not render HTML for each request, we do not splice data together, we *don't even read the client request*. And I'm sure we can improve the performance even more by profiling.
|
||||
|
||||
We saw a journey from typed XML over key value stores, HTTP, TLS, and HTTPS. The actual application code of our unikernel serving [nqsb,io](https://nqsb.io) is less than 100 lines of OCaml. We used MirageOS for our minimal HTTPS website, serving a *single resource per hostname*. We depend (directly) on the tyxml library, the mirage tool and network stack, and the tls library. That's it.
|
||||
We saw a journey from typed XML over key value stores, HTTP, TLS, and HTTPS. The actual application code of our unikernel serving nqsb.io is less than 100 lines of OCaml. We used MirageOS for our minimal HTTPS website, serving a *single resource per hostname*. We depend (directly) on the tyxml library, the mirage tool and network stack, and the tls library. That's it.
|
||||
|
||||
There is a long list of potential features, such as full HTTP protocol compliance (caching, favicon, ...), logging, natively getting let's encrypt certificates -- but in the web out there it is sufficient to get picked up by search engines, and the maintenance is marginal.
|
||||
|
||||
|
|
@ -262,10 +262,3 @@ For a start in MirageOS unikernels, look into our [mirage-skeleton](https://gith
|
|||
|
||||
I'm interested in feedback, either via
|
||||
[twitter](https://twitter.com/h4nnes) or via eMail.
|
||||
|
||||
## Other updates in the MirageOS ecosystem
|
||||
|
||||
- [Canopy](https://github.com/Engil/Canopy) improvements: [no bower anymore](https://github.com/Engil/Canopy/pull/26), [HTTP caching support (via etags)](https://github.com/Engil/Canopy/pull/27), [listings now include dates](https://github.com/Engil/Canopy/pull/31), [dates are now in big-endian (y-m-d)](https://github.com/Engil/Canopy/pull/32)
|
||||
- [MirageOS call irclog from 2014-04-20](http://canopy.mirage.io/irclogs/20-04-2016)
|
||||
- blog article about [naive authentication service using MirageOS](https://abailly.github.io/posts/mirage-os-newbie.html)
|
||||
- [OCaml 4.03 post](https://lwn.net/SubscriberLink/684128/1436601f401c1f09/)
|
||||
|
|
|
|||
Loading…
Reference in a new issue