updated from main (commit 968592210d)
This commit is contained in:
Canopy bot
parent
5cf66b4a7b
commit
9c032cedd2
6 changed files with 24 additions and 24 deletions
6
About
6
About
|
|
@ -1,5 +1,5 @@
|
|||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>About</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="About" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>About</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/myself" class="tag">myself</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-01 (last updated: 2021-11-19)</span><article><h2 id="what-is-a-full-stack-engineer">What is a "full stack engineer"?</h2>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>About</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="About" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>About</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/overview" class="tag">overview</a><a href="/tags/myself" class="tag">myself</a><a href="/tags/background" class="tag">background</a></div><span class="date">Published: 2016-04-01 (last updated: 2023-11-20)</span><article><h2 id="what-is-a-full-stack-engineer">What is a "full stack engineer"?</h2>
|
||||
<p>Analysing the word literally, we should start with silicon and some electrons,
|
||||
maybe a soldering iron, and build everything all the way up to our favourite
|
||||
communication system.</p>
|
||||
|
|
@ -72,7 +72,7 @@ Morocco. A good friend of mine pointed me to <a href="https://mirage.io">Mirage
|
|||
clean-slate operating system written in the high-level language <a href="https://ocaml.org">OCaml</a>. I got
|
||||
hooked pretty fast, after some experience with LISP machines I imagined a modern
|
||||
OS written in a single functional programming language.</p>
|
||||
<p>From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the <a href="https://www.cl.cam.ac.uk/~pes20/rems">rigorous engineering of mainstream systems</a> project) with <a href="https://www.cl.cam.ac.uk/~pes20/">Peter Sewell</a>. I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on <a href="http://robur.io">robur</a>.</p>
|
||||
<p>From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the <a href="https://www.cl.cam.ac.uk/~pes20/rems">rigorous engineering of mainstream systems</a> project) with <a href="https://www.cl.cam.ac.uk/~pes20/">Peter Sewell</a>. I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on <a href="http://robur.coop">robur</a>.</p>
|
||||
<p>MirageOS had various bits and pieces into place, including infrastructure for
|
||||
building and testing (and a neat self-hosted website). A big gap was security.
|
||||
No access control, no secure sockets layer, nothing. This will be the topic of
|
||||
|
|
@ -84,5 +84,5 @@ not invalidated :-)" <a href="https://lwn.net/Articles/19378/">Xavier Leroy
|
|||
<h3 id="me-on-the-intertubes">Me on the intertubes</h3>
|
||||
<p>You can find me on <a href="https://twitter.com/h4nnes">twitter</a> and on
|
||||
<a href="https://github.com/hannesm">GitHub</a>.</p>
|
||||
<p>The data of this blog is <a href="https://git.robur.io/hannes/hannes.robur.coop">stored in a git repository</a>.</p>
|
||||
<p>The data of this blog is <a href="https://git.robur.coop/hannes/hannes.robur.coop">stored in a git repository</a>.</p>
|
||||
</article></div></div></main></body></html>
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Conex, establish trust in community repositories</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Conex, establish trust in community repositories" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/package signing" class="tag">package signing</a><a href="/tags/security" class="tag">security</a><a href="/tags/overview" class="tag">overview</a></div><span class="date">Published: 2017-02-16 (last updated: 2021-11-19)</span><article><p>Less than two years after the initial proposal, we're happy to present conex
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Conex, establish trust in community repositories</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Conex, establish trust in community repositories" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Conex, establish trust in community repositories</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/package signing" class="tag">package signing</a><a href="/tags/security" class="tag">security</a><a href="/tags/overview" class="tag">overview</a></div><span class="date">Published: 2017-02-16 (last updated: 2023-11-20)</span><article><p>Less than two years after the initial proposal, we're happy to present conex
|
||||
0.9.2. Pleas note that this is still work in progress, to be deployed with opam
|
||||
2.0 and the <a href="https://github.com/ocaml/opam-repository">opam repository</a>.</p>
|
||||
<p><img src="/static/img/conex.png" alt="screenshot" /></p>
|
||||
|
|
@ -313,5 +313,5 @@ cannot enable them.</p>
|
|||
verification experiments, and opam2 integration.</p>
|
||||
<p>I'm interested in feedback, please open an issue on the <a href="https://github.com/hannesm/conex">conex
|
||||
repository</a>. This article itself is stored as
|
||||
Markdown <a href="https://git.robur.io/hannes/hannes.robur.coop">in a different repository</a>.</p>
|
||||
Markdown <a href="https://git.robur.coop/hannes/hannes.robur.coop">in a different repository</a>.</p>
|
||||
</article></div></div></main></body></html>
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>My 2018 contains robur and starts with re-engineering DNS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="My 2018 contains robur and starts with re-engineering DNS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2018-01-11 (last updated: 2021-11-19)</span><article><h2 id="section">2018</h2>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>My 2018 contains robur and starts with re-engineering DNS</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="My 2018 contains robur and starts with re-engineering DNS" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>My 2018 contains robur and starts with re-engineering DNS</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/protocol" class="tag">protocol</a></div><span class="date">Published: 2018-01-11 (last updated: 2023-11-20)</span><article><h2 id="section">2018</h2>
|
||||
<p>At the end of 2017, I resigned from my PostDoc position at University of
|
||||
Cambridge (in the <a href="https://www.cl.cam.ac.uk/~pes20/rems/">rems</a> project). Early
|
||||
December 2017 I organised the <a href="https://mirage.io/blog/2017-winter-hackathon-roundup">4th MirageOS hack
|
||||
|
|
@ -8,12 +8,12 @@ very satisfied. In March 2018 the <a href="http://retreat.mirage.io">5th retrea
|
|||
happen (please sign up!).</p>
|
||||
<p>In 2018 I moved to Berlin and started to work for the (non-profit) <a href="https://techcultivation.org">Center for
|
||||
the cultivation of technology</a> with our
|
||||
<a href="http://robur.io">robur.io</a> project "At robur, we build performant bespoke
|
||||
<a href="http://robur.coop">robur.coop</a> project "At robur, we build performant bespoke
|
||||
minimal operating systems for high-assurance services". robur is only possible
|
||||
by generous donations in autumn 2017, enthusiastic collaborateurs, supportive
|
||||
friends, and a motivated community, thanks to all. We will receive funding from
|
||||
the <a href="https://prototypefund.de/project/robur-io/">prototypefund</a> to work on a
|
||||
<a href="https://robur.io/Our%20Work/Projects#CalDAV-Server">CalDAV server</a> implementation in OCaml
|
||||
<a href="https://robur.coop/Our%20Work/Projects#CalDAV-Server">CalDAV server</a> implementation in OCaml
|
||||
targeting MirageOS. We're still looking for donations and further funding,
|
||||
please get in touch. Apart from CalDAV, I want to start the year by finishing
|
||||
several projects which I discovered on my hard drive. This includes DNS, <a href="/Posts/Conex">opam
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying binary MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying binary MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2021-06-30 (last updated: 2021-11-15)</span><article><h2 id="introduction">Introduction</h2>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Deploying binary MirageOS unikernels</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Deploying binary MirageOS unikernels" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Deploying binary MirageOS unikernels</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a></div><span class="date">Published: 2021-06-30 (last updated: 2023-11-20)</span><article><h2 id="introduction">Introduction</h2>
|
||||
<p>MirageOS development focus has been a lot on tooling and the developer experience, but to accomplish <a href="https://robur.coop">our</a> goal to "get MirageOS into production", we need to lower the barrier. This means for us to release binary unikernels. As described <a href="/Posts/NGI">earlier</a>, we received a grant for "Deploying MirageOS" from <a href="https://pointer.ngi.eu">NGI Pointer</a> to work on the required infrastructure. This is joint work with <a href="https://reynir.dk/">Reynir</a>.</p>
|
||||
<p>We provide at <a href="https://builds.robur.coop">builds.robur.coop</a> binary unikernel images (and supplementary software). Doing binary releases of MirageOS unikernels is challenging in two aspects: firstly to be useful for everyone, a binary unikernel should not contain any configuration (such as private keys, certificates, etc.). Secondly, the binaries should be <a href="https://reproducible-builds.org">reproducible</a>. This is crucial for security; everyone can reproduce the exact same binary and verify that our build service did only use the sources. No malware or backdoors included.</p>
|
||||
<p>This post describes how you can deploy MirageOS unikernels without compiling it from source, then dives into the two issues outlined above - configuration and reproducibility - and finally describes how to setup your own reproducible build infrastructure for MirageOS, and how to bootstrap it.</p>
|
||||
|
|
@ -24,7 +24,7 @@
|
|||
<p>With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the <code>opam switch export</code> includes <a href="https://github.com/ocaml/opam/pull/4040">extra-files (patches)</a> and <a href="https://github.com/ocaml/opam/pull/4055">records the VCS version</a>. Based on this functionality, <a href="https://github.com/roburio/orb">orb</a>, an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are <code>PATH</code> (sanitized with an allow list, <code>/bin</code>, <code>/sbin</code>, with <code>/usr</code>, <code>/usr/local</code>, and <code>/opt</code> prefixes), and <code>HOME</code>. To enable Debian's <code>apt</code> to install packages, <code>DEBIAN_FRONTEND</code> is set to <code>noninteractive</code>. The <code>SWITCH_PATH</code> is recorded to allow orb to use the same path during a rebuild. The <code>SOURCE_DATE_EPOCH</code> is set to enable tools that record a timestamp to use a static one. The <code>OS*</code> variables are only used for recording the host OS and version.</p>
|
||||
<p>The goal of reproducible builds can certainly be achieved in several ways, including to store all sources and used executables in a huge tarball (or docker container), which is preserved for rebuilders. The question of minimal trusted computing base and how such a container could be rebuild from sources in reproducible way are open.</p>
|
||||
<p>The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe.</p>
|
||||
<p>For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. <em>hvt</em>), and we occasionally development versions of opam packages, we use <a href="https://git.robur.io/robur/unikernel-repo">the unikernel-repo</a> as overlay.</p>
|
||||
<p>For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. <em>hvt</em>), and we occasionally development versions of opam packages, we use <a href="https://git.robur.coop/robur/unikernel-repo">the unikernel-repo</a> as overlay.</p>
|
||||
<p>If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes.</p>
|
||||
<p>Each unikernel (and non-unikernel) job (e.g. <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/">dns-primary</a> outputs some artifacts:</p>
|
||||
<ul>
|
||||
|
|
@ -47,7 +47,7 @@
|
|||
</li>
|
||||
<li><a href="https://github.com/roburio/builder">builder</a> scheduling builds in contained environments (<a href="https://builds.robur.coop/job/builder/">packages</a>)
|
||||
</li>
|
||||
<li><a href="https://git.robur.io/robur/builder-web">builder-web</a> storing builds in a database and providing a HTTP interface (<a href="https://builds.robur.coop/job/builder-web/">packages</a>)
|
||||
<li><a href="https://git.robur.coop/robur/builder-web">builder-web</a> storing builds in a database and providing a HTTP interface (<a href="https://builds.robur.coop/job/builder-web/">packages</a>)
|
||||
</li>
|
||||
</ul>
|
||||
<p>These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.</p>
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
<!DOCTYPE html>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Mirroring the opam repository and all tarballs</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Mirroring the opam repository and all tarballs" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a><a href="/tags/opam" class="tag">opam</a></div><span class="date">Published: 2022-09-29 (last updated: 2022-10-11)</span><article><p>We at <a href="https://robur.coop">robur</a> developed <a href="https://git.robur.io/robur/opam-mirror">opam-mirror</a> in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).</p>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Mirroring the opam repository and all tarballs</title><meta charset="UTF-8"/><link rel="stylesheet" href="/static/css/style.css"/><link rel="stylesheet" href="/static/css/highlight.css"/><script src="/static/js/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script><link rel="alternate" href="/atom" title="Mirroring the opam repository and all tarballs" type="application/atom+xml"/><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover"/></head><body><nav class="navbar navbar-default navbar-fixed-top"><div class="container"><div class="navbar-header"><a class="navbar-brand" href="/Posts">full stack engineer</a></div><div class="collapse navbar-collapse collapse"><ul class="nav navbar-nav navbar-right"><li><a href="/About"><span>About</span></a></li><li><a href="/Posts"><span>Posts</span></a></li></ul></div></div></nav><main><div class="flex-container"><div class="post"><h2>Mirroring the opam repository and all tarballs</h2><span class="author">Written by hannes</span><br/><div class="tags">Classified under: <a href="/tags/mirageos" class="tag">mirageos</a><a href="/tags/deployment" class="tag">deployment</a><a href="/tags/opam" class="tag">opam</a></div><span class="date">Published: 2022-09-29 (last updated: 2023-11-20)</span><article><p>We at <a href="https://robur.coop">robur</a> developed <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).</p>
|
||||
<h1 id="what-is-opam-and-why-should-i-care">What is opam and why should I care?</h1>
|
||||
<p><a href="https://opam.ocaml.org">Opam</a> is the OCaml package manager (also used by other projects such as <a href="https://coq.inria.fr">coq</a>). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as <a href="https://github.com/ocaml/opam-repository">ocaml/opam-repository</a>, where authors of OCaml software can contribute (as pull request) their latest releases.</p>
|
||||
<p>When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state "my package will not work with <a href="https://erratique.ch/software/cmdliner">cmdliner</a> in version 1.2.0").</p>
|
||||
|
|
@ -12,7 +12,7 @@
|
|||
<p>According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses <code>opam admin index</code> periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was <a href="https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/">a reported downtime</a>.</p>
|
||||
<p>Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS <a href="http://retreat.mirage.io">hack retreats</a> in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).</p>
|
||||
<h1 id="re-developing-opam-admin-create-as-mirageos-unikernel">Re-developing "opam admin create" as MirageOS unikernel</h1>
|
||||
<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.io/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
|
||||
<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
|
||||
<p>Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.</p>
|
||||
<p>In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with <a href="https://github.com/mirage/ocaml-crunch">crunch</a>, or using a remote git repository), but recent development, e.g. of <a href="https://somerandomidiot.com/blog/2022/03/04/chamelon/">chamelon</a> sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.</p>
|
||||
<p>Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.</p>
|
||||
|
|
@ -21,9 +21,9 @@
|
|||
<p>There is already a gap in the above plan: which http client to use - in the best case something similar to our <a href="https://github.com/roburio/http-lwt-client">http-lwt-client</a> - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using <a href="https://github.com/roburio/happy-eyeballs">happy-eyeballs</a> to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.</p>
|
||||
<p>On the path (over the last month), we fixed file descriptor leaks (memory leaks) in <a href="https://github.com/dinosaure/paf-le-chien">paf</a> -- which is used as a runtime for httpaf and h2.</p>
|
||||
<p>Then we ran into some trouble with chamelon (<a href="https://github.com/yomimono/chamelon/issues/11">out of memory</a>, some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive.</p>
|
||||
<p>We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed <a href="https://git.robur.io/robur/git-kv">git-kv</a> which can dump and restore the git state.</p>
|
||||
<p>We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed <a href="https://git.robur.coop/robur/git-kv">git-kv</a> which can dump and restore the git state.</p>
|
||||
<p>Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well.</p>
|
||||
<p>Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as <a href="https://github.com/mirage/ocaml-fat">fat</a> may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed <a href="https://github.com/reynir/mirage-block-partition">mirage-block-partition</a> to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed <a href="https://git.robur.io/reynir/oneffs">oneffs</a> which stores a single value on the block device.</p>
|
||||
<p>Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as <a href="https://github.com/mirage/ocaml-fat">fat</a> may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed <a href="https://github.com/reynir/mirage-block-partition">mirage-block-partition</a> to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed <a href="https://git.robur.coop/reynir/oneffs">oneffs</a> which stores a single value on the block device.</p>
|
||||
<p>We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains <a href="https://github.com/mirage/mirage-kv/pull/28">recent enhancements</a> of the mirage-kv API, there is a <code>get_partial</code>, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).</p>
|
||||
<p>What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy <a href="https://github.com/roburio/tlstunnel">tlstunnel</a>) similar to how <a href="https://github.com/roburio/unipi">unipi</a> does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).</p>
|
||||
<h1 id="conclusion">Conclusion</h1>
|
||||
|
|
|
|||
18
atom
18
atom
|
|
@ -1,4 +1,4 @@
|
|||
<feed xmlns="http://www.w3.org/2005/Atom"><link href="https://hannes.robur.coop/atom" rel="self"/><id>urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156</id><title type="text">full stack engineer</title><updated>2023-05-16T17:21:47-00:00</updated><entry><summary type="html"><p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
|
||||
<feed xmlns="http://www.w3.org/2005/Atom"><link href="https://hannes.robur.coop/atom" rel="self"/><id>urn:uuid:981361ca-e71d-4997-a52c-baeee78e4156</id><title type="text">full stack engineer</title><updated>2023-11-20T16:58:35-00:00</updated><entry><summary type="html"><p>fleet management for MirageOS unikernels using a mutually authenticated TLS handshake</p>
|
||||
</summary><published>2022-11-17T12:41:11-00:00</published><link href="/Posts/Albatross" rel="alternate"/><content type="html"><p>EDIT (2023-05-16): Updated with albatross release version 2.0.0.</p>
|
||||
<h2 id="deploying-mirageos-unikernels">Deploying MirageOS unikernels</h2>
|
||||
<p>More than five years ago, I posted <a href="/Posts/VMM">how to deploy MirageOS unikernels</a>. My motivation to work on this topic is that I'm convinced of reduced complexity, improved security, and more sustainable resource footprint of MirageOS unikernels, and want to ease deployment thereof. More than one year ago, I described <a href="/Posts/Deploy">how to deploy reproducible unikernels</a>.</p>
|
||||
|
|
@ -81,7 +81,7 @@ $ pkg install solo5 albatross
|
|||
<p>After five years of development and operating albatross, feel free to get it and try it out. Or read the code, discuss issues and shortcomings with us - either at the issue tracker or via eMail.</p>
|
||||
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
|
||||
</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:1f354218-e8c3-5136-a2ca-c88f3c2878d8</id><title type="text">Deploying reproducible unikernels with albatross</title><updated>2023-05-16T17:21:47-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html"><p>Re-developing an opam cache from scratch, as a MirageOS unikernel</p>
|
||||
</summary><published>2022-09-29T13:04:14-00:00</published><link href="/Posts/OpamMirror" rel="alternate"/><content type="html"><p>We at <a href="https://robur.coop">robur</a> developed <a href="https://git.robur.io/robur/opam-mirror">opam-mirror</a> in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).</p>
|
||||
</summary><published>2022-09-29T13:04:14-00:00</published><link href="/Posts/OpamMirror" rel="alternate"/><content type="html"><p>We at <a href="https://robur.coop">robur</a> developed <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).</p>
|
||||
<h1 id="what-is-opam-and-why-should-i-care">What is opam and why should I care?</h1>
|
||||
<p><a href="https://opam.ocaml.org">Opam</a> is the OCaml package manager (also used by other projects such as <a href="https://coq.inria.fr">coq</a>). It is a source based system: the so-called repository contains the metadata (url to source tarballs, build dependencies, author, homepage, development repository) of all packages. The main repository is hosted on GitHub as <a href="https://github.com/ocaml/opam-repository">ocaml/opam-repository</a>, where authors of OCaml software can contribute (as pull request) their latest releases.</p>
|
||||
<p>When opening a pull request, automated systems attempt to build not only the newly released package on various platforms and OCaml versions, but also all reverse dependencies, and also with dependencies with the lowest allowed version numbers. That's crucial since neither semantic versioning has been adapted across the OCaml ecosystem (which is tricky, for example due to local opens any newly introduced binding will lead to a major version bump), neither do many people add upper bounds of dependencies when releasing a package (nobody is keen to state &quot;my package will not work with <a href="https://erratique.ch/software/cmdliner">cmdliner</a> in version 1.2.0&quot;).</p>
|
||||
|
|
@ -94,7 +94,7 @@ $ pkg install solo5 albatross
|
|||
<p>According to DNS, opam.ocaml.org is a machine at amazon. It likely, apart from the website, uses <code>opam admin index</code> periodically to create the index tarball and the cache. There's an observable delay between a package merge in the opam-repository and when it shows up at opam.ocaml.org. Recently, there was <a href="https://discuss.ocaml.org/t/opam-ocaml-org-is-currently-down-is-that-where-indices-are-kept-still/">a reported downtime</a>.</p>
|
||||
<p>Apart from being a single point of failure, if you're compiling a lot of opam projects (e.g. a continuous integration / continuous build system), it makes sense from a network usage (and thus sustainability perspective) to move the cache closer to where you need the source archives. We're also organising the MirageOS <a href="http://retreat.mirage.io">hack retreats</a> in a northern African country with poor connectivity - so if you gather two dozen camels you better bring your opam repository cache with you to reduce the bandwidth usage (NB: this requires at the moment cooperation of all participants to configure their default opam repository accordingly).</p>
|
||||
<h1 id="re-developing-opam-admin-create-as-mirageos-unikernel">Re-developing &quot;opam admin create&quot; as MirageOS unikernel</h1>
|
||||
<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.io/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
|
||||
<p>The need for a local opam cache at our <a href="https://builds.robur.coop">reproducible build infrastructure</a> and the retreats, we decided to develop <a href="https://git.robur.coop/robur/opam-mirror">opam-mirror</a> as a <a href="https://mirage.io">MirageOS unikernel</a>. Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the <code>opam admin index</code> is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.</p>
|
||||
<p>Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.</p>
|
||||
<p>In earlier years, we avoided persistent storage and block devices in MirageOS (by embedding it into the source code with <a href="https://github.com/mirage/ocaml-crunch">crunch</a>, or using a remote git repository), but recent development, e.g. of <a href="https://somerandomidiot.com/blog/2022/03/04/chamelon/">chamelon</a> sparked some interest in actually using file systems and figuring out whether MirageOS is ready in that area. A month ago we started the opam-mirror project.</p>
|
||||
<p>Opam-mirror takes a remote repository URL, and downloads all referenced archives. It serves as a cache and opam-repository - and does periodic updates from the remote repository. The idea is to validate all available checksums and store the tarballs only once, and store overlays (as maps) from the other hash algorithms.</p>
|
||||
|
|
@ -103,15 +103,15 @@ $ pkg install solo5 albatross
|
|||
<p>There is already a gap in the above plan: which http client to use - in the best case something similar to our <a href="https://github.com/roburio/http-lwt-client">http-lwt-client</a> - in MirageOS: it should support HTTP 1.1 and HTTP 2, TLS (with certificate validation), and using <a href="https://github.com/roburio/happy-eyeballs">happy-eyeballs</a> to seemlessly support both IPv6 and legacy IPv4. Of course it should follow redirect, without that we won't get far in the current Internet.</p>
|
||||
<p>On the path (over the last month), we fixed file descriptor leaks (memory leaks) in <a href="https://github.com/dinosaure/paf-le-chien">paf</a> -- which is used as a runtime for httpaf and h2.</p>
|
||||
<p>Then we ran into some trouble with chamelon (<a href="https://github.com/yomimono/chamelon/issues/11">out of memory</a>, some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive.</p>
|
||||
<p>We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed <a href="https://git.robur.io/robur/git-kv">git-kv</a> which can dump and restore the git state.</p>
|
||||
<p>We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed <a href="https://git.robur.coop/robur/git-kv">git-kv</a> which can dump and restore the git state.</p>
|
||||
<p>Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well.</p>
|
||||
<p>Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as <a href="https://github.com/mirage/ocaml-fat">fat</a> may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed <a href="https://github.com/reynir/mirage-block-partition">mirage-block-partition</a> to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed <a href="https://git.robur.io/reynir/oneffs">oneffs</a> which stores a single value on the block device.</p>
|
||||
<p>Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as <a href="https://github.com/mirage/ocaml-fat">fat</a> may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed <a href="https://github.com/reynir/mirage-block-partition">mirage-block-partition</a> to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed <a href="https://git.robur.coop/reynir/oneffs">oneffs</a> which stores a single value on the block device.</p>
|
||||
<p>We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains <a href="https://github.com/mirage/mirage-kv/pull/28">recent enhancements</a> of the mirage-kv API, there is a <code>get_partial</code>, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).</p>
|
||||
<p>What is next? Downloading and writing to the tar archive could be done chunk-wise as well; also dumping and restoring the git state is quite CPU intensive, we would like to improve that. Adding the TLS frontend (currently done on our site by our TLS termination proxy <a href="https://github.com/roburio/tlstunnel">tlstunnel</a>) similar to how <a href="https://github.com/roburio/unipi">unipi</a> does it, including let's encrypt provisioning -- should be straightforward (drop us a note if you'd be interesting in that feature).</p>
|
||||
<h1 id="conclusion">Conclusion</h1>
|
||||
<p>To conclude, we managed within a month to develop this opam-mirror cache from scratch. It has a reasonable footprint (CPU and memory-wise), is easy to maintain and easy to update - if you want to use it, we also provide <a href="https://builds.robur.coop/job/opam-mirror">reproducible binaries</a> for solo5-hvt. You can use our opam mirror with <code>opam repository set-url default https://opam.robur.coop</code> (revert to the other with <code>opam repository set-url default https://opam.ocaml.org</code>) or use it as a backup with <code>opam repository add robur --rank 2 https://opam.robur.coop</code>.</p>
|
||||
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions. We are a non-profit company, and rely on <a href="https://robur.coop/Donate">donations</a> for doing our work - everyone can contribute.</p>
|
||||
</content><category scheme="https://hannes.robur.coop/tags/opam" term="opam"/><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:0dbd251f-32c7-57bd-8e8f-7392c0833a09</id><title type="text">Mirroring the opam repository and all tarballs</title><updated>2022-10-11T12:14:07-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||
</content><category scheme="https://hannes.robur.coop/tags/opam" term="opam"/><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:0dbd251f-32c7-57bd-8e8f-7392c0833a09</id><title type="text">Mirroring the opam repository and all tarballs</title><updated>2023-11-20T16:58:35-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html"><p>How to monitor your MirageOS unikernel with albatross and monitoring-experiments</p>
|
||||
</summary><published>2022-03-08T11:26:31-00:00</published><link href="/Posts/Monitoring" rel="alternate"/><content type="html"><h1 id="introduction-to-monitoring">Introduction to monitoring</h1>
|
||||
<p>At <a href="https://robur.coop">robur</a> we use a range of MirageOS unikernels. Recently, we worked on improving the operations story thereof. One part is shipping binaries using our <a href="https://builds.robur.coop">reproducible builds infrastructure</a>. Another part is, once deployed we want to observe what is going on.</p>
|
||||
<p>I first got into touch with monitoring - collecting and graphing metrics - with <a href="https://oss.oetiker.ch/mrtg/">MRTG</a> and <a href="https://munin-monitoring.org/">munin</a> - and the simple network management protocol <a href="https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol">SNMP</a>. From the whole system perspective, I find it crucial that the monitoring part of a system does not add pressure. This favours a push-based design, where reporting is done at the disposition of the system.</p>
|
||||
|
|
@ -251,7 +251,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
|
|||
<p>With opam, we already have precise tracking which opam packages are used, and since opam 2.1 the <code>opam switch export</code> includes <a href="https://github.com/ocaml/opam/pull/4040">extra-files (patches)</a> and <a href="https://github.com/ocaml/opam/pull/4055">records the VCS version</a>. Based on this functionality, <a href="https://github.com/roburio/orb">orb</a>, an alternative command line application using the opam-client library, can be used to collect (a) the switch export, (b) host system packages, and (c) the environment variables. Only required environment variables are kept, all others are unset while conducting a build. The only required environment variables are <code>PATH</code> (sanitized with an allow list, <code>/bin</code>, <code>/sbin</code>, with <code>/usr</code>, <code>/usr/local</code>, and <code>/opt</code> prefixes), and <code>HOME</code>. To enable Debian's <code>apt</code> to install packages, <code>DEBIAN_FRONTEND</code> is set to <code>noninteractive</code>. The <code>SWITCH_PATH</code> is recorded to allow orb to use the same path during a rebuild. The <code>SOURCE_DATE_EPOCH</code> is set to enable tools that record a timestamp to use a static one. The <code>OS*</code> variables are only used for recording the host OS and version.</p>
|
||||
<p>The goal of reproducible builds can certainly be achieved in several ways, including to store all sources and used executables in a huge tarball (or docker container), which is preserved for rebuilders. The question of minimal trusted computing base and how such a container could be rebuild from sources in reproducible way are open.</p>
|
||||
<p>The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe.</p>
|
||||
<p>For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. <em>hvt</em>), and we occasionally development versions of opam packages, we use <a href="https://git.robur.io/robur/unikernel-repo">the unikernel-repo</a> as overlay.</p>
|
||||
<p>For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. <em>hvt</em>), and we occasionally development versions of opam packages, we use <a href="https://git.robur.coop/robur/unikernel-repo">the unikernel-repo</a> as overlay.</p>
|
||||
<p>If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes.</p>
|
||||
<p>Each unikernel (and non-unikernel) job (e.g. <a href="https://builds.robur.coop/job/dns-primary-git/build/latest/">dns-primary</a> outputs some artifacts:</p>
|
||||
<ul>
|
||||
|
|
@ -274,7 +274,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
|
|||
</li>
|
||||
<li><a href="https://github.com/roburio/builder">builder</a> scheduling builds in contained environments (<a href="https://builds.robur.coop/job/builder/">packages</a>)
|
||||
</li>
|
||||
<li><a href="https://git.robur.io/robur/builder-web">builder-web</a> storing builds in a database and providing a HTTP interface (<a href="https://builds.robur.coop/job/builder-web/">packages</a>)
|
||||
<li><a href="https://git.robur.coop/robur/builder-web">builder-web</a> storing builds in a database and providing a HTTP interface (<a href="https://builds.robur.coop/job/builder-web/">packages</a>)
|
||||
</li>
|
||||
</ul>
|
||||
<p>These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.</p>
|
||||
|
|
@ -283,7 +283,7 @@ _stack.V4V6) (_ : sig end) (Management : Mirage_stack.V4V6) = struct
|
|||
<p>We also have an index over sha-256 checksum of binaries - in the case you find a running unikernel image where you forgot which exact packages were used, you can do a reverse lookup.</p>
|
||||
<p>We are aware that the web interface can be improved (PRs welcome). We will also work on the rebuilder setup and run some rebuilds.</p>
|
||||
<p>Please reach out to us (at team AT robur DOT coop) if you have feedback and suggestions.</p>
|
||||
</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:331831d8-6093-5dd7-9164-445afff953cb</id><title type="text">Deploying binary MirageOS unikernels</title><updated>2021-11-15T11:17:23-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||
</content><category scheme="https://hannes.robur.coop/tags/deployment" term="deployment"/><category scheme="https://hannes.robur.coop/tags/mirageos" term="mirageos"/><id>urn:uuid:331831d8-6093-5dd7-9164-445afff953cb</id><title type="text">Deploying binary MirageOS unikernels</title><updated>2023-11-20T16:58:35-00:00</updated><author><name>hannes</name></author></entry><entry><summary type="html"><p>Elliptic curves (ECDSA/ECDH) are supported in a maintainable and secure way.</p>
|
||||
</summary><published>2021-04-23T13:33:06-00:00</published><link href="/Posts/EC" rel="alternate"/><content type="html"><h2 id="introduction">Introduction</h2>
|
||||
<p>Tl;DR: mirage-crypto-ec, with x509 0.12.0, and tls 0.13.0, provide fast and secure elliptic curve support in OCaml and MirageOS - using the verified <a href="https://github.com/mit-plv/fiat-crypto/">fiat-crypto</a> stack (Coq to OCaml to executable which generates C code that is interfaced by OCaml). In x509, a long standing issue (countryName encoding), and archive (PKCS 12) format is now supported, in addition to EC keys. In tls, ECDH key exchanges are supported, and ECDSA and EdDSA certificates.</p>
|
||||
<h2 id="elliptic-curve-cryptography">Elliptic curve cryptography</h2>
|
||||
|
|
|
|||
Loading…
Reference in a new issue