From 968592210d77e1b88b2b18d4e0b02bc528171277 Mon Sep 17 00:00:00 2001 From: Hannes Mehnert Date: Mon, 20 Nov 2023 17:58:35 +0100 Subject: [PATCH] robur.coop it is --- About | 4 ++-- Posts/Conex | 2 +- Posts/DNS | 4 ++-- Posts/Deploy | 4 ++-- Posts/OpamMirror | 8 ++++---- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/About b/About index 7a457aa..65d0f22 100644 --- a/About +++ b/About @@ -93,7 +93,7 @@ clean-slate operating system written in the high-level language [OCaml](https:// hooked pretty fast, after some experience with LISP machines I imagined a modern OS written in a single functional programming language. -From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the [rigorous engineering of mainstream systems](https://www.cl.cam.ac.uk/~pes20/rems) project) with [Peter Sewell](https://www.cl.cam.ac.uk/~pes20/). I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on [robur](http://robur.io). +From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the [rigorous engineering of mainstream systems](https://www.cl.cam.ac.uk/~pes20/rems) project) with [Peter Sewell](https://www.cl.cam.ac.uk/~pes20/). I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on [robur](http://robur.coop). MirageOS had various bits and pieces into place, including infrastructure for building and testing (and a neat self-hosted website). A big gap was security. @@ -110,4 +110,4 @@ not invalidated :-)" [Xavier Leroy](https://lwn.net/Articles/19378/)), and the [ You can find me on [twitter](https://twitter.com/h4nnes) and on [GitHub](https://github.com/hannesm). -The data of this blog is [stored in a git repository](https://git.robur.io/hannes/hannes.robur.coop). \ No newline at end of file +The data of this blog is [stored in a git repository](https://git.robur.coop/hannes/hannes.robur.coop). \ No newline at end of file diff --git a/Posts/Conex b/Posts/Conex index cae76b7..b9dd89a 100644 --- a/Posts/Conex +++ b/Posts/Conex @@ -389,4 +389,4 @@ verification experiments, and opam2 integration. I'm interested in feedback, please open an issue on the [conex repository](https://github.com/hannesm/conex). This article itself is stored as -Markdown [in a different repository](https://git.robur.io/hannes/hannes.robur.coop). +Markdown [in a different repository](https://git.robur.coop/hannes/hannes.robur.coop). diff --git a/Posts/DNS b/Posts/DNS index 661a283..4fcd3e2 100644 --- a/Posts/DNS +++ b/Posts/DNS @@ -16,12 +16,12 @@ happen (please sign up!). In 2018 I moved to Berlin and started to work for the (non-profit) [Center for the cultivation of technology](https://techcultivation.org) with our -[robur.io](http://robur.io) project "At robur, we build performant bespoke +[robur.coop](http://robur.coop) project "At robur, we build performant bespoke minimal operating systems for high-assurance services". robur is only possible by generous donations in autumn 2017, enthusiastic collaborateurs, supportive friends, and a motivated community, thanks to all. We will receive funding from the [prototypefund](https://prototypefund.de/project/robur-io/) to work on a -[CalDAV server](https://robur.io/Our%20Work/Projects#CalDAV-Server) implementation in OCaml +[CalDAV server](https://robur.coop/Our%20Work/Projects#CalDAV-Server) implementation in OCaml targeting MirageOS. We're still looking for donations and further funding, please get in touch. Apart from CalDAV, I want to start the year by finishing several projects which I discovered on my hard drive. This includes DNS, [opam diff --git a/Posts/Deploy b/Posts/Deploy index 384b163..3d9336b 100644 --- a/Posts/Deploy +++ b/Posts/Deploy @@ -55,7 +55,7 @@ The goal of reproducible builds can certainly be achieved in several ways, inclu The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe. -For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. *hvt*), and we occasionally development versions of opam packages, we use [the unikernel-repo](https://git.robur.io/robur/unikernel-repo) as overlay. +For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. *hvt*), and we occasionally development versions of opam packages, we use [the unikernel-repo](https://git.robur.coop/robur/unikernel-repo) as overlay. If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes. @@ -73,7 +73,7 @@ You can [browse](https://builds.robur.coop/job/dns-primary-git/) the different b The opam reproducible build infrastructure is driven by: - [orb](https://github.com/roburio/orb) conducting reproducible builds ([packages](https://builds.robur.coop/job/orb/)) - [builder](https://github.com/roburio/builder) scheduling builds in contained environments ([packages](https://builds.robur.coop/job/builder/)) -- [builder-web](https://git.robur.io/robur/builder-web) storing builds in a database and providing a HTTP interface ([packages](https://builds.robur.coop/job/builder-web/)) +- [builder-web](https://git.robur.coop/robur/builder-web) storing builds in a database and providing a HTTP interface ([packages](https://builds.robur.coop/job/builder-web/)) These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released. diff --git a/Posts/OpamMirror b/Posts/OpamMirror index a10bf34..4402395 100644 --- a/Posts/OpamMirror +++ b/Posts/OpamMirror @@ -5,7 +5,7 @@ tags: mirageos, deployment, opam abstract: Re-developing an opam cache from scratch, as a MirageOS unikernel --- -We at [robur](https://robur.coop) developed [opam-mirror](https://git.robur.io/robur/opam-mirror) in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly). +We at [robur](https://robur.coop) developed [opam-mirror](https://git.robur.coop/robur/opam-mirror) in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly). # What is opam and why should I care? @@ -31,7 +31,7 @@ Apart from being a single point of failure, if you're compiling a lot of opam pr # Re-developing "opam admin create" as MirageOS unikernel -The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.io/robur/opam-mirror) as a [MirageOS unikernel](https://mirage.io). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface. +The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.coop/robur/opam-mirror) as a [MirageOS unikernel](https://mirage.io). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface. Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky. @@ -49,11 +49,11 @@ On the path (over the last month), we fixed file descriptor leaks (memory leaks) Then we ran into some trouble with chamelon ([out of memory](https://github.com/yomimono/chamelon/issues/11), some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive. -We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed [git-kv](https://git.robur.io/robur/git-kv) which can dump and restore the git state. +We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed [git-kv](https://git.robur.coop/robur/git-kv) which can dump and restore the git state. Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well. -Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as [fat](https://github.com/mirage/ocaml-fat) may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed [mirage-block-partition](https://github.com/reynir/mirage-block-partition) to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed [oneffs](https://git.robur.io/reynir/oneffs) which stores a single value on the block device. +Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as [fat](https://github.com/mirage/ocaml-fat) may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed [mirage-block-partition](https://github.com/reynir/mirage-block-partition) to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed [oneffs](https://git.robur.coop/reynir/oneffs) which stores a single value on the block device. We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains [recent enhancements](https://github.com/mirage/mirage-kv/pull/28) of the mirage-kv API, there is a `get_partial`, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).