hannes/hannes.robur.coop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

robur.coop it is

Browse source
This commit is contained in:
Hannes Mehnert 2023-11-20 17:58:35 +01:00
parent 8db22fe152
commit 968592210d
5 changed files with 11 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
About
View file

@ -93,7 +93,7 @@ clean-slate operating system written in the high-level language [OCaml](https://
hooked pretty fast, after some experience with LISP machines I imagined a modern hooked pretty fast, after some experience with LISP machines I imagined a modern
OS written in a single functional programming language. OS written in a single functional programming language.
From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the [rigorous engineering of mainstream systems](https://www.cl.cam.ac.uk/~pes20/rems) project) with [Peter Sewell](https://www.cl.cam.ac.uk/~pes20/). I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on [robur](http://robur.io). From summer 2014 until end of 2017 I worked as a postdoctoral researcher at University of Cambridge (in the [rigorous engineering of mainstream systems](https://www.cl.cam.ac.uk/~pes20/rems) project) with [Peter Sewell](https://www.cl.cam.ac.uk/~pes20/). I primarily worked on TLS, MirageOS, opam signing, and network semantics. In 2018 I relocated back to Berlin and am working on [robur](http://robur.coop).
MirageOS had various bits and pieces into place, including infrastructure for MirageOS had various bits and pieces into place, including infrastructure for
building and testing (and a neat self-hosted website). A big gap was security. building and testing (and a neat self-hosted website). A big gap was security.
@ -110,4 +110,4 @@ not invalidated :-)" [Xavier Leroy](https://lwn.net/Articles/19378/)), and the [
You can find me on [twitter](https://twitter.com/h4nnes) and on You can find me on [twitter](https://twitter.com/h4nnes) and on
[GitHub](https://github.com/hannesm). [GitHub](https://github.com/hannesm).
The data of this blog is [stored in a git repository](https://git.robur.io/hannes/hannes.robur.coop). The data of this blog is [stored in a git repository](https://git.robur.coop/hannes/hannes.robur.coop).

2
Posts/Conex
View file

@ -389,4 +389,4 @@ verification experiments, and opam2 integration.
I'm interested in feedback, please open an issue on the [conex I'm interested in feedback, please open an issue on the [conex
repository](https://github.com/hannesm/conex). This article itself is stored as repository](https://github.com/hannesm/conex). This article itself is stored as
Markdown [in a different repository](https://git.robur.io/hannes/hannes.robur.coop). Markdown [in a different repository](https://git.robur.coop/hannes/hannes.robur.coop).

4
Posts/DNS
View file

@ -16,12 +16,12 @@ happen (please sign up!).
In 2018 I moved to Berlin and started to work for the (non-profit) [Center for In 2018 I moved to Berlin and started to work for the (non-profit) [Center for
the cultivation of technology](https://techcultivation.org) with our the cultivation of technology](https://techcultivation.org) with our
[robur.io](http://robur.io) project "At robur, we build performant bespoke [robur.coop](http://robur.coop) project "At robur, we build performant bespoke
minimal operating systems for high-assurance services". robur is only possible minimal operating systems for high-assurance services". robur is only possible
by generous donations in autumn 2017, enthusiastic collaborateurs, supportive by generous donations in autumn 2017, enthusiastic collaborateurs, supportive
friends, and a motivated community, thanks to all. We will receive funding from friends, and a motivated community, thanks to all. We will receive funding from
the [prototypefund](https://prototypefund.de/project/robur-io/) to work on a the [prototypefund](https://prototypefund.de/project/robur-io/) to work on a
[CalDAV server](https://robur.io/Our%20Work/Projects#CalDAV-Server) implementation in OCaml [CalDAV server](https://robur.coop/Our%20Work/Projects#CalDAV-Server) implementation in OCaml
targeting MirageOS. We're still looking for donations and further funding, targeting MirageOS. We're still looking for donations and further funding,
please get in touch. Apart from CalDAV, I want to start the year by finishing please get in touch. Apart from CalDAV, I want to start the year by finishing
several projects which I discovered on my hard drive. This includes DNS, [opam several projects which I discovered on my hard drive. This includes DNS, [opam

4
Posts/Deploy
View file

@ -55,7 +55,7 @@ The goal of reproducible builds can certainly be achieved in several ways, inclu
The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe. The opam-repository is a community repository, where packages are released to on a daily basis by a lot of OCaml developers. Package dependencies usually only use lower bounds of other packages, and the continuous integration system of the opam repository takes care that upon API changes all reverse dependencies include the right upper bounds. Using the head commit of opam-repository usually leads to a working package universe.
For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. *hvt*), and we occasionally development versions of opam packages, we use [the unikernel-repo](https://git.robur.io/robur/unikernel-repo) as overlay. For our MirageOS unikernels, we don't want to stay behind with ancient versions of libraries. That's why our automated building is done on a daily basis with the head commit of opam-repository. Since our unikernels are not part of the main opam repository (they include the configuration information which target to use, e.g. *hvt*), and we occasionally development versions of opam packages, we use [the unikernel-repo](https://git.robur.coop/robur/unikernel-repo) as overlay.
If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes. If no dependent package got a new release, the resulting binary has the same checksum. If any dependency was released with a newer release, this is picked up, and eventually the checksum changes.
@ -73,7 +73,7 @@ You can [browse](https://builds.robur.coop/job/dns-primary-git/) the different b
The opam reproducible build infrastructure is driven by: The opam reproducible build infrastructure is driven by:
- [orb](https://github.com/roburio/orb) conducting reproducible builds ([packages](https://builds.robur.coop/job/orb/)) - [orb](https://github.com/roburio/orb) conducting reproducible builds ([packages](https://builds.robur.coop/job/orb/))
- [builder](https://github.com/roburio/builder) scheduling builds in contained environments ([packages](https://builds.robur.coop/job/builder/)) - [builder](https://github.com/roburio/builder) scheduling builds in contained environments ([packages](https://builds.robur.coop/job/builder/))
- [builder-web](https://git.robur.io/robur/builder-web) storing builds in a database and providing a HTTP interface ([packages](https://builds.robur.coop/job/builder-web/)) - [builder-web](https://git.robur.coop/robur/builder-web) storing builds in a database and providing a HTTP interface ([packages](https://builds.robur.coop/job/builder-web/))
These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released. These tools are themselves reproducible, and built on a daily basis. The infrastructure executing the build jobs installs the most recent packages of orb and builder before conducting a build. This means that our build infrastructure is reproducible as well, and uses the latest code when it is released.

8
Posts/OpamMirror
View file

@ -5,7 +5,7 @@ tags: mirageos, deployment, opam
abstract: Re-developing an opam cache from scratch, as a MirageOS unikernel abstract: Re-developing an opam cache from scratch, as a MirageOS unikernel
--- ---
We at [robur](https://robur.coop) developed [opam-mirror](https://git.robur.io/robur/opam-mirror) in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly). We at [robur](https://robur.coop) developed [opam-mirror](https://git.robur.coop/robur/opam-mirror) in the last month and run a public opam mirror at https://opam.robur.coop (updated hourly).
# What is opam and why should I care? # What is opam and why should I care?
@ -31,7 +31,7 @@ Apart from being a single point of failure, if you're compiling a lot of opam pr
# Re-developing "opam admin create" as MirageOS unikernel # Re-developing "opam admin create" as MirageOS unikernel
The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.io/robur/opam-mirror) as a [MirageOS unikernel](https://mirage.io). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface. The need for a local opam cache at our [reproducible build infrastructure](https://builds.robur.coop) and the retreats, we decided to develop [opam-mirror](https://git.robur.coop/robur/opam-mirror) as a [MirageOS unikernel](https://mirage.io). Apart from a useful showcase using persistent storage (that won't fit into memory), and having fun while developing it, our aim was to reduce our time spent on system administration (the `opam admin index` is only one part of the story, it needs a Unix system and a webserver next to it - plus remote access for doing software updates - which has quite some attack surface.
Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky. Another reason for re-developing the functionality was that the opam code (what opam admin index actually does) is part of the opam source code, which totals to 50_000 lines of code -- looking up whether one or all checksums are verified before adding the tarball to the cache, was rather tricky.
@ -49,11 +49,11 @@ On the path (over the last month), we fixed file descriptor leaks (memory leaks)
Then we ran into some trouble with chamelon ([out of memory](https://github.com/yomimono/chamelon/issues/11), some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive. Then we ran into some trouble with chamelon ([out of memory](https://github.com/yomimono/chamelon/issues/11), some degraded peformance, it reporting out of disk space), and re-thought our demands for opam-mirror. Since the cache is only ever growing (new packages are released), there's no need to ever remove anything: it is append-only. Once we figured that out, we investigated what needs to be done in ocaml-tar (where tar is in fact a tape archive, and was initially designed as file format to be appended to) to support appending to an archive.
We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed [git-kv](https://git.robur.io/robur/git-kv) which can dump and restore the git state. We also re-thought our bandwidth usage, and instead of cloning the git remote at startup, we developed [git-kv](https://git.robur.coop/robur/git-kv) which can dump and restore the git state.
Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well. Also, initially we computed all hashes of all tarballs, but with the size increasing (all archives are around 7.5GB) this lead to a major issue of startup time (around 5 minutes on a laptop), so we wanted to save and restore the maps as well.
Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as [fat](https://github.com/mirage/ocaml-fat) may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed [mirage-block-partition](https://github.com/reynir/mirage-block-partition) to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed [oneffs](https://git.robur.io/reynir/oneffs) which stores a single value on the block device. Since neither git state nor the maps are suitable for tar's append-only semantics, and we didn't want to investigate yet another file system - such as [fat](https://github.com/mirage/ocaml-fat) may just work fine, but the code looks slightly bitrot, and the reported issues and non-activity doesn't make this package very trustworthy from our point of view. Instead, we developed [mirage-block-partition](https://github.com/reynir/mirage-block-partition) to partition a block device into two. Then we just store the maps and the git state at the end - the end of a tar archive is 2 blocks of zeroes, so stuff at the far end aren't considered by any tooling. Extending the tar archive is also possible, only the maps and git state needs to be moved to the end (or recomputed). As file system, we developed [oneffs](https://git.robur.coop/reynir/oneffs) which stores a single value on the block device.
We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains [recent enhancements](https://github.com/mirage/mirage-kv/pull/28) of the mirage-kv API, there is a `get_partial`, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory). We observed a high memory usage, since each requested archive was first read from the block device into memory, and then sent out. Thanks to Pierre Alains [recent enhancements](https://github.com/mirage/mirage-kv/pull/28) of the mirage-kv API, there is a `get_partial`, that we use to chunk-wise read the archive and send it via HTTP. Now, the memory usage is around 20MB (the git repository and the generated tarball are kept in memory).