From 0d893bfc27e4c616498ec3502c2877fc2b700a12 Mon Sep 17 00:00:00 2001 From: The Robur Team Date: Tue, 17 Dec 2024 15:36:32 +0000 Subject: [PATCH] Pushed by YOCaml 2 from ec0dec16ef37517b8e979c093d7f2edeeed07482 --- articles/2024-10-29-ptt.html | 206 ++++++++++ articles/2024-12-04-github-sponsor.html | 62 +++ articles/arguments.html | 479 ++++++++++++++++++++++++ articles/dnsvizor01.html | 116 ++++++ articles/finances.html | 410 ++++++++++++++++++++ articles/gptar-update.html | 110 ++++++ tags.html | 188 ++++++++++ 7 files changed, 1571 insertions(+) create mode 100644 articles/2024-10-29-ptt.html create mode 100644 articles/2024-12-04-github-sponsor.html create mode 100644 articles/arguments.html create mode 100644 articles/dnsvizor01.html create mode 100644 articles/finances.html create mode 100644 articles/gptar-update.html create mode 100644 tags.html diff --git a/articles/2024-10-29-ptt.html b/articles/2024-10-29-ptt.html new file mode 100644 index 0000000..0c525a6 --- /dev/null +++ b/articles/2024-10-29-ptt.html @@ -0,0 +1,206 @@ + + + + + + + + Robur's blog - Postes, télégraphes et téléphones, next steps + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

Postes, télégraphes et téléphones, next steps

+

As you know from our article on Robur's +finances, we've just received +funding for our email project. This project +started when I was doing my internship in Cambridge and it's great to see that +it's been able to evolve over time and remain functional. This article will +introduce you to the latest changes to our PTT +project and how far we've got towards providing +an OCaml mailing list service.

+

A Git repository or a simple block device as a database?

+

One issue that came up quickly in our latest experiments with our SMTP stack was +the database of users with an email address. Since we had decided to ‘break +down’ the various stages of an email submission to offer simple unikernels, we +ended up having to deploy 4 unikernels to have a service that worked.

+
    +
  • a unikernel for authentication
    • +
  • a unikernel DKIM-signing the incoming email
    • +
  • one unikernel as primary DNS server
    • +
  • one unikernel sending the signed email to its real destination
    • +
+

And we're only talking here about the submission of an email, the reception +concerns another ‘pipe’.

+

The problem with such an architecture is that some unikernels need to have the +same data: the users. In this case, the first unikernel needs to know the user's +password in order to verify authentication. The final unikernel needs to know +the real destinations of the users.

+

Let's take the example of two users: foo@robur.coop and bar@robur.coop. The +first points to hannes@foo.org and the second to reynir@example.com.

+

If Hannes wants to send a message to bar@robur.coop under the identity of +foo@robur.coop, he will need to authenticate himself to our first unikernel. +This first unikernel must therefore:

+
    +
  1. check that the user foo exists
    2. +
  2. the hashed password used by Hannes is the same as the one in the database
    3. +
+

Next, the email will be signed by our second unikernel. It will then forward the +email to the last unikernel, which will do the actual translation of the +recipients and DNS resolution. In other words:

+
    +
  1. it will see that one (the only) recipient is bar@robur.coop
    2. +
  2. check that bar@robur.coop exists and obtain its real address
    3. +
  3. it will obtain reynir@example.com and perform DNS resolution on +example.com to find out the email server for this domain
    4. +
  4. finally send the email signed by foo@robur.coop to reynir@example.com!
    5. +
+

So the first and last unikernels need to have the same information about our +users. One for the passwords, the second for the real email addresses.

+

But as you know, we're talking about unikernels that exist independently of each +other. What's more, they can't share files and the possibility of them sharing +block-devices remains an open question (and a complex one where parallel access +may be involved). In short, the only way to ‘synchronise’ these unikernels in +relation to common data is with a Git repository.

+

Git has the advantage of being widely used for our unikernels +(primary-git, pasteur, unipi and +contruno). The advantage is that you can track changes, modify +files and notify the unikernel to update itself (using nsupdate, a simple ping +or an http request to the unikernel).

+

The problem is that this requires certain skills. Even if it's ‘simple’ to set +up a Git server and then deploy our unikernels, we can restructure our +architecture and simplify the deployment of an SMTP stack!

+

Elit and OneFFS

+

We have therefore decided to merge the email exchange service and email +submission into a unikernel so that this is the only user information requester.

+

So we decided to use OneFFS as the file system for our database, +which will be a plain JSON file. This is perhaps one of the advantages of +MirageOS, which is that you can decide exactly what you need to implement +specific objectives.

+

In this case, those with experience of Postfix, LDAP or MariaDB could confirm +that configuring an email service should be ‘simpler’ than implementing a +multitude of pipes between different applications and authentication methods.

+

The JSON file is therefore very simple and so is the creation of an OneFFS +image:

+
$ cat >database.json<<EOF
+> [ { "name": "din"
+>   , "password": "xxxxxx"
+>   , "mailboxes": [ "romain.calascibetta@gmail.com" ] } ]
+> EOF
+$ opam install oneffs
+$ oneffs create -i database.json -o database.img
+
+

All you have to do is register this image as a block with albatross and launch +our Elit unikernel with this block-device.

+
$ albatross-client create-block --data=database.img database 1024
+$ albatross-client create --net=service:br0 --block=database:database \
+    elit elit.hvt \
+    --arg=...
+
+

At this stage, and if we add our unikernel signing incoming emails, we have more +or less the same thing as what I've described in my previous articles on +deploying an email service.

+

Multiplex receiving & sending emails

+

The PTT project is a toolkit for implementing SMTP servers. It gives developers +the choice of implementing their logic as they see fit:

+
    +
  • sign an email
    • +
  • resolve destinations according to a database
    • +
  • check SPF information
    • +
  • annotate the email as spam or not
    • +
  • etc.
    • +
+

Previously, PTT was split into 2 parts:

+
    +
  1. management of incoming clients/emails
    2. +
  2. the logic to be applied to incoming emails and their delivery
    3. +
+

The second point was becoming increasingly complex, however, and errors in +sending emails are legion (DMARC non-alignment, the email is too big for the +destination, the destination doesn't exist, etc.). All the more so since, up to +now, PTT could only report these errors via the logs...

+

Hannes immediately mentioned the possibility of separating the logic of the +unikernel from the delivery. This will allow us to deal with temporary failures +(greylisting) as well. So a fundamental change was made:

+
    +
  • improve the sendmail and sendmail-lwt packages (as well as proposing +sendmail-miou!) when sending or submitting an email
    • +
  • improve PTT so that there are now 3 distinct jobs: receiving, what to do with +incoming emails and sending emails
    • +
+

SMTP

+

This finally allows us to describe a clearer error management policy that is +independent of what we want to do with incoming emails. At this stage, we can +look for the Return-Path in emails that we haven't managed to send and notify +the senders!

+

All this is still in the experimental stage and practical cases are needed to +observe how we should handle errors and how others do.

+

Insights & Next goals

+

We're already starting to have a bit of fun with email and we can start sending +and receiving emails right away.

+

We're also already seeing hacking attempts on our unikernel:

+
    +
  • people trying to authenticate themselves without STARTTLS (or with it, +depending on how clever the bot is)
    • +
  • people trying to send emails as non-existent users in our database
    • +
  • we're also seeing content that has nothing to do with SMTP
    • +
+

Above all, this shows that, very early on, bots try to usurp the identity linked +to your server (in our case, osau.re) in order to send spam, authenticate +themselves or simply send ‘stuff’ and observe what happens. In this case, for +all the cases mentioned, Elit (and PTT) reacts well: in other words, it simply +cuts off the connection.

+

We were also able to observe how services such as gmail work. In addition, for +the purposes of a mailing list, email forwarding distorts DMARC verification +(specifically, SPF verification). The case is very simple:

+

foo@gmail.com tries to reply to robur@osau.re. robur@osau.re is a mailing list +to several addresses (one of them is bar@gmail.com). The unikernel will receive +the email and send it to bar@gmail.com. The problem is the alignment between +the From field (which corresponds to foo@gmail.com) and our osau.re server. +From gmail.com's point of view, there is a misalignment between these two +pieces of information and it therefore refuses to receive the email.

+

This is where our next objectives come in:

+
    +
  • finish our DMARC implementation
    • +
  • implement ARC so that our server notifies us that, on our side, the DMARC +check went well and that gmail.com should trust us on this.
    • +
+

There is another way of solving the problem, perhaps a little more problematic, +modify the incoming email and in particular the From field. Although this +could be done quite simply with mrmime, it's better to concentrate on +DMARC and ARC so that we can send our emails as they are and never alter them +(especially as this will invalidate previous DKIM signatures!).

+

Conclusion

+

It's always satisfying to see your projects working ‘more or less’ correctly. +This article will surely be the start of a series on the intricacies of email +and the difficulty of deploying such a service at home.

+

We hope that this NLnet-funded work will enable us to replace our current email +system with unikernels. We're already past the stage where we can, more or less +(without DMARC checking), send emails to each other, which is a big step!

+

So follow our work on our blog and if you like what we're producing (which +involves a whole bunch of protocols and formats - much more than just SMTP), you +can make a donation here!

+ +
+ +
+ + + + diff --git a/articles/2024-12-04-github-sponsor.html b/articles/2024-12-04-github-sponsor.html new file mode 100644 index 0000000..dc1baac --- /dev/null +++ b/articles/2024-12-04-github-sponsor.html @@ -0,0 +1,62 @@ + + + + + + + + Robur's blog - Sponsor us via GitHub + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

Sponsor us via GitHub

+

We're delighted to announce the possibility of helping our cooperative through +the GitHub Sponsors platform. The link is available here:

+

https://github.com/sponsors/robur-coop

+

We would also like to reiterate the possibility of making a donation[1] to our +cooperative via the IBAN of Änderwerk available here (if you need +a tax-deductible donation receipt, please use this form).

+
Account holder: Änderwerk gGmbH
+Subject: robur
+IBAN: DE46 4306 0967 1289 8604 00
+BIC: GENODEM1GLS
+Bank: GLS Gemeinschaftsbank, Christstrasse 9, 44789 Bochum, Germany
+
+

More generally, you can refer to our article which explains our +funding since the creation of Robur and we would like to point out that, +despite our funding, part of our work remains unfunded: in particular with +regard to the maintenance of certain software as well as certain services made +available to our users.

+

We would therefore be delighted if users of our software and services could +finance our work according to their means. GitHub in particular offers an +easy-to-use platform for funding us (even if, in all transparency, it takes a +certain amount from each transaction).

+
    +
  1. +

    In fact, this method is preferable to us as this means it will go directly to us instead of through GitHub and Stripe who will take a small cut of the donation in fees.

    +↩︎︎
+ +
+ +
+ + + + diff --git a/articles/arguments.html b/articles/arguments.html new file mode 100644 index 0000000..e7a9023 --- /dev/null +++ b/articles/arguments.html @@ -0,0 +1,479 @@ + + + + + + + + Robur's blog - Runtime arguments in MirageOS + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

Runtime arguments in MirageOS

+

TL;DR: Passing runtime arguments around is tricky, and prone to change every other month.

+

Motivation

+

Sometimes, as an unikernel developer and also as operator, it's nice to have +some runtime arguments passed to an unikernel. Now, if you're into OCaml, +command-line parsing - together with error messages, man page generation, ... - +can be done by the amazing cmdliner +package from Daniel Bünzli.

+

MirageOS uses cmdliner for command line argument passing. This also enabled +us from the early days to have nice man pages for unikernels (see +my-unikernel-binary --help). There are two kinds +of arguments: those at configuration time (mirage configure), such as the +target to compile for, and those at runtime - when the unikernel is executed.

+

In Mirage 4.8.1 and 4.8.0 (released October 2024) there have been some changes +to command-line arguments, which were motivated by 4.5.0 (released April 2024) +and user feedback.

+

First of all, our current way to pass a custom runtime argument to a unikernel +(unikernel.ml):

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  let term = Arg.(value & opt string "Hello World!" doc) in
+  Mirage_runtime.register_arg term
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+          Logs.info (fun f -> f "%s" (hello ()));
+          Time.sleep_ns (Duration.of_sec 1) >>= fun () -> loop (n - 1)
+    in
+    loop 4
+end
+
+

We define the Cmdliner.Term.t +in line 6 (let term = ..) - which provides documentation ("How to say hello."), the option to +use (["hello"] - which is then translated to --hello=), that it is optional, +of type string (cmdliner allows you to convert the incoming strings to more +complex (or more narrow) data types, with decent error handling).

+

The defined argument is directly passed to Mirage_runtime.register_arg, +(in line 7) so our binding hello is of type unit -> string. +In line 14, the value of the runtime argument is used (hello ()) for printing +a log message.

+

The nice property is that it is all local in unikernel.ml, there are no other +parts involved. It is just a bunch of API calls. The downside is that hello () +should only be evaluated after the function start was called - since the +Mirage_runtime needs to parse and fill in the command line arguments. If you +call hello () earlier, you'll get an exception "Called too early. Please delay +this call to after the start function of the unikernel.". Also, since +Mirage_runtime needs to collect and evaluate the command line arguments, the +Mirage_runtime.register_arg may only be called at top-level, otherwise you'll +get another exception "The function register_arg was called to late. Please call +register_arg before the start function is executed (e.g. in a top-level binding).".

+

Another advantage is, having it all in unikernel.ml means adding and removing +arguments doesn't need another execution of mirage configure. Also, any +type can be used that the unikernel depends on - the config.ml is compiled only +with a small set of dependencies (mirage itself) - and we don't want to impose a +large dependency cone for mirage just because someone may like to use +X509.Key_type.t as argument type.

+

Earlier, before mirage 4.5.0, we had runtime and configure arguments mixed +together. And code was generated when mirage configure was executed to +deal with these arguments. The downsides included: we needed serialization for +all command-line arguments (at configure time you could fill the argument, which +was then serialized, and deserialized at runtime and used unless the argument +was provided explicitly), they had to appear in config.ml (which also means +changing any would need an execution of mirage configure), since they generated code +potential errors were in code that the developer didn't write (though we had +some __POS__ arguments to provide error locations in the developer code).

+

Related recent changes are:

+
    +
  • in mirage 4.8.1, the runtime arguments to configure the OCaml runtime system +(such as GC settings, randomization of hashtables, recording of backtraces) +are now provided using the cmdliner-stdlib +package.
    • +
  • in mirage 4.8.0, for git, dns-client, and happy-eyeballs devices the optional +arguments are generated by default - so they are always available and don't +need to be manually done by the unikernel developer.
    • +
+

Let's dive a bit deeper into the history.

+

History

+

In MirageOS, since the early stages (I'll go back to 2.7.0 (February 2016) where +functoria was introduced) used an embedded fork of cmdliner to handle command +line arguments.

+

Animated changes to the hello world unikernel

+

February 2016 (Mirage 2.7.0)

+

When looking into the MirageOS 2.x series, here's the code for our hello world +unikernel:

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    "Unikernel.Hello" (console @-> job)
+
+let () = register "hello-key" [main $ default_console]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (C: V1_LWT.CONSOLE) = struct
+  let start c =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        C.log c (Key_gen.hello ());
+        OS.Time.sleep 1.0 >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

As you can see, the cmdliner term was provided in config.ml, and in +unikernel.ml the expression Key_gen.hello () was used - Key_gen was +a module generated by the mirage configure invocation.

+

You can as well see that the term was wrapped in Key.create "hello" - where +this string was used as the identifier for the code generation.

+

As mentioned above, a change needed to be done in config.ml and a +mirage configure to take effect.

+

July 2016 (Mirage 2.9.1)

+

The OS.Time was functorized with a Time functor:

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    "Unikernel.Hello" (console @-> time @-> job)
+
+let () = register "hello-key" [main $ default_console $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (C: V1_LWT.CONSOLE) (Time : V1_LWT.TIME) = struct
+  let start c _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        C.log c (Key_gen.hello ());
+        Time.sleep 1.0 >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

February 2017 (Mirage pre3)

+

The Time signature changed, now the sleep_ns function sleeps in nanoseconds. +This avoids floating point numbers at the core of MirageOS. The helper package +duration is used to avoid manual conversions.

+

Also, the console signature changed - and log is now inside the Lwt monad.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (console @-> time @-> job)
+
+let () = register "hello-key" [main $ default_console $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (C: V1_LWT.CONSOLE) (Time : V1_LWT.TIME) = struct
+  let start c _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        C.log c (Key_gen.hello ()) >>= fun () ->
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

February 2017 (Mirage 3)

+

Another big change is that now console is not used anymore, but +logs.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time_lwt.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

January 2020 (Mirage 3.7.0)

+

The _lwt is dropped from the interfaces (we used to have Mirage_time and +Mirage_time_lwt - where the latter was instantiating the former with concrete +types: type 'a io = Lwt.t and type buffer = Cstruct.t -- in a cleanup +session we dropped the _lwt interfaces and opam packages. The reasoning was +that when we'll get around to move to another IO system, we'll move everything +at once anyways. No need to have lwt and something else (async, or nowadays +miou or eio) in a single unikernel.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

October 2021 (Mirage 3.10)

+

Some renamings to fix warnings. Only config.ml changed.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  main
+    ~keys:[key hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

June 2023 (Mirage 4.4)

+

The argument was moved to runtime.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt ~stage:`Run string "Hello World!" doc))
+
+let main =
+  main
+    ~keys:[key hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ());
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

March 2024 (Mirage 4.5)

+

The runtime argument is in config.ml refering to the argument as string +("Unikernel.hello"), and being passed to the start function as argument.

+

config.ml

+
open Mirage
+
+let runtime_args = [ runtime_arg ~pos:__POS__ "Unikernel.hello" ]
+
+let main =
+  main
+    ~runtime_args
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  Arg.(value & opt string "Hello World!" doc)
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time hello =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" hello);
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

October 2024 (Mirage 4.8)

+

Again, moved out of config.ml.

+

config.ml

+
open Mirage
+
+let main =
+  main
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  Mirage_runtime.register_arg Arg.(value & opt string "Hello World!" doc)
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

2024 (Not yet released)

+

This is the future with time defunctorized. Read more in the discussion. +To delay the start function, a dep of noop is introduced.

+

config.ml

+
open Mirage
+
+let main =
+  main
+    ~packages:[package "duration"]
+    ~dep:[dep noop]
+    "Unikernel" job
+
+let () = register "hello-key" [main]
+
+

and unikernel.ml

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  Mirage_runtime.register_arg Arg.(value & opt string "Hello World!" doc)
+
+let start () =
+  let rec loop = function
+    | 0 -> Lwt.return_unit
+    | n ->
+      Logs.info (fun f -> f "%s" (hello ()));
+      Mirage_timer.sleep_ns (Duration.of_sec 1) >>= fun () ->
+      loop (n-1)
+  in
+  loop 4
+
+

Conclusion

+

The history of hello world shows that over time we slowly improve the developer +experience, and removing the boilerplate needed to get MirageOS unikernels up +and running. This is work over a decade including lots of other (here invisible) +improvements to the mirage utility.

+

Our current goal is to minimize the code generated by mirage, since code +generation has lots of issues (e.g. error locations, naming, binary size). It +is a long journey. At the same time, we are working on improving the performance +of MirageOS unikernels, developing unikernels that are useful in the real +world (VPN endpoint, DNSmasq replacement, ...), and also simplifying the +deployment of MirageOS unikernels.

+

If you're interested in MirageOS and using it in your domain, don't hesitate +to reach out to us (via eMail: team@robur.coop) - we're keen to deploy MirageOS +and find more domains where it is useful. If you can spare a dime, we're a +registered non-profit in Germany - and can provide tax-deductable receipts for +donations (more information).

+ +
+ +
+ + + + diff --git a/articles/dnsvizor01.html b/articles/dnsvizor01.html new file mode 100644 index 0000000..fbc488f --- /dev/null +++ b/articles/dnsvizor01.html @@ -0,0 +1,116 @@ + + + + + + + + Robur's blog - Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel

+

TL;DR: We got NGI0 Entrust (via NLnet) funding for developing +DNSvizor - a DNS resolver and +DHCP server. Please help us by sharing with us your dnsmasq +configuration, so we can +prioritize the configuration options to support.

+

Introduction

+

The dynamic host configuration protocol (DHCP) +is fundamental in today's Internet and local networks. It usually runs on your +router (or as a dedicated independent service) and automatically configures +computers that join your network (for example wireless laptops, smartphones) +with an IP address, routing information, a DNS resolver, etc. No manual +configuration is needed once your friends' smartphone got the password of your +wireless network \o/

+

The domain name system (DNS) +is responsible for translating domain names (such as "robur.coop", "nlnet.nl") +to IP addresses (such as 193.30.40.138 or 2a0f:7cc7:7cc7:7c40::138) - used by +computers to talk to each other. Humans can remember domain names instead of +memorizing IP addresses. Computers then use DNS to translate these domain names +to IP addresses to communicate with. DNS is a hierarchic, distributed, +faul-tolerant service.

+

These two protocols are fundamental to today's Internet: without them it would +be much harder for humans to use it.

+

DNSvizor

+

We at robur got funding (from +NGI0 Entrust via NLnet) to continue our work on +DNSvizor - a +MirageOS unikernel that provides DNS resolution and +DHCP service for a network. This is fully implemented in +OCaml.

+

Already at our MirageOS retreats we deployed +such unikernel, to test our DHCP implementation +and our DNS resolver - and found and +fixed issues on-site. At the retreats we have a very limited Internet uplink, +thus caching DNS queries and answers is great for reducing the load on the +uplink.

+

Thanks to the funding we received, we'll be able to work on improving the +performance, but also to finish our DNSSec implementation, provide DNS-over-TLS +and DNS-over-HTTPS services, and also a web interface. DNSvizor will use the +existing dnsmasq configuration +syntax, and provide lots of features from dnsmasq, and also provide features +such as block lists from pi-hole.

+

We are at a point where the basic unikernel (our MVP)

+
    +
  • providing DNS and DHCP services - is ready, and we provide +reproducible binary builds. Phew. This +means that the first step is done. The --dhcp-range from dnsmasq is already +being parsed.
    • +
+

We are now curious on concrete usages of dnsmasq and the configurations you use. +If you're interested in dnsvizor, please open an issue at our repository +with your dnsmasq configuration. This will help us to guide which parts of the configuration to prioritize.

+

Usages of DNSvizor

+

We have several use cases for DNSvizor:

+
    +
  • at your home router to provide DNS resolution and DHCP service, filtering ads,
    • +
  • in the datacenter auto-configuring your machine park,
    • +
  • when running your unikernel swarm to auto-configure them.
    • +
+

The first one is where pi-hole as well fits into, and where dnsmasq is used quite +a lot. The second one is also a domain where dnsmasq is used. The third one is +from our experience that lots of people struggle with deploying MirageOS +unikernels since they have to manually do IP configuration etc. We ourselves +also pass additional information to the unikernels, such as syslog host, +monitoring sink, X.509 certificates or host names, do some DNS provisioning, ...

+

With DNSvizor we will leverage the common configuration options of all +unikernels (reducing the need for boot arguments), and also go a bit further +and make deployment seamless (including adding hostnames to DNS, forwarding +from our reverse TLS proxy, etc.).

+

Conclusion

+

DNSvizor provides DNS resolution and +DHCP service for your network, and already exists :). +Please report issues you +encounter and questions you may have. Also, if you use dnsmasq, please +show us your configuration.

+

If you're interested in MirageOS and using it in your domain, don't hesitate +to reach out to us (via eMail: team@robur.coop) - we're keen to deploy MirageOS +and find more domains where it is useful. If you can +spare a dime, we're a registered non-profit in +Germany - and can provide tax-deductable receipts in Europe.

+ +
+ +
+ + + + diff --git a/articles/finances.html b/articles/finances.html new file mode 100644 index 0000000..6fd86aa --- /dev/null +++ b/articles/finances.html @@ -0,0 +1,410 @@ + + + + + + + + Robur's blog - How has robur financially been doing since 2018? + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

How has robur financially been doing since 2018?

+

Since the beginning, robur has been working on MirageOS unikernels and getting +them deployed. Due to our experience in hierarchical companies, we wanted to +create something different - a workplace without bosses and management. Instead, +we are a collective where everybody has a say on what we do, and who gets how +much money at the end of the month. This means nobody has to write report and +meet any goals - there's no KPI involved. We strive to be a bunch of people +working together nicely and projects that we own and want to bring forward. If +we discover lack of funding, we reach out to (potential) customers to fill our +cash register. Or reach out to people to donate money.

+

Since our mission is fulfilling and already complex - organising ourselves in a +hierarchy-free environment, including the payment, and work on software in a +niche market - we decided from the early days that bookeeping and invoicing +should not be part of our collective. Especially since we want to be free in +what kind of funding we accept - donations, commercial contracts, public +funding. In the books, robur is part of the non-profit company +Änderwerk in Germany - and friends of ours run that +company. They get a cut on each income we generate.

+

To be inclusive and enable everyone to participate in decisions, we are 100% +transparent in our books - every collective member has access to the financial +spreadsheets, contracts, etc. We use a needs-based payment model, so we talk +about the needs everyone has on a regular basis and adjust the salary, everyone +agreeing to all the numbers.

+

2018

+

We started operations in 2018. In late 2017, we got donations (in the form of +bitcoins) by friends who were convinced of our mission. This was 54,194.91 €. +So, in 2018 we started with that money, and tried to find a mission, and +generate income to sustain our salaries.

+

Also, already in 2017, we applied for funding from +Prototypefund on a CalDAV server, +and we received the grant in early 2018. This was another 48,500 €, paid to +individuals (due to reasons, Prototype fund can't cash out to the non-profit - +this put us into some struggle, since we needed some double bookkeeping and +individuals had to dig into health care etc.).

+

We also did in the second half of 2018 a security audit for +Least Authority +(invoicing 19,600 €).

+

And later in 2018 we started on what is now called NetHSM with an initial +design workshop (5,000 €).

+

And lastly, we started to work on a grant to implement TLS 1.3, +funded by Jane Street (via OCaml Labs Consulting). In 2018, we received 12,741.71 €

+

We applied at NLNet for improving the QubesOS firewall developed in MirageOS +(without success), tried to get the IT security prize in Germany (without +success), and to DIAL OSC (without success).

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProjectAmount
Donation54,194.91
Prototypefund48,500.00
Least Authority19,600.00
TLS 1.312,741.71
Nitrokey5,000.00
Total140,036.62

2019

+

We were keen to finish the CalDAV implementation (and start a CardDAV +implementation), and received some financial support from Tarides for it +(15,000 €).

+

The TLS 1.3 work continued, we got in total 68,887.53 €.

+

We also applied to (and got funding from) Prototypefund, once with an OpenVPN-compatible +MirageOS unikernel, +and once with improving the QubesOS firewall developed as MirageOS unikernel. +This means again twice 48,500 €.

+

We also started the implementation work of NetHSM - which still included a lot +of design work - in total the contract was over 82,500 €. In 2019, we invoiced +Nitrokey in 2019 in total 40,500 €.

+

We also received a total of 516.48 € as donations from source unknown to us.

+

We also applied to NLnet with DNSvizor, and +got a grant, but due to buerocratic reasons they couldn't transfer the money to +our non-profit (which was involved with NLnet in some EU grants), and we didn't +get any money in the end.

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProjectAmount
CardDAV15,000.00
TLS 1.368,887.53
OpenVPN48,500.00
QubesOS48,500.00
Donation516.48
Nitrokey40,500.00
Total221,904.01

2020

+

In 2020, we agreed with OCaml Labs Consulting to work on maintenance of OCaml +packages in the MirageOS ecosystem. This was a contract where at the end of the +month, we reported on which PRs and issues we spent how much time. For us, this +was great to have the freedom to work on which OCaml packages we were keen to +get up to speed. In 2020, we received 45,000 € for this maintenance.

+

We finished the TLS 1.3 work (18,659.01 €)

+

We continued to work on the NetHSM project, and invoiced 55,500 €.

+

We received a total of 255 € in donations from sources unknown to us.

+

We applied at reset.tech again with DNSvizor, unfortunately without success.

+

We also applied at NGI pointer to work on reproducible +builds for MirageOS, and a web frontend. Here we got the grant of 200,000 €, +which we worked on in 2021 and 2022.

+
+ + + + + + + + + + + + + + + + + + + + + + + + +
ProjectAmount
OCLC45,000.00
TLS 1.318,659.01
Nitrokey55,500.00
Donations255.00
Total119,414.01

2021

+

As outlined, we worked on reproducible builds of unikernels - rethinking the way +how a unikernel is configured: no more compiled-in secrets, but instead using +boot parameters. We setup the infrastructure for doing daily reproducible +builds, serving system packages via a package repository, and a +web frontend hosting the reproducible builds. +We received in total 120,000 € from NGI Pointer in 2021.

+

Our work on NetHSM continued, including the introduction of elliptic curves +in mirage-crypto (using fiat). The +invoices to Nitrokey summed up to 26,000 € in 2021.

+

We developed in a short timeframe two packages, u2f +and later webauthn for Skolem Labs based +on gift economy. This resulted in +donations of 18,976 €.

+

We agreed with OCSF to work on +conex, which we have not delivered yet +(lots of other things had to be cleared first: we did a security review of opam +(leading to a security advisory), +we got rid of extra-files +in the opam-repository, and we removed the weak hash md5 +from the opam-repository.

+
+ + + + + + + + + + + + + + + + + + + + +
CustomerAmount
NGI Pointer120,000.00
Nitrokey26,000.00
Skolem18,976.00
Total164,976.00

2022

+

We finished our NGI pointer project, and received another 80,000 €.

+

We also did some minor maintenance for Nitrokey, and invoiced 4,500 €.

+

For Tarides, we started another maintaining MirageOS packages (and continuing +our TCP/IP stack), and invoiced in +total 22,500 €.

+

A grant application for bob was rejected, +but a grant application for MirageVPN +got accepted. Both at NLnet within the EU NGI project.

+
+ + + + + + + + + + + + + + + + + + + + +
ProjectAmount
NGI Pointer80,000.00
Nitrokey4,500.00
Tarides22,500.00
Total107,000.00

2023

+

We finished the NetHSM project, and had a final invoice over 2,500 €.

+

We started a collaboration for semgrep, porting some of +their Python code to OCaml. We received in total 37,500 €.

+

We continued the MirageOS opam package maintenance and invoiced in total +89,250 € to Tarides.

+

A grant application on MirageVPN got +accepted (NGI Assure), and we received in total 12,000 € for our work on it. +This is a continuation of our 2019 work funded by Prototypefund.

+

We also wrote various funding applications, including one for +DNSvizor that was +accepted (NGI0 Entrust).

+
+ + + + + + + + + + + + + + + + + + + + + + + + +
CustomerAmount
Nitrokey2,500.00
semgrep37,500.00
Tarides89,250.00
MirageVPN12,000.00
Total141,250.00

2024

+

We're still in the middle of it, but so far we continued the Tarides maintenance +contract (54,937.50 €).

+

We also finished the MirageVPN work, and received another 45,000 €.

+

We had a contract with Semgrep again on porting Python code to OCaml and received 18,559.40 €.

+

We again worked on several successful funding applications, one on +PTT (NGI Zero Core), a continuation of the +NGI DAPSI project - +now realizing mailing lists with our SMTP stack.

+

We also got MTE (NGI Taler) accepted.

+

The below table is until end of September 2024.

+
+ + + + + + + + + + + + + + + + + + + + +
ProjectAmount
Semgrep18,559.40
Tarides62,812.50
MirageVPN45,000.00
Total126,371.90

Total

+

In a single table, here's our income since robur started.

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
YearAmount
2018140,036.62
2019221,904.01
2020119,414.01
2021164,976.00
2022107,000.00
2023141,250.00
2024126,371.90
Total1,020,952.54

Plot of above income table

+

As you can spot, it varies quite a bit. In some years we have fewer money +available than in other years.

+

Expenses

+

As mentioned, the non-profit company Änderwerk running +the bookkeeping and legal stuff (invoices, tax statements, contracts, etc.) gets +a cut on each income we produce. They are doing amazing work and are very +quick responding to our queries.

+

We spend most of our income on salary. Some money we spend on travel. We also +pay monthly for our server (plus some extra for hardware, and in June 2024 a +huge amount for trying to recover data from failed SSDs).

+

Conclusion

+

We have provided an overview of our income, we were three to five people working +at robur over the entire time. As written at the beginning, we use needs-based +payment. Our experience with this is great! It provides a lot of trust into each +other.

+

Our funding is diverse from multiple sources - donations, commercial work, +public funding. This was our initial goal, and we're very happy that it works +fine over the last five years.

+

Taking the numbers into account, we are not paying ourselves "industry standard" +rates - but we really love what we do - and sometimes we just take some time off. +We do work on various projects that we really really enjoy - but where (at the +moment) no funding is available for.

+

We are always happy to discuss how our collective operates. If you're +interested, please drop us a message.

+

Of course, if we receive donations, we use them wisely - mainly for working on +the currently not funded projects (bob, albatross, miou, mollymawk - to name a few). If you +can spare a dime or two, don't hesitate to donate. +Donations are tax-deductable in Germany (and should be in Europe) since we're a +registered non-profit.

+

If you're interested in MirageOS and using it in your domain, don't hesitate +to reach out to us (via eMail: team@robur.coop) so we can start to chat - we're keen to deploy MirageOS +and find more domains where it is useful.

+ +
+ +
+ + + + diff --git a/articles/gptar-update.html b/articles/gptar-update.html new file mode 100644 index 0000000..8bde6d8 --- /dev/null +++ b/articles/gptar-update.html @@ -0,0 +1,110 @@ + + + + + + + + Robur's blog - GPTar (update) + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

GPTar (update)

+

In a previous post I describe how I craft a hybrid GUID partition table (GPT) and tar archive by exploiting that there are disjoint areas of a 512 byte block that are important to tar headers and protective master boot records used in GPT respectively. +I recommend reading it first if you haven't already for context.

+

After writing the above post I read an excellent and fun and totally normal article by Emily on how she created executable tar archives. +Therein I learned a clever hack: +GNU tar has a tar extension for volume headers. +These are essentially labels for your tape archives when you're forced to split an archive across multiple tapes. +They can (seemingly) hold any text as label including shell scripts. +What's more is GNU tar and bsdtar does not extract these as files! +This is excellent, because I don't actually want to extract or list the GPT header when using GNU tar or bsdtar. +This prompted me to use a different link indicator.

+

This worked pretty great. +Listing the archive using GNU tar I still get GPTAR, but with verbose listing it's displayed as a --Volume Header--:

+
$ tar -tvf disk.img
+Vr-------- 0/0           16896 1970-01-01 01:00 GPTAR--Volume Header--
+-rw-r--r-- 0/0              14 1970-01-01 01:00 test.txt
+
+

And more importantly the GPTAR entry is ignored when extracting:

+
$ mkdir tmp
+$ cd tmp/
+$ tar -xf ../disk.img
+$ ls
+test.txt
+
+

BSD tar / libarchive

+

Unfortunately, this broke bsdtar!

+
$ bsdtar -tf disk.img
+bsdtar: Damaged tar archive
+bsdtar: Error exit delayed from previous errors.
+
+

This is annoying because we run FreeBSD on the host for opam.robur.coop, our instance of opam-mirror. +This Autumn we updated opam-mirror to use the hybrid GPT+tar GPTar tartition table[1] instead of hard coded or boot parameter specified disk offsets for the different partitions - which was extremely brittle! +So we were no longer able to inspect the contents of the tar partition from the host! +Unacceptable! +So I started to dig into libarchive where bsdtar comes from. +To my surprise, after building bsdtar from the git clone of the source code it ran perfectly fine!

+
$ ./bsdtar -tf ../gptar/disk.img
+test.txt
+
+

I eventually figure out this change fixed it for me. +I got in touch with Emily to let her know that bsdtar recently fixed this (ab)use of GNU volume headers. +Her reply was basically "as of when I wrote the article, I was pretty sure bsdtar ignored it." +And indeed it did. +Examining the diff further revealed that it ignored the GNU volume header - just not "correctly" when the GNU volume header was abused to carry file content as I did:

+
 /*
+  * Interpret 'V' GNU tar volume header.
+  */
+ static int
+ header_volume(struct archive_read *a, struct tar *tar,
+     struct archive_entry *entry, const void *h, size_t *unconsumed)
+ {
+-       (void)h;
++       const struct archive_entry_header_ustar *header;
++       int64_t size, to_consume;
++
++       (void)a; /* UNUSED */
++       (void)tar; /* UNUSED */
++       (void)entry; /* UNUSED */
+
+-       /* Just skip this and read the next header. */
+-       return (tar_read_header(a, tar, entry, unconsumed));
++       header = (const struct archive_entry_header_ustar *)h;
++       size = tar_atol(header->size, sizeof(header->size));
++       to_consume = ((size + 511) & ~511);
++       *unconsumed += to_consume;
++       return (ARCHIVE_OK);
+ }
+
+

So thanks to the above change we can expect a release of libarchive supporting further flavors of abuse of GNU volume headers! +🥳

+
    +
  1. +

    Emily came up with the much better term "tartition table" than what I had come up with - "GPTar".

    +↩︎︎
+ +
+ +
+ + + + diff --git a/tags.html b/tags.html new file mode 100644 index 0000000..b6f9e91 --- /dev/null +++ b/tags.html @@ -0,0 +1,188 @@ + + + + + + + + Robur's blog - Tags + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

+ Community +

+ +
+

+ MirageVPN +

+ +
+

+ OpenVPN +

+ +
+

+ Python +

+ +
+

+ Scheduler +

+ +
+

+ Unikernel +

+ +
+

+ github +

+ +
+

+ gpt +

+ +
+

+ mbr +

+ +
+

+ persistent storage +

+ +
+

+ tar +

+ +
+

+ unicode +

+ +
+
+ + + +