forked from robur/blog.robur.coop
Pushed by YOCaml 2 from ec0dec16ef
This commit is contained in:
parent
534480737c
commit
0d893bfc27
7 changed files with 1571 additions and 0 deletions
206
articles/2024-10-29-ptt.html
Normal file
206
articles/2024-10-29-ptt.html
Normal file
|
@ -0,0 +1,206 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - Postes, télégraphes et téléphones, next steps
|
||||
</title>
|
||||
<meta name="description" content="An update of our email stack">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<article>
|
||||
<h1>Postes, télégraphes et téléphones, next steps</h1>
|
||||
<ul class="tags-list"><li><a href="/tags.html#tag-SMTP">SMTP</a></li><li><a href="/tags.html#tag-emails">emails</a></li><li><a href="/tags.html#tag-mailing-lists">mailing-lists</a></li></ul><p>As you know from <a href="https://blog.robur.coop/articles/finances.html">our article on Robur's
|
||||
finances</a>, we've just received
|
||||
<a href="https://nlnet.nl/project/PTT">funding for our email project</a>. This project
|
||||
started when I was doing my internship in Cambridge and it's great to see that
|
||||
it's been able to evolve over time and remain functional. This article will
|
||||
introduce you to the latest changes to <a href="https://github.com/mirage/ptt">our PTT
|
||||
project</a> and how far we've got towards providing
|
||||
an OCaml mailing list service.</p>
|
||||
<h2 id="a-git-repository-or-a-simple-block-device-as-a-database"><a class="anchor" aria-hidden="true" href="#a-git-repository-or-a-simple-block-device-as-a-database"></a>A Git repository or a simple block device as a database?</h2>
|
||||
<p>One issue that came up quickly in our latest experiments with our SMTP stack was
|
||||
the database of users with an email address. Since we had decided to ‘break
|
||||
down’ the various stages of an email submission to offer simple unikernels, we
|
||||
ended up having to deploy 4 unikernels to have a service that worked.</p>
|
||||
<ul>
|
||||
<li>a unikernel for authentication</li>
|
||||
<li>a unikernel DKIM-signing the incoming email</li>
|
||||
<li>one unikernel as primary DNS server</li>
|
||||
<li>one unikernel sending the signed email to its real destination</li>
|
||||
</ul>
|
||||
<p>And we're only talking here about the submission of an email, the reception
|
||||
concerns another ‘pipe’.</p>
|
||||
<p>The problem with such an architecture is that some unikernels need to have the
|
||||
same data: the users. In this case, the first unikernel needs to know the user's
|
||||
password in order to verify authentication. The final unikernel needs to know
|
||||
the real destinations of the users.</p>
|
||||
<p>Let's take the example of two users: foo@robur.coop and bar@robur.coop. The
|
||||
first points to hannes@foo.org and the second to reynir@example.com.</p>
|
||||
<p>If Hannes wants to send a message to bar@robur.coop under the identity of
|
||||
foo@robur.coop, he will need to authenticate himself to our first unikernel.
|
||||
This first unikernel must therefore:</p>
|
||||
<ol>
|
||||
<li>check that the user <code>foo</code> exists</li>
|
||||
<li>the hashed password used by Hannes is the same as the one in the database</li>
|
||||
</ol>
|
||||
<p>Next, the email will be signed by our second unikernel. It will then forward the
|
||||
email to the last unikernel, which will do the actual translation of the
|
||||
recipients and DNS resolution. In other words:</p>
|
||||
<ol>
|
||||
<li>it will see that one (the only) recipient is bar@robur.coop</li>
|
||||
<li>check that bar@robur.coop exists and obtain its real address</li>
|
||||
<li>it will obtain reynir@example.com and perform DNS resolution on
|
||||
<code>example.com</code> to find out the email server for this domain</li>
|
||||
<li>finally send the email signed by foo@robur.coop to reynir@example.com!</li>
|
||||
</ol>
|
||||
<p>So the first and last unikernels need to have the same information about our
|
||||
users. One for the passwords, the second for the real email addresses.</p>
|
||||
<p>But as you know, we're talking about unikernels that exist independently of each
|
||||
other. What's more, they can't share files and the possibility of them sharing
|
||||
block-devices remains an open question (and a complex one where parallel access
|
||||
may be involved). In short, the only way to ‘synchronise’ these unikernels in
|
||||
relation to common data is with a Git repository.</p>
|
||||
<p><a href="https://github.com/robur-coop/git-kv">Git</a> has the advantage of being widely used for our unikernels
|
||||
(<a href="https://github.com/robur-coop/dns-primary-git/">primary-git</a>, <a href="https://github.com/dinosaure/pasteur">pasteur</a>, <a href="https://github.com/robur-coop/unipi">unipi</a> and
|
||||
<a href="https://github.com/dinosaure/contruno">contruno</a>). The advantage is that you can track changes, modify
|
||||
files and notify the unikernel to update itself (using nsupdate, a simple ping
|
||||
or an http request to the unikernel).</p>
|
||||
<p>The problem is that this requires certain skills. Even if it's ‘simple’ to set
|
||||
up a Git server and then deploy our unikernels, we can restructure our
|
||||
architecture and simplify the deployment of an SMTP stack!</p>
|
||||
<h2 id="elit-and-oneffs"><a class="anchor" aria-hidden="true" href="#elit-and-oneffs"></a>Elit and OneFFS</h2>
|
||||
<p>We have therefore decided to merge the email exchange service and email
|
||||
submission into a unikernel so that this is the only user information requester.</p>
|
||||
<p>So we decided to use <a href="https://github.com/robur-coop/oneffs">OneFFS</a> as the file system for our database,
|
||||
which will be a plain JSON file. This is perhaps one of the advantages of
|
||||
MirageOS, which is that you can decide exactly what you need to implement
|
||||
specific objectives.</p>
|
||||
<p>In this case, those with experience of Postfix, LDAP or MariaDB could confirm
|
||||
that configuring an email service should be ‘simpler’ than implementing a
|
||||
multitude of pipes between different applications and authentication methods.</p>
|
||||
<p>The JSON file is therefore very simple and so is the creation of an OneFFS
|
||||
image:</p>
|
||||
<pre><code class="language-sh">$ cat >database.json<<EOF
|
||||
> [ { "name": "din"
|
||||
> , "password": "xxxxxx"
|
||||
> , "mailboxes": [ "romain.calascibetta@gmail.com" ] } ]
|
||||
> EOF
|
||||
$ opam install oneffs
|
||||
$ oneffs create -i database.json -o database.img
|
||||
</code></pre>
|
||||
<p>All you have to do is register this image as a block with <a href="https://github.com/robur-coop/albatross">albatross</a> and launch
|
||||
our Elit unikernel with this block-device.</p>
|
||||
<pre><code class="language-sh">$ albatross-client create-block --data=database.img database 1024
|
||||
$ albatross-client create --net=service:br0 --block=database:database \
|
||||
elit elit.hvt \
|
||||
--arg=...
|
||||
</code></pre>
|
||||
<p>At this stage, and if we add our unikernel signing incoming emails, we have more
|
||||
or less the same thing as what I've described in <a href="https://blog.osau.re/articles/smtp_1.html">my previous articles</a> on
|
||||
<a href="https://blog.osau.re/articles/smtp_2.html">deploying</a> an <a href="https://blog.osau.re/articles/smtp_3.html">email service</a>.</p>
|
||||
<h2 id="multiplex-receiving--sending-emails"><a class="anchor" aria-hidden="true" href="#multiplex-receiving--sending-emails"></a>Multiplex receiving & sending emails</h2>
|
||||
<p>The PTT project is a toolkit for implementing SMTP servers. It gives developers
|
||||
the choice of implementing their logic as they see fit:</p>
|
||||
<ul>
|
||||
<li>sign an email</li>
|
||||
<li>resolve destinations according to a database</li>
|
||||
<li>check SPF information</li>
|
||||
<li>annotate the email as spam or not</li>
|
||||
<li>etc.</li>
|
||||
</ul>
|
||||
<p>Previously, PTT was split into 2 parts:</p>
|
||||
<ol>
|
||||
<li>management of incoming clients/emails</li>
|
||||
<li>the logic to be applied to incoming emails and their delivery</li>
|
||||
</ol>
|
||||
<p>The second point was becoming increasingly complex, however, and errors in
|
||||
sending emails are legion (DMARC non-alignment, the email is too big for the
|
||||
destination, the destination doesn't exist, etc.). All the more so since, up to
|
||||
now, PTT could only report these errors via the logs...</p>
|
||||
<p>Hannes immediately mentioned the possibility of separating the logic of the
|
||||
unikernel from the delivery. This will allow us to deal with temporary failures
|
||||
(greylisting) as well. So a fundamental change was made:</p>
|
||||
<ul>
|
||||
<li>improve the <a href="https://github.com/mirage/colombe">sendmail</a> and <code>sendmail-lwt</code> packages (as well as proposing
|
||||
<code>sendmail-miou</code>!) when sending or submitting an email</li>
|
||||
<li>improve PTT so that there are now 3 distinct jobs: receiving, what to do with
|
||||
incoming emails and sending emails</li>
|
||||
</ul>
|
||||
<p><img src="../images/smtp.jpg" alt="SMTP" ></p>
|
||||
<p>This finally allows us to describe a clearer error management policy that is
|
||||
independent of what we want to do with incoming emails. At this stage, we can
|
||||
look for the <code>Return-Path</code> in emails that we haven't managed to send and notify
|
||||
the senders!</p>
|
||||
<p>All this is still in the experimental stage and practical cases are needed to
|
||||
observe how we should handle errors and how others do.</p>
|
||||
<h2 id="insights--next-goals"><a class="anchor" aria-hidden="true" href="#insights--next-goals"></a>Insights & Next goals</h2>
|
||||
<p>We're already starting to have a bit of fun with email and we can start sending
|
||||
and receiving emails right away.</p>
|
||||
<p>We're also already seeing hacking attempts on our unikernel:</p>
|
||||
<ul>
|
||||
<li>people trying to authenticate themselves without <code>STARTTLS</code> (or with it,
|
||||
depending on how clever the bot is)</li>
|
||||
<li>people trying to send emails as non-existent users in our database</li>
|
||||
<li>we're also seeing content that has nothing to do with SMTP</li>
|
||||
</ul>
|
||||
<p>Above all, this shows that, very early on, bots try to usurp the identity linked
|
||||
to your server (in our case, osau.re) in order to send spam, authenticate
|
||||
themselves or simply send ‘stuff’ and observe what happens. In this case, for
|
||||
all the cases mentioned, Elit (and PTT) reacts well: in other words, it simply
|
||||
cuts off the connection.</p>
|
||||
<p>We were also able to observe how services such as gmail work. In addition, for
|
||||
the purposes of a mailing list, email forwarding distorts DMARC verification
|
||||
(specifically, SPF verification). The case is very simple:</p>
|
||||
<p>foo@gmail.com tries to reply to robur@osau.re. robur@osau.re is a mailing list
|
||||
to several addresses (one of them is bar@gmail.com). The unikernel will receive
|
||||
the email and send it to bar@gmail.com. The problem is the alignment between
|
||||
the <code>From</code> field (which corresponds to foo@gmail.com) and our osau.re server.
|
||||
From gmail.com's point of view, there is a misalignment between these two
|
||||
pieces of information and it therefore refuses to receive the email.</p>
|
||||
<p>This is where our next objectives come in:</p>
|
||||
<ul>
|
||||
<li>finish our DMARC implementation</li>
|
||||
<li>implement ARC so that our server notifies us that, on our side, the DMARC
|
||||
check went well and that gmail.com should trust us on this.</li>
|
||||
</ul>
|
||||
<p>There is another way of solving the problem, perhaps a little more problematic,
|
||||
modify the incoming email and in particular the <code>From</code> field. Although this
|
||||
could be done quite simply with <a href="https://github.com/mirage/mrmime">mrmime</a>, it's better to concentrate on
|
||||
DMARC and ARC so that we can send our emails as they are and never alter them
|
||||
(especially as this will invalidate previous DKIM signatures!).</p>
|
||||
<h2 id="conclusion"><a class="anchor" aria-hidden="true" href="#conclusion"></a>Conclusion</h2>
|
||||
<p>It's always satisfying to see your projects working ‘more or less’ correctly.
|
||||
This article will surely be the start of a series on the intricacies of email
|
||||
and the difficulty of deploying such a service at home.</p>
|
||||
<p>We hope that this NLnet-funded work will enable us to replace our current email
|
||||
system with unikernels. We're already past the stage where we can, more or less
|
||||
(without DMARC checking), send emails to each other, which is a big step!</p>
|
||||
<p>So follow our work on our blog and if you like what we're producing (which
|
||||
involves a whole bunch of protocols and formats - much more than just SMTP), you
|
||||
can make <a href="https://robur.coop/Donate">a donation here</a>!</p>
|
||||
|
||||
</article>
|
||||
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
62
articles/2024-12-04-github-sponsor.html
Normal file
62
articles/2024-12-04-github-sponsor.html
Normal file
|
@ -0,0 +1,62 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - Sponsor us via GitHub
|
||||
</title>
|
||||
<meta name="description" content="A new way to sponsor our cooperative">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<article>
|
||||
<h1>Sponsor us via GitHub</h1>
|
||||
<ul class="tags-list"><li><a href="/tags.html#tag-cooperative">cooperative</a></li><li><a href="/tags.html#tag-github">github</a></li></ul><p>We're delighted to announce the possibility of helping our cooperative through
|
||||
the GitHub Sponsors platform. The link is available here:</p>
|
||||
<p><a href="https://github.com/sponsors/robur-coop">https://github.com/sponsors/robur-coop</a></p>
|
||||
<p>We would also like to reiterate the possibility of making a donation<sup><a href="#fn-preferable" id="ref-1-fn-preferable" role="doc-noteref" class="fn-label">[1]</a></sup> to our
|
||||
cooperative via the IBAN of <a href="https://aenderwerk.de/">Änderwerk</a> available here (if you need
|
||||
a tax-deductible donation receipt, please use <a href="https://aenderwerk.de/donate">this form</a>).</p>
|
||||
<pre><code>Account holder: Änderwerk gGmbH
|
||||
Subject: robur
|
||||
IBAN: DE46 4306 0967 1289 8604 00
|
||||
BIC: GENODEM1GLS
|
||||
Bank: GLS Gemeinschaftsbank, Christstrasse 9, 44789 Bochum, Germany
|
||||
</code></pre>
|
||||
<p>More generally, you can refer to our <a href="https://blog.robur.coop/articles/finances.html">article</a> which explains our
|
||||
funding since the creation of Robur and we would like to point out that,
|
||||
despite our funding, part of our work remains unfunded: in particular with
|
||||
regard to the maintenance of certain software as well as certain services made
|
||||
available to our users.</p>
|
||||
<p>We would therefore be delighted if users of our software and services could
|
||||
finance our work according to their means. GitHub in particular offers an
|
||||
easy-to-use platform for funding us (even if, in all transparency, it takes a
|
||||
certain amount from each transaction).</p>
|
||||
<section role="doc-endnotes"><ol>
|
||||
<li id="fn-preferable">
|
||||
<p>In fact, this method is preferable to us as this means it will go directly to us instead of through GitHub and Stripe who will take a small cut of the donation in fees.</p>
|
||||
<span><a href="#ref-1-fn-preferable" role="doc-backlink" class="fn-label">↩︎︎</a></span></li></ol></section>
|
||||
|
||||
</article>
|
||||
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
479
articles/arguments.html
Normal file
479
articles/arguments.html
Normal file
|
@ -0,0 +1,479 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - Runtime arguments in MirageOS
|
||||
</title>
|
||||
<meta name="description" content="The history of runtime arguments to a MirageOS unikernel">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<article>
|
||||
<h1>Runtime arguments in MirageOS</h1>
|
||||
<ul class="tags-list"><li><a href="/tags.html#tag-OCaml">OCaml</a></li><li><a href="/tags.html#tag-MirageOS">MirageOS</a></li></ul><p>TL;DR: Passing runtime arguments around is tricky, and prone to change every other month.</p>
|
||||
<h2 id="motivation"><a class="anchor" aria-hidden="true" href="#motivation"></a>Motivation</h2>
|
||||
<p>Sometimes, as an unikernel developer and also as operator, it's nice to have
|
||||
some runtime arguments passed to an unikernel. Now, if you're into OCaml,
|
||||
command-line parsing - together with error messages, man page generation, ... -
|
||||
can be done by the amazing <a href="https://erratique.ch/software/cmdliner">cmdliner</a>
|
||||
package from Daniel Bünzli.</p>
|
||||
<p>MirageOS uses cmdliner for command line argument passing. This also enabled
|
||||
us from the early days to have nice man pages for unikernels (see
|
||||
<code>my-unikernel-binary --help</code>). There are two kinds
|
||||
of arguments: those at configuration time (<code>mirage configure</code>), such as the
|
||||
target to compile for, and those at runtime - when the unikernel is executed.</p>
|
||||
<p>In Mirage 4.8.1 and 4.8.0 (released October 2024) there have been some changes
|
||||
to command-line arguments, which were motivated by 4.5.0 (released April 2024)
|
||||
and user feedback.</p>
|
||||
<p>First of all, our current way to pass a custom runtime argument to a unikernel
|
||||
(<code>unikernel.ml</code>):</p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
open Cmdliner
|
||||
|
||||
let hello =
|
||||
let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
|
||||
let term = Arg.(value & opt string "Hello World!" doc) in
|
||||
Mirage_runtime.register_arg term
|
||||
|
||||
module Hello (Time : Mirage_time.S) = struct
|
||||
let start _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (hello ()));
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () -> loop (n - 1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<p>We define the <a href="https://erratique.ch/software/cmdliner/doc/Cmdliner/Term/index.html#type-t">Cmdliner.Term.t</a>
|
||||
in line 6 (<code>let term = ..</code>) - which provides documentation ("How to say hello."), the option to
|
||||
use (<code>["hello"]</code> - which is then translated to <code>--hello=</code>), that it is optional,
|
||||
of type <code>string</code> (cmdliner allows you to convert the incoming strings to more
|
||||
complex (or more narrow) data types, with decent error handling).</p>
|
||||
<p>The defined argument is directly passed to <a href="https://ocaml.org/p/mirage-runtime/4.8.1/doc/Mirage_runtime/index.html#val-register_arg"><code>Mirage_runtime.register_arg</code></a>,
|
||||
(in line 7) so our binding <code>hello</code> is of type <code>unit -> string</code>.
|
||||
In line 14, the value of the runtime argument is used (<code>hello ()</code>) for printing
|
||||
a log message.</p>
|
||||
<p>The nice property is that it is all local in <code>unikernel.ml</code>, there are no other
|
||||
parts involved. It is just a bunch of API calls. The downside is that <code>hello ()</code>
|
||||
should only be evaluated after the function <code>start</code> was called - since the
|
||||
<code>Mirage_runtime</code> needs to parse and fill in the command line arguments. If you
|
||||
call <code>hello ()</code> earlier, you'll get an exception "Called too early. Please delay
|
||||
this call to after the start function of the unikernel.". Also, since
|
||||
Mirage_runtime needs to collect and evaluate the command line arguments, the
|
||||
<code>Mirage_runtime.register_arg</code> may only be called at top-level, otherwise you'll
|
||||
get another exception "The function register_arg was called to late. Please call
|
||||
register_arg before the start function is executed (e.g. in a top-level binding).".</p>
|
||||
<p>Another advantage is, having it all in unikernel.ml means adding and removing
|
||||
arguments doesn't need another execution of <code>mirage configure</code>. Also, any
|
||||
type can be used that the unikernel depends on - the config.ml is compiled only
|
||||
with a small set of dependencies (mirage itself) - and we don't want to impose a
|
||||
large dependency cone for mirage just because someone may like to use
|
||||
X509.Key_type.t as argument type.</p>
|
||||
<p>Earlier, before mirage 4.5.0, we had runtime and configure arguments mixed
|
||||
together. And code was generated when <code>mirage configure</code> was executed to
|
||||
deal with these arguments. The downsides included: we needed serialization for
|
||||
all command-line arguments (at configure time you could fill the argument, which
|
||||
was then serialized, and deserialized at runtime and used unless the argument
|
||||
was provided explicitly), they had to appear in <code>config.ml</code> (which also means
|
||||
changing any would need an execution of <code>mirage configure</code>), since they generated code
|
||||
potential errors were in code that the developer didn't write (though we had
|
||||
some <code>__POS__</code> arguments to provide error locations in the developer code).</p>
|
||||
<p>Related recent changes are:</p>
|
||||
<ul>
|
||||
<li>in mirage 4.8.1, the runtime arguments to configure the OCaml runtime system
|
||||
(such as GC settings, randomization of hashtables, recording of backtraces)
|
||||
are now provided using the <a href="https://ocaml.org/p/cmdliner-stdlib">cmdliner-stdlib</a>
|
||||
package.</li>
|
||||
<li>in mirage 4.8.0, for git, dns-client, and happy-eyeballs devices the optional
|
||||
arguments are generated by default - so they are always available and don't
|
||||
need to be manually done by the unikernel developer.</li>
|
||||
</ul>
|
||||
<p>Let's dive a bit deeper into the history.</p>
|
||||
<h2 id="history"><a class="anchor" aria-hidden="true" href="#history"></a>History</h2>
|
||||
<p>In MirageOS, since the early stages (I'll go back to 2.7.0 (February 2016) where
|
||||
functoria was introduced) used an embedded fork of <code>cmdliner</code> to handle command
|
||||
line arguments.</p>
|
||||
<p><a href="https://asciinema.org/a/ruHoadi2oZGOzgzMKk5ZYoFgf"><img src="https://asciinema.org/a/ruHoadi2oZGOzgzMKk5ZYoFgf.svg" alt="Animated changes to the hello world unikernel" ></a></p>
|
||||
<h3 id="february-2016-mirage-270"><a class="anchor" aria-hidden="true" href="#february-2016-mirage-270"></a>February 2016 (Mirage 2.7.0)</h3>
|
||||
<p>When looking into the MirageOS 2.x series, here's the code for our hello world
|
||||
unikernel:</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
foreign
|
||||
~keys:[Key.abstract hello]
|
||||
"Unikernel.Hello" (console @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_console]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (C: V1_LWT.CONSOLE) = struct
|
||||
let start c =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
C.log c (Key_gen.hello ());
|
||||
OS.Time.sleep 1.0 >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<p>As you can see, the cmdliner term was provided in <code>config.ml</code>, and in
|
||||
<code>unikernel.ml</code> the expression <code>Key_gen.hello ()</code> was used - <code>Key_gen</code> was
|
||||
a module generated by the <code>mirage configure</code> invocation.</p>
|
||||
<p>You can as well see that the term was wrapped in <code>Key.create "hello"</code> - where
|
||||
this string was used as the identifier for the code generation.</p>
|
||||
<p>As mentioned above, a change needed to be done in <code>config.ml</code> and a
|
||||
<code>mirage configure</code> to take effect.</p>
|
||||
<h3 id="july-2016-mirage-291"><a class="anchor" aria-hidden="true" href="#july-2016-mirage-291"></a>July 2016 (Mirage 2.9.1)</h3>
|
||||
<p>The <code>OS.Time</code> was functorized with a <code>Time</code> functor:</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
foreign
|
||||
~keys:[Key.abstract hello]
|
||||
"Unikernel.Hello" (console @-> time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_console $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (C: V1_LWT.CONSOLE) (Time : V1_LWT.TIME) = struct
|
||||
let start c _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
C.log c (Key_gen.hello ());
|
||||
Time.sleep 1.0 >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="february-2017-mirage-pre3"><a class="anchor" aria-hidden="true" href="#february-2017-mirage-pre3"></a>February 2017 (Mirage pre3)</h3>
|
||||
<p>The <code>Time</code> signature changed, now the <code>sleep_ns</code> function sleeps in nanoseconds.
|
||||
This avoids floating point numbers at the core of MirageOS. The helper package
|
||||
<code>duration</code> is used to avoid manual conversions.</p>
|
||||
<p>Also, the console signature changed - and <code>log</code> is now inside the Lwt monad.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
foreign
|
||||
~keys:[Key.abstract hello]
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (console @-> time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_console $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (C: V1_LWT.CONSOLE) (Time : V1_LWT.TIME) = struct
|
||||
let start c _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
C.log c (Key_gen.hello ()) >>= fun () ->
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="february-2017-mirage-3"><a class="anchor" aria-hidden="true" href="#february-2017-mirage-3"></a>February 2017 (Mirage 3)</h3>
|
||||
<p>Another big change is that now console is not used anymore, but
|
||||
<a href="https://erratique.ch/software/logs">logs</a>.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
foreign
|
||||
~keys:[Key.abstract hello]
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (Time : Mirage_time_lwt.S) = struct
|
||||
let start _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (Key_gen.hello ()));
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="january-2020-mirage-370"><a class="anchor" aria-hidden="true" href="#january-2020-mirage-370"></a>January 2020 (Mirage 3.7.0)</h3>
|
||||
<p>The <code>_lwt</code> is dropped from the interfaces (we used to have Mirage_time and
|
||||
Mirage_time_lwt - where the latter was instantiating the former with concrete
|
||||
types: <code>type 'a io = Lwt.t</code> and <code>type buffer = Cstruct.t</code> -- in a cleanup
|
||||
session we dropped the <code>_lwt</code> interfaces and opam packages. The reasoning was
|
||||
that when we'll get around to move to another IO system, we'll move everything
|
||||
at once anyways. No need to have <code>lwt</code> and something else (<code>async</code>, or nowadays
|
||||
<code>miou</code> or <code>eio</code>) in a single unikernel.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
foreign
|
||||
~keys:[Key.abstract hello]
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (Time : Mirage_time.S) = struct
|
||||
let start _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (Key_gen.hello ()));
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="october-2021-mirage-310"><a class="anchor" aria-hidden="true" href="#october-2021-mirage-310"></a>October 2021 (Mirage 3.10)</h3>
|
||||
<p>Some renamings to fix warnings. Only <code>config.ml</code> changed.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
main
|
||||
~keys:[key hello]
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (Time : Mirage_time.S) = struct
|
||||
let start _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (Key_gen.hello ()));
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="june-2023-mirage-44"><a class="anchor" aria-hidden="true" href="#june-2023-mirage-44"></a>June 2023 (Mirage 4.4)</h3>
|
||||
<p>The argument was moved to runtime.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let hello =
|
||||
let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
|
||||
Key.(create "hello" Arg.(opt ~stage:`Run string "Hello World!" doc))
|
||||
|
||||
let main =
|
||||
main
|
||||
~keys:[key hello]
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
|
||||
module Hello (Time : Mirage_time.S) = struct
|
||||
let start _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (Key_gen.hello ());
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="march-2024-mirage-45"><a class="anchor" aria-hidden="true" href="#march-2024-mirage-45"></a>March 2024 (Mirage 4.5)</h3>
|
||||
<p>The runtime argument is in <code>config.ml</code> refering to the argument as string
|
||||
("Unikernel.hello"), and being passed to the <code>start</code> function as argument.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let runtime_args = [ runtime_arg ~pos:__POS__ "Unikernel.hello" ]
|
||||
|
||||
let main =
|
||||
main
|
||||
~runtime_args
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
open Cmdliner
|
||||
|
||||
let hello =
|
||||
let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
|
||||
Arg.(value & opt string "Hello World!" doc)
|
||||
|
||||
module Hello (Time : Mirage_time.S) = struct
|
||||
let start _time hello =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" hello);
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="october-2024-mirage-48"><a class="anchor" aria-hidden="true" href="#october-2024-mirage-48"></a>October 2024 (Mirage 4.8)</h3>
|
||||
<p>Again, moved out of <code>config.ml</code>.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let main =
|
||||
main
|
||||
~packages:[package "duration"]
|
||||
"Unikernel.Hello" (time @-> job)
|
||||
|
||||
let () = register "hello-key" [main $ default_time]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
open Cmdliner
|
||||
|
||||
let hello =
|
||||
let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
|
||||
Mirage_runtime.register_arg Arg.(value & opt string "Hello World!" doc)
|
||||
|
||||
module Hello (Time : Mirage_time.S) = struct
|
||||
let start _time =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (hello ()));
|
||||
Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
end
|
||||
</code></pre>
|
||||
<h3 id="2024-not-yet-released"><a class="anchor" aria-hidden="true" href="#2024-not-yet-released"></a>2024 (Not yet released)</h3>
|
||||
<p>This is the future with time defunctorized. Read more in the <a href="https://github.com/mirage/mirage/issues/1513">discussion</a>.
|
||||
To delay the start function, a <code>dep</code> of <code>noop</code> is introduced.</p>
|
||||
<p><code>config.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Mirage
|
||||
|
||||
let main =
|
||||
main
|
||||
~packages:[package "duration"]
|
||||
~dep:[dep noop]
|
||||
"Unikernel" job
|
||||
|
||||
let () = register "hello-key" [main]
|
||||
</code></pre>
|
||||
<p>and <code>unikernel.ml</code></p>
|
||||
<pre><code class="language-OCaml">open Lwt.Infix
|
||||
open Cmdliner
|
||||
|
||||
let hello =
|
||||
let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
|
||||
Mirage_runtime.register_arg Arg.(value & opt string "Hello World!" doc)
|
||||
|
||||
let start () =
|
||||
let rec loop = function
|
||||
| 0 -> Lwt.return_unit
|
||||
| n ->
|
||||
Logs.info (fun f -> f "%s" (hello ()));
|
||||
Mirage_timer.sleep_ns (Duration.of_sec 1) >>= fun () ->
|
||||
loop (n-1)
|
||||
in
|
||||
loop 4
|
||||
</code></pre>
|
||||
<h2 id="conclusion"><a class="anchor" aria-hidden="true" href="#conclusion"></a>Conclusion</h2>
|
||||
<p>The history of hello world shows that over time we slowly improve the developer
|
||||
experience, and removing the boilerplate needed to get MirageOS unikernels up
|
||||
and running. This is work over a decade including lots of other (here invisible)
|
||||
improvements to the mirage utility.</p>
|
||||
<p>Our current goal is to minimize the code generated by mirage, since code
|
||||
generation has lots of issues (e.g. error locations, naming, binary size). It
|
||||
is a long journey. At the same time, we are working on improving the performance
|
||||
of MirageOS unikernels, developing unikernels that are useful in the real
|
||||
world (<a href="https://github.com/robur-coop/miragevpn">VPN endpoint</a>, <a href="https://github.com/robur-coop/dnsvizor">DNSmasq replacement</a>, ...), and also <a href="https://github.com/robur-coop/mollymawk">simplifying the
|
||||
deployment of MirageOS unikernels</a>.</p>
|
||||
<p>If you're interested in MirageOS and using it in your domain, don't hesitate
|
||||
to reach out to us (via eMail: team@robur.coop) - we're keen to deploy MirageOS
|
||||
and find more domains where it is useful. If you can spare a dime, we're a
|
||||
registered non-profit in Germany - and can provide tax-deductable receipts for
|
||||
donations (<a href="https://robur.coop/Donate">more information</a>).</p>
|
||||
|
||||
</article>
|
||||
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
116
articles/dnsvizor01.html
Normal file
116
articles/dnsvizor01.html
Normal file
|
@ -0,0 +1,116 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel
|
||||
</title>
|
||||
<meta name="description" content="The NGI-funded DNSvizor provides core network services on your network; DNS resolution and DHCP.">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<article>
|
||||
<h1>Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel</h1>
|
||||
<ul class="tags-list"><li><a href="/tags.html#tag-OCaml">OCaml</a></li><li><a href="/tags.html#tag-MirageOS">MirageOS</a></li><li><a href="/tags.html#tag-DNSvizor">DNSvizor</a></li></ul><p>TL;DR: We got <a href="https://nlnet.nl/entrust/">NGI0 Entrust (via NLnet)</a> funding for developing
|
||||
<a href="https://nlnet.nl/project/DNSvizor/">DNSvizor</a> - a DNS resolver and
|
||||
DHCP server. Please help us by <a href="https://github.com/robur-coop/dnsvizor/issues/new">sharing with us your dnsmasq
|
||||
configuration</a>, so we can
|
||||
prioritize the configuration options to support.</p>
|
||||
<h2 id="introduction"><a class="anchor" aria-hidden="true" href="#introduction"></a>Introduction</h2>
|
||||
<p>The <a href="https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol">dynamic host configuration protocol (DHCP)</a>
|
||||
is fundamental in today's Internet and local networks. It usually runs on your
|
||||
router (or as a dedicated independent service) and automatically configures
|
||||
computers that join your network (for example wireless laptops, smartphones)
|
||||
with an IP address, routing information, a DNS resolver, etc. No manual
|
||||
configuration is needed once your friends' smartphone got the password of your
|
||||
wireless network \o/</p>
|
||||
<p>The <a href="https://en.wikipedia.org/wiki/Domain_Name_System">domain name system (DNS)</a>
|
||||
is responsible for translating domain names (such as "robur.coop", "nlnet.nl")
|
||||
to IP addresses (such as 193.30.40.138 or 2a0f:7cc7:7cc7:7c40::138) - used by
|
||||
computers to talk to each other. Humans can remember domain names instead of
|
||||
memorizing IP addresses. Computers then use DNS to translate these domain names
|
||||
to IP addresses to communicate with. DNS is a hierarchic, distributed,
|
||||
faul-tolerant service.</p>
|
||||
<p>These two protocols are fundamental to today's Internet: without them it would
|
||||
be much harder for humans to use it.</p>
|
||||
<h2 id="dnsvizor"><a class="anchor" aria-hidden="true" href="#dnsvizor"></a>DNSvizor</h2>
|
||||
<p>We at <a href="https://robur.coop">robur</a> got funding (from
|
||||
<a href="https://nlnet.nl/project/DNSvizor/">NGI0 Entrust via NLnet</a>) to continue our work on
|
||||
<a href="https://github.com/robur-coop/dnsvizor">DNSvizor</a> - a
|
||||
<a href="https://mirageos.org">MirageOS unikernel</a> that provides DNS resolution and
|
||||
DHCP service for a network. This is fully implemented in
|
||||
<a href="https://ocaml.org">OCaml</a>.</p>
|
||||
<p>Already at our <a href="https://retreat.mirageos.org">MirageOS retreats</a> we deployed
|
||||
such unikernel, to test our <a href="https://github.com/mirage/charrua">DHCP implementation</a>
|
||||
and our <a href="https://github.com/mirage/ocaml-dns">DNS resolver</a> - and found and
|
||||
fixed issues on-site. At the retreats we have a very limited Internet uplink,
|
||||
thus caching DNS queries and answers is great for reducing the load on the
|
||||
uplink.</p>
|
||||
<p>Thanks to the funding we received, we'll be able to work on improving the
|
||||
performance, but also to finish our DNSSec implementation, provide DNS-over-TLS
|
||||
and DNS-over-HTTPS services, and also a web interface. DNSvizor will use the
|
||||
existing <a href="https://thekelleys.org.uk/dnsmasq/doc.html">dnsmasq</a> configuration
|
||||
syntax, and provide lots of features from dnsmasq, and also provide features
|
||||
such as block lists from <a href="https://pi-hole.net/">pi-hole</a>.</p>
|
||||
<p>We are at a point where the <a href="https://github.com/robur-coop/dnsvizor">basic unikernel (our MVP)</a></p>
|
||||
<ul>
|
||||
<li>providing DNS and DHCP services - is ready, and we provide
|
||||
<a href="https://builds.robur.coop/job/dnsvizor">reproducible binary builds</a>. Phew. This
|
||||
means that the first step is done. The <code>--dhcp-range</code> from dnsmasq is already
|
||||
being parsed.</li>
|
||||
</ul>
|
||||
<p>We are now curious on concrete usages of dnsmasq and the configurations you use.
|
||||
If you're interested in dnsvizor, please <a href="https://github.com/robur-coop/dnsvizor/issues/new">open an issue at our repository</a>
|
||||
with your dnsmasq configuration. This will help us to guide which parts of the configuration to prioritize.</p>
|
||||
<h2 id="usages-of-dnsvizor"><a class="anchor" aria-hidden="true" href="#usages-of-dnsvizor"></a>Usages of DNSvizor</h2>
|
||||
<p>We have several use cases for DNSvizor:</p>
|
||||
<ul>
|
||||
<li>at your home router to provide DNS resolution and DHCP service, filtering ads,</li>
|
||||
<li>in the datacenter auto-configuring your machine park,</li>
|
||||
<li>when running your unikernel swarm to auto-configure them.</li>
|
||||
</ul>
|
||||
<p>The first one is where pi-hole as well fits into, and where dnsmasq is used quite
|
||||
a lot. The second one is also a domain where dnsmasq is used. The third one is
|
||||
from our experience that lots of people struggle with deploying MirageOS
|
||||
unikernels since they have to manually do IP configuration etc. We ourselves
|
||||
also pass additional information to the unikernels, such as syslog host,
|
||||
monitoring sink, X.509 certificates or host names, do some DNS provisioning, ...</p>
|
||||
<p>With DNSvizor we will leverage the common configuration options of all
|
||||
unikernels (reducing the need for boot arguments), and also go a bit further
|
||||
and make deployment seamless (including adding hostnames to DNS, forwarding
|
||||
from our reverse TLS proxy, etc.).</p>
|
||||
<h2 id="conclusion"><a class="anchor" aria-hidden="true" href="#conclusion"></a>Conclusion</h2>
|
||||
<p><a href="https://github.com/robur-coop/dnsvizor">DNSvizor</a> provides DNS resolution and
|
||||
DHCP service for your network, and <a href="https://builds.robur.coop/job/dnsvizor">already exists</a> :).
|
||||
Please <a href="https://github.com/robur-coop/dnsvizor/issues/">report issues</a> you
|
||||
encounter and questions you may have. Also, if you use dnsmasq, please
|
||||
<a href="https://github.com/robur-coop/dnsvizor/issues/new">show us your configuration</a>.</p>
|
||||
<p>If you're interested in MirageOS and using it in your domain, don't hesitate
|
||||
to reach out to us (via eMail: team@robur.coop) - we're keen to deploy MirageOS
|
||||
and find more domains where it is useful. If you can
|
||||
<a href="https://robur.coop/Donate">spare a dime</a>, we're a registered non-profit in
|
||||
Germany - and can provide tax-deductable receipts in Europe.</p>
|
||||
|
||||
</article>
|
||||
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
410
articles/finances.html
Normal file
410
articles/finances.html
Normal file
|
@ -0,0 +1,410 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - How has robur financially been doing since 2018?
|
||||
</title>
|
||||
<meta name="description" content="How we organise as a collective, and why we're doing that.">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<article>
|
||||
<h1>How has robur financially been doing since 2018?</h1>
|
||||
<ul class="tags-list"><li><a href="/tags.html#tag-finances">finances</a></li><li><a href="/tags.html#tag-cooperative">cooperative</a></li></ul><p>Since the beginning, robur has been working on MirageOS unikernels and getting
|
||||
them deployed. Due to our experience in hierarchical companies, we wanted to
|
||||
create something different - a workplace without bosses and management. Instead,
|
||||
we are a collective where everybody has a say on what we do, and who gets how
|
||||
much money at the end of the month. This means nobody has to write report and
|
||||
meet any goals - there's no KPI involved. We strive to be a bunch of people
|
||||
working together nicely and projects that we own and want to bring forward. If
|
||||
we discover lack of funding, we reach out to (potential) customers to fill our
|
||||
cash register. Or reach out to people to donate money.</p>
|
||||
<p>Since our mission is fulfilling and already complex - organising ourselves in a
|
||||
hierarchy-free environment, including the payment, and work on software in a
|
||||
niche market - we decided from the early days that bookeeping and invoicing
|
||||
should not be part of our collective. Especially since we want to be free in
|
||||
what kind of funding we accept - donations, commercial contracts, public
|
||||
funding. In the books, robur is part of the non-profit company
|
||||
<a href="https://aenderwerk.de">Änderwerk</a> in Germany - and friends of ours run that
|
||||
company. They get a cut on each income we generate.</p>
|
||||
<p>To be inclusive and enable everyone to participate in decisions, we are 100%
|
||||
transparent in our books - every collective member has access to the financial
|
||||
spreadsheets, contracts, etc. We use a needs-based payment model, so we talk
|
||||
about the needs everyone has on a regular basis and adjust the salary, everyone
|
||||
agreeing to all the numbers.</p>
|
||||
<h2 id="2018"><a class="anchor" aria-hidden="true" href="#2018"></a>2018</h2>
|
||||
<p>We started operations in 2018. In late 2017, we got donations (in the form of
|
||||
bitcoins) by friends who were convinced of our mission. This was 54,194.91 €.
|
||||
So, in 2018 we started with that money, and tried to find a mission, and
|
||||
generate income to sustain our salaries.</p>
|
||||
<p>Also, already in 2017, we applied for funding from
|
||||
<a href="https://prototypefund.de">Prototypefund</a> on a <a href="https://prototypefund.de/project/robur-io/">CalDAV server</a>,
|
||||
and we received the grant in early 2018. This was another 48,500 €, paid to
|
||||
individuals (due to reasons, Prototype fund can't cash out to the non-profit -
|
||||
this put us into some struggle, since we needed some double bookkeeping and
|
||||
individuals had to dig into health care etc.).</p>
|
||||
<p>We also did in the second half of 2018 a security audit for
|
||||
<a href="https://leastauthority.com/blog/audits/five-security-audits-for-the-tezos-foundation/">Least Authority</a>
|
||||
(invoicing 19,600 €).</p>
|
||||
<p>And later in 2018 we started on what is now called NetHSM with an initial
|
||||
design workshop (5,000 €).</p>
|
||||
<p>And lastly, we started to work on a grant to implement <a href="https://datatracker.ietf.org/doc/html/rfc8446">TLS 1.3</a>,
|
||||
funded by Jane Street (via OCaml Labs Consulting). In 2018, we received 12,741.71 €</p>
|
||||
<p>We applied at NLNet for improving the QubesOS firewall developed in MirageOS
|
||||
(without success), tried to get the IT security prize in Germany (without
|
||||
success), and to DIAL OSC (without success).</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Project</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Donation</td>
|
||||
<td class="right">54,194.91</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Prototypefund</td>
|
||||
<td class="right">48,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Least Authority</td>
|
||||
<td class="right">19,600.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>TLS 1.3</td>
|
||||
<td class="right">12,741.71</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Nitrokey</td>
|
||||
<td class="right">5,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>140,036.62</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="2019"><a class="anchor" aria-hidden="true" href="#2019"></a>2019</h2>
|
||||
<p>We were keen to finish the CalDAV implementation (and start a CardDAV
|
||||
implementation), and received some financial support from Tarides for it
|
||||
(15,000 €).</p>
|
||||
<p>The TLS 1.3 work continued, we got in total 68,887.53 €.</p>
|
||||
<p>We also applied to (and got funding from) Prototypefund, once with an <a href="https://prototypefund.de/en/project/robust-openvpn-client-with-low-use-of-resources/">OpenVPN-compatible
|
||||
MirageOS unikernel</a>,
|
||||
and once with <a href="https://prototypefund.de/project/portable-firewall-fuer-qubesos/">improving the QubesOS firewall developed as MirageOS unikernel</a>.
|
||||
This means again twice 48,500 €.</p>
|
||||
<p>We also started the implementation work of NetHSM - which still included a lot
|
||||
of design work - in total the contract was over 82,500 €. In 2019, we invoiced
|
||||
Nitrokey in 2019 in total 40,500 €.</p>
|
||||
<p>We also received a total of 516.48 € as donations from source unknown to us.</p>
|
||||
<p>We also applied to NLnet with <a href="https://nlnet.nl/project/Robur/">DNSvizor</a>, and
|
||||
got a grant, but due to buerocratic reasons they couldn't transfer the money to
|
||||
our non-profit (which was involved with NLnet in some EU grants), and we didn't
|
||||
get any money in the end.</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Project</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>CardDAV</td>
|
||||
<td class="right">15,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>TLS 1.3</td>
|
||||
<td class="right">68,887.53</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>OpenVPN</td>
|
||||
<td class="right">48,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>QubesOS</td>
|
||||
<td class="right">48,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Donation</td>
|
||||
<td class="right">516.48</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Nitrokey</td>
|
||||
<td class="right">40,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>221,904.01</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="2020"><a class="anchor" aria-hidden="true" href="#2020"></a>2020</h2>
|
||||
<p>In 2020, we agreed with OCaml Labs Consulting to work on maintenance of OCaml
|
||||
packages in the MirageOS ecosystem. This was a contract where at the end of the
|
||||
month, we reported on which PRs and issues we spent how much time. For us, this
|
||||
was great to have the freedom to work on which OCaml packages we were keen to
|
||||
get up to speed. In 2020, we received 45,000 € for this maintenance.</p>
|
||||
<p>We finished the TLS 1.3 work (18,659.01 €)</p>
|
||||
<p>We continued to work on the NetHSM project, and invoiced 55,500 €.</p>
|
||||
<p>We received a total of 255 € in donations from sources unknown to us.</p>
|
||||
<p>We applied at reset.tech again with DNSvizor, unfortunately without success.</p>
|
||||
<p>We also applied at <a href="https://pointer.ngi.eu">NGI pointer</a> to work on reproducible
|
||||
builds for MirageOS, and a web frontend. Here we got the grant of 200,000 €,
|
||||
which we worked on in 2021 and 2022.</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Project</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>OCLC</td>
|
||||
<td class="right">45,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>TLS 1.3</td>
|
||||
<td class="right">18,659.01</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Nitrokey</td>
|
||||
<td class="right">55,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Donations</td>
|
||||
<td class="right">255.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>119,414.01</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="2021"><a class="anchor" aria-hidden="true" href="#2021"></a>2021</h2>
|
||||
<p>As outlined, we worked on reproducible builds of unikernels - rethinking the way
|
||||
how a unikernel is configured: no more compiled-in secrets, but instead using
|
||||
boot parameters. We setup the infrastructure for doing daily reproducible
|
||||
builds, serving system packages via a package repository, and a
|
||||
<a href="https://builds.robur.coop">web frontend</a> hosting the reproducible builds.
|
||||
We received in total 120,000 € from NGI Pointer in 2021.</p>
|
||||
<p>Our work on NetHSM continued, including the introduction of elliptic curves
|
||||
in mirage-crypto (using <a href="https://github.com/mit-plv/fiat-crypto/">fiat</a>). The
|
||||
invoices to Nitrokey summed up to 26,000 € in 2021.</p>
|
||||
<p>We developed in a short timeframe two packages, <a href="https://github.com/robur-coop/u2f">u2f</a>
|
||||
and later <a href="https://git.robur.coop/robur/webauthn">webauthn</a> for Skolem Labs based
|
||||
on <a href="https://en.wikipedia.org/wiki/Gift_economy">gift economy</a>. This resulted in
|
||||
donations of 18,976 €.</p>
|
||||
<p>We agreed with <a href="https://ocaml-sf.org/">OCSF</a> to work on
|
||||
<a href="https://github.com/hannesm/conex">conex</a>, which we have not delivered yet
|
||||
(lots of other things had to be cleared first: we did a security review of opam
|
||||
(leading to <a href="https://opam.ocaml.org/blog/opam-2-1-5-local-cache/">a security advisory</a>),
|
||||
we got rid of <a href="https://discuss.ocaml.org/t/ann-opam-repository-policy-change-checksums-no-md5-and-no-extra-files"><code>extra-files</code></a>
|
||||
in the opam-repository, and we <a href="https://discuss.ocaml.org/t/ann-opam-repository-policy-change-checksums-no-md5-and-no-extra-files">removed the weak hash md5</a>
|
||||
from the opam-repository.</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Customer</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>NGI Pointer</td>
|
||||
<td class="right">120,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Nitrokey</td>
|
||||
<td class="right">26,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Skolem</td>
|
||||
<td class="right">18,976.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>164,976.00</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="2022"><a class="anchor" aria-hidden="true" href="#2022"></a>2022</h2>
|
||||
<p>We finished our NGI pointer project, and received another 80,000 €.</p>
|
||||
<p>We also did some minor maintenance for Nitrokey, and invoiced 4,500 €.</p>
|
||||
<p>For Tarides, we started another maintaining MirageOS packages (and continuing
|
||||
<a href="https://github.com/robur-coop/utcp">our TCP/IP stack</a>), and invoiced in
|
||||
total 22,500 €.</p>
|
||||
<p>A grant application for <a href="https://github.com/dinosaure/bob/">bob</a> was rejected,
|
||||
but a grant application for <a href="https://github.com/robur-coop/miragevpn">MirageVPN</a>
|
||||
got accepted. Both at NLnet within the EU NGI project.</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Project</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>NGI Pointer</td>
|
||||
<td class="right">80,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Nitrokey</td>
|
||||
<td class="right">4,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Tarides</td>
|
||||
<td class="right">22,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>107,000.00</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="2023"><a class="anchor" aria-hidden="true" href="#2023"></a>2023</h2>
|
||||
<p>We finished the NetHSM project, and had a final invoice over 2,500 €.</p>
|
||||
<p>We started a collaboration for <a href="https://semgrep.dev">semgrep</a>, porting some of
|
||||
their Python code to OCaml. We received in total 37,500 €.</p>
|
||||
<p>We continued the MirageOS opam package maintenance and invoiced in total
|
||||
89,250 € to Tarides.</p>
|
||||
<p>A grant application on <a href="https://nlnet.nl/project/MirageVPN/">MirageVPN</a> got
|
||||
accepted (NGI Assure), and we received in total 12,000 € for our work on it.
|
||||
This is a continuation of our 2019 work funded by Prototypefund.</p>
|
||||
<p>We also wrote various funding applications, including one for
|
||||
<a href="https://github.com/robur-coop/dnsvizor">DNSvizor</a> that was
|
||||
<a href="https://nlnet.nl/project/DNSvizor/">accepted</a> (NGI0 Entrust).</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Customer</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Nitrokey</td>
|
||||
<td class="right">2,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>semgrep</td>
|
||||
<td class="right">37,500.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Tarides</td>
|
||||
<td class="right">89,250.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MirageVPN</td>
|
||||
<td class="right">12,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>141,250.00</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="2024"><a class="anchor" aria-hidden="true" href="#2024"></a>2024</h2>
|
||||
<p>We're still in the middle of it, but so far we continued the Tarides maintenance
|
||||
contract (54,937.50 €).</p>
|
||||
<p>We also finished the MirageVPN work, and received another 45,000 €.</p>
|
||||
<p>We had a contract with Semgrep again on porting Python code to OCaml and received 18,559.40 €.</p>
|
||||
<p>We again worked on several successful funding applications, one on
|
||||
<a href="https://nlnet.nl/project/PTT/">PTT</a> (NGI Zero Core), a continuation of the
|
||||
<a href="https://www.ngi.eu/funded_solution/ngi-dapsiproject-24/">NGI DAPSI</a> project -
|
||||
now realizing mailing lists with our SMTP stack.</p>
|
||||
<p>We also got <a href="https://nlnet.nl/project/MTE/">MTE</a> (NGI Taler) accepted.</p>
|
||||
<p>The below table is until end of September 2024.</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Project</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Semgrep</td>
|
||||
<td class="right">18,559.40</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Tarides</td>
|
||||
<td class="right">62,812.50</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MirageVPN</td>
|
||||
<td class="right">45,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>126,371.90</strong></td>
|
||||
</tr>
|
||||
</table></div><h2 id="total"><a class="anchor" aria-hidden="true" href="#total"></a>Total</h2>
|
||||
<p>In a single table, here's our income since robur started.</p>
|
||||
<div role="region"><table>
|
||||
<tr>
|
||||
<th>Year</th>
|
||||
<th class="right">Amount</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2018</td>
|
||||
<td class="right">140,036.62</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2019</td>
|
||||
<td class="right">221,904.01</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2020</td>
|
||||
<td class="right">119,414.01</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2021</td>
|
||||
<td class="right">164,976.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2022</td>
|
||||
<td class="right">107,000.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2023</td>
|
||||
<td class="right">141,250.00</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2024</td>
|
||||
<td class="right">126,371.90</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="right"><strong>1,020,952.54</strong></td>
|
||||
</tr>
|
||||
</table></div><p><img src="../images/finances.png" alt="Plot of above income table" ></p>
|
||||
<p>As you can spot, it varies quite a bit. In some years we have fewer money
|
||||
available than in other years.</p>
|
||||
<h2 id="expenses"><a class="anchor" aria-hidden="true" href="#expenses"></a>Expenses</h2>
|
||||
<p>As mentioned, the non-profit company <a href="https://aenderwerk.de">Änderwerk</a> running
|
||||
the bookkeeping and legal stuff (invoices, tax statements, contracts, etc.) gets
|
||||
a cut on each income we produce. They are doing amazing work and are very
|
||||
quick responding to our queries.</p>
|
||||
<p>We spend most of our income on salary. Some money we spend on travel. We also
|
||||
pay monthly for our server (plus some extra for hardware, and in June 2024 a
|
||||
huge amount for trying to recover data from failed SSDs).</p>
|
||||
<h2 id="conclusion"><a class="anchor" aria-hidden="true" href="#conclusion"></a>Conclusion</h2>
|
||||
<p>We have provided an overview of our income, we were three to five people working
|
||||
at robur over the entire time. As written at the beginning, we use needs-based
|
||||
payment. Our experience with this is great! It provides a lot of trust into each
|
||||
other.</p>
|
||||
<p>Our funding is diverse from multiple sources - donations, commercial work,
|
||||
public funding. This was our initial goal, and we're very happy that it works
|
||||
fine over the last five years.</p>
|
||||
<p>Taking the numbers into account, we are not paying ourselves "industry standard"
|
||||
rates - but we really love what we do - and sometimes we just take some time off.
|
||||
We do work on various projects that we really really enjoy - but where (at the
|
||||
moment) no funding is available for.</p>
|
||||
<p>We are always happy to discuss how our collective operates. If you're
|
||||
interested, please drop us a message.</p>
|
||||
<p>Of course, if we receive donations, we use them wisely - mainly for working on
|
||||
the currently not funded projects (bob, albatross, miou, mollymawk - to name a few). If you
|
||||
can spare a dime or two, don't hesitate to <a href="https://robur.coop/Donate">donate</a>.
|
||||
Donations are tax-deductable in Germany (and should be in Europe) since we're a
|
||||
registered non-profit.</p>
|
||||
<p>If you're interested in MirageOS and using it in your domain, don't hesitate
|
||||
to reach out to us (via eMail: team@robur.coop) so we can start to chat - we're keen to deploy MirageOS
|
||||
and find more domains where it is useful.</p>
|
||||
|
||||
</article>
|
||||
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
110
articles/gptar-update.html
Normal file
110
articles/gptar-update.html
Normal file
|
@ -0,0 +1,110 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - GPTar (update)
|
||||
</title>
|
||||
<meta name="description" content="libarchive vs hybrid GUID partition table and GNU tar volume header">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<article>
|
||||
<h1>GPTar (update)</h1>
|
||||
<ul class="tags-list"><li><a href="/tags.html#tag-OCaml">OCaml</a></li><li><a href="/tags.html#tag-gpt">gpt</a></li><li><a href="/tags.html#tag-tar">tar</a></li><li><a href="/tags.html#tag-mbr">mbr</a></li><li><a href="/tags.html#tag-persistent storage">persistent storage</a></li></ul><p>In a <a href="gptar.html">previous post</a> I describe how I craft a hybrid GUID partition table (GPT) and tar archive by exploiting that there are disjoint areas of a 512 byte <em>block</em> that are important to tar headers and <em>protective</em> master boot records used in GPT respectively.
|
||||
I recommend reading it first if you haven't already for context.</p>
|
||||
<p>After writing the above post I read an excellent and fun <em>and totally normal</em> article by Emily on how <a href="https://uni.horse/executable-tarballs.html">she created <strong>executable</strong> tar archives</a>.
|
||||
Therein I learned a clever hack:
|
||||
GNU tar has a tar extension for <em>volume headers</em>.
|
||||
These are essentially labels for your tape archives when you're forced to split an archive across multiple tapes.
|
||||
They can (seemingly) hold any text as label including shell scripts.
|
||||
What's more is GNU tar and bsdtar <strong>does not</strong> extract these as files!
|
||||
This is excellent, because I don't actually want to extract or list the GPT header when using GNU tar or bsdtar.
|
||||
This prompted me to <a href="https://github.com/reynir/gptar/pull/1">use a different link indicator</a>.</p>
|
||||
<p>This worked pretty great.
|
||||
Listing the archive using GNU tar I still get <code>GPTAR</code>, but with verbose listing it's displayed as a <code>--Volume Header--</code>:</p>
|
||||
<pre><code class="language-shell">$ tar -tvf disk.img
|
||||
Vr-------- 0/0 16896 1970-01-01 01:00 GPTAR--Volume Header--
|
||||
-rw-r--r-- 0/0 14 1970-01-01 01:00 test.txt
|
||||
</code></pre>
|
||||
<p>And more importantly the <code>GPTAR</code> entry is ignored when extracting:</p>
|
||||
<pre><code class="language-shell">$ mkdir tmp
|
||||
$ cd tmp/
|
||||
$ tar -xf ../disk.img
|
||||
$ ls
|
||||
test.txt
|
||||
</code></pre>
|
||||
<h2 id="bsd-tar--libarchive"><a class="anchor" aria-hidden="true" href="#bsd-tar--libarchive"></a>BSD tar / libarchive</h2>
|
||||
<p>Unfortunately, this broke bsdtar!</p>
|
||||
<pre><code class="language-shell">$ bsdtar -tf disk.img
|
||||
bsdtar: Damaged tar archive
|
||||
bsdtar: Error exit delayed from previous errors.
|
||||
</code></pre>
|
||||
<p>This is annoying because we run FreeBSD on the host for <a href="https://opam.robur.coop">opam.robur.coop</a>, our instance of <a href="https://git.robur.coop/robur/opam-mirror/">opam-mirror</a>.
|
||||
This Autumn we updated <a href="https://git.robur.coop/robur/opam-mirror/">opam-mirror</a> to use the hybrid GPT+tar GPTar <em>tartition table</em><sup><a href="#fn-tartition" id="ref-1-fn-tartition" role="doc-noteref" class="fn-label">[1]</a></sup> instead of hard coded or boot parameter specified disk offsets for the different partitions - which was extremely brittle!
|
||||
So we were no longer able to inspect the contents of the tar partition from the host!
|
||||
Unacceptable!
|
||||
So I started to dig into libarchive where bsdtar comes from.
|
||||
To my surprise, after building bsdtar from the git clone of the source code it ran perfectly fine!</p>
|
||||
<pre><code class="language-shell">$ ./bsdtar -tf ../gptar/disk.img
|
||||
test.txt
|
||||
</code></pre>
|
||||
<p>I eventually figure out <a href="https://github.com/libarchive/libarchive/pull/2127">this change</a> fixed it for me.
|
||||
I got in touch with Emily to let her know that bsdtar recently fixed this (ab)use of GNU volume headers.
|
||||
Her reply was basically "as of when I wrote the article, I was pretty sure bsdtar ignored it."
|
||||
And indeed it did.
|
||||
Examining the diff further revealed that it ignored the GNU volume header - just not "correctly" when the GNU volume header was abused to carry file content as I did:</p>
|
||||
<pre><code class="language-diff"> /*
|
||||
* Interpret 'V' GNU tar volume header.
|
||||
*/
|
||||
static int
|
||||
header_volume(struct archive_read *a, struct tar *tar,
|
||||
struct archive_entry *entry, const void *h, size_t *unconsumed)
|
||||
{
|
||||
- (void)h;
|
||||
+ const struct archive_entry_header_ustar *header;
|
||||
+ int64_t size, to_consume;
|
||||
+
|
||||
+ (void)a; /* UNUSED */
|
||||
+ (void)tar; /* UNUSED */
|
||||
+ (void)entry; /* UNUSED */
|
||||
|
||||
- /* Just skip this and read the next header. */
|
||||
- return (tar_read_header(a, tar, entry, unconsumed));
|
||||
+ header = (const struct archive_entry_header_ustar *)h;
|
||||
+ size = tar_atol(header->size, sizeof(header->size));
|
||||
+ to_consume = ((size + 511) & ~511);
|
||||
+ *unconsumed += to_consume;
|
||||
+ return (ARCHIVE_OK);
|
||||
}
|
||||
</code></pre>
|
||||
<p>So thanks to the above change we can expect a release of libarchive supporting further flavors of abuse of GNU volume headers!
|
||||
🥳</p>
|
||||
<section role="doc-endnotes"><ol>
|
||||
<li id="fn-tartition">
|
||||
<p>Emily came up with the much better term "tartition table" than what I had come up with - "GPTar".</p>
|
||||
<span><a href="#ref-1-fn-tartition" role="doc-backlink" class="fn-label">↩︎︎</a></span></li></ol></section>
|
||||
|
||||
</article>
|
||||
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
188
tags.html
Normal file
188
tags.html
Normal file
|
@ -0,0 +1,188 @@
|
|||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>
|
||||
Robur's blog - Tags
|
||||
</title>
|
||||
<meta name="description" content="">
|
||||
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||
<script src="/js/hl.js"></script>
|
||||
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>blog.robur.coop</h1>
|
||||
<blockquote>
|
||||
The <strong>Robur</strong> cooperative blog.
|
||||
</blockquote>
|
||||
</header>
|
||||
<main><a href="/index.html">Back to index</a>
|
||||
|
||||
<ul class="tags-list aeration"><li><a href="#tag-Community">Community</a></li><li><a href="#tag-Cstruct">Cstruct</a></li><li><a href="#tag-DNSvizor">DNSvizor</a></li><li><a href="#tag-Git">Git</a></li><li><a href="#tag-MirageOS">MirageOS</a></li><li><a href="#tag-MirageVPN">MirageVPN</a></li><li><a href="#tag-OCaml">OCaml</a></li><li><a href="#tag-OpenVPN">OpenVPN</a></li><li><a href="#tag-Python">Python</a></li><li><a href="#tag-QubesOS">QubesOS</a></li><li><a href="#tag-SMTP">SMTP</a></li><li><a href="#tag-Scheduler">Scheduler</a></li><li><a href="#tag-Unikernel">Unikernel</a></li><li><a href="#tag-VPN">VPN</a></li><li><a href="#tag-cooperative">cooperative</a></li><li><a href="#tag-cryptography">cryptography</a></li><li><a href="#tag-emails">emails</a></li><li><a href="#tag-finances">finances</a></li><li><a href="#tag-functors">functors</a></li><li><a href="#tag-github">github</a></li><li><a href="#tag-gpt">gpt</a></li><li><a href="#tag-mailing-lists">mailing-lists</a></li><li><a href="#tag-mbr">mbr</a></li><li><a href="#tag-performance">performance</a></li><li><a href="#tag-persistent storage">persistent storage</a></li><li><a href="#tag-security">security</a></li><li><a href="#tag-tar">tar</a></li><li><a href="#tag-testing">testing</a></li><li><a href="#tag-unicode">unicode</a></li><li><a href="#tag-unikernel">unikernel</a></li><li><a href="#tag-vpn">vpn</a></li></ul><div class="tag-box" id="tag-Community">
|
||||
<h3>
|
||||
<span>Community</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/lwt_pause.html">Cooperation and Lwt.pause</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-Cstruct">
|
||||
<h3>
|
||||
<span>Cstruct</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/tar-release.html">The new Tar release, a retrospective</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-DNSvizor">
|
||||
<h3>
|
||||
<span>DNSvizor</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/dnsvizor01.html">Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-Git">
|
||||
<h3>
|
||||
<span>Git</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/lwt_pause.html">Cooperation and Lwt.pause</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-MirageOS">
|
||||
<h3>
|
||||
<span>MirageOS</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/miragevpn.html">MirageVPN & tls-crypt-v2</a></li><li><a href="/articles/miragevpn-ncp.html">MirageVPN updated (AEAD, NCP)</a></li><li><a href="/articles/speeding-ec-string.html">Speeding elliptic curve cryptography</a></li><li><a href="/articles/miragevpn-performance.html">Speeding up MirageVPN and use it in the wild</a></li><li><a href="/articles/miragevpn-server.html">MirageVPN server</a></li><li><a href="/articles/miragevpn-testing.html">Testing MirageVPN against OpenVPN™</a></li><li><a href="/articles/arguments.html">Runtime arguments in MirageOS</a></li><li><a href="/articles/dnsvizor01.html">Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-MirageVPN">
|
||||
<h3>
|
||||
<span>MirageVPN</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-08-21-OpenVPN-and-MirageVPN.html">MirageVPN and OpenVPN</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-OCaml">
|
||||
<h3>
|
||||
<span>OCaml</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/miragevpn.html">MirageVPN & tls-crypt-v2</a></li><li><a href="/articles/miragevpn-ncp.html">MirageVPN updated (AEAD, NCP)</a></li><li><a href="/articles/2024-02-03-python-str-repr.html">Python's `str.__repr__()`</a></li><li><a href="/articles/lwt_pause.html">Cooperation and Lwt.pause</a></li><li><a href="/articles/speeding-ec-string.html">Speeding elliptic curve cryptography</a></li><li><a href="/articles/gptar.html">GPTar</a></li><li><a href="/articles/miragevpn-performance.html">Speeding up MirageVPN and use it in the wild</a></li><li><a href="/articles/miragevpn-server.html">MirageVPN server</a></li><li><a href="/articles/qubes-miragevpn.html">qubes-miragevpn, a MirageVPN client for QubesOS</a></li><li><a href="/articles/miragevpn-testing.html">Testing MirageVPN against OpenVPN™</a></li><li><a href="/articles/tar-release.html">The new Tar release, a retrospective</a></li><li><a href="/articles/arguments.html">Runtime arguments in MirageOS</a></li><li><a href="/articles/dnsvizor01.html">Meet DNSvizor: run your own DHCP and DNS MirageOS unikernel</a></li><li><a href="/articles/gptar-update.html">GPTar (update)</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-OpenVPN">
|
||||
<h3>
|
||||
<span>OpenVPN</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-08-21-OpenVPN-and-MirageVPN.html">MirageVPN and OpenVPN</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-Python">
|
||||
<h3>
|
||||
<span>Python</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-02-03-python-str-repr.html">Python's `str.__repr__()`</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-QubesOS">
|
||||
<h3>
|
||||
<span>QubesOS</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/qubes-miragevpn.html">qubes-miragevpn, a MirageVPN client for QubesOS</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-SMTP">
|
||||
<h3>
|
||||
<span>SMTP</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-10-29-ptt.html">Postes, télégraphes et téléphones, next steps</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-Scheduler">
|
||||
<h3>
|
||||
<span>Scheduler</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/lwt_pause.html">Cooperation and Lwt.pause</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-Unikernel">
|
||||
<h3>
|
||||
<span>Unikernel</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/lwt_pause.html">Cooperation and Lwt.pause</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-VPN">
|
||||
<h3>
|
||||
<span>VPN</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/miragevpn.html">MirageVPN & tls-crypt-v2</a></li><li><a href="/articles/miragevpn-ncp.html">MirageVPN updated (AEAD, NCP)</a></li><li><a href="/articles/miragevpn-performance.html">Speeding up MirageVPN and use it in the wild</a></li><li><a href="/articles/miragevpn-server.html">MirageVPN server</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-cooperative">
|
||||
<h3>
|
||||
<span>cooperative</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/finances.html">How has robur financially been doing since 2018?</a></li><li><a href="/articles/2024-12-04-github-sponsor.html">Sponsor us via GitHub</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-cryptography">
|
||||
<h3>
|
||||
<span>cryptography</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/speeding-ec-string.html">Speeding elliptic curve cryptography</a></li><li><a href="/articles/miragevpn-performance.html">Speeding up MirageVPN and use it in the wild</a></li><li><a href="/articles/miragevpn-server.html">MirageVPN server</a></li><li><a href="/articles/miragevpn-testing.html">Testing MirageVPN against OpenVPN™</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-emails">
|
||||
<h3>
|
||||
<span>emails</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-10-29-ptt.html">Postes, télégraphes et téléphones, next steps</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-finances">
|
||||
<h3>
|
||||
<span>finances</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/finances.html">How has robur financially been doing since 2018?</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-functors">
|
||||
<h3>
|
||||
<span>functors</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/tar-release.html">The new Tar release, a retrospective</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-github">
|
||||
<h3>
|
||||
<span>github</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-12-04-github-sponsor.html">Sponsor us via GitHub</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-gpt">
|
||||
<h3>
|
||||
<span>gpt</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/gptar.html">GPTar</a></li><li><a href="/articles/gptar-update.html">GPTar (update)</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-mailing-lists">
|
||||
<h3>
|
||||
<span>mailing-lists</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-10-29-ptt.html">Postes, télégraphes et téléphones, next steps</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-mbr">
|
||||
<h3>
|
||||
<span>mbr</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/gptar.html">GPTar</a></li><li><a href="/articles/gptar-update.html">GPTar (update)</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-performance">
|
||||
<h3>
|
||||
<span>performance</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/miragevpn-performance.html">Speeding up MirageVPN and use it in the wild</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-persistent storage">
|
||||
<h3>
|
||||
<span>persistent storage</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/gptar.html">GPTar</a></li><li><a href="/articles/gptar-update.html">GPTar (update)</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-security">
|
||||
<h3>
|
||||
<span>security</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/miragevpn.html">MirageVPN & tls-crypt-v2</a></li><li><a href="/articles/miragevpn-ncp.html">MirageVPN updated (AEAD, NCP)</a></li><li><a href="/articles/speeding-ec-string.html">Speeding elliptic curve cryptography</a></li><li><a href="/articles/miragevpn-performance.html">Speeding up MirageVPN and use it in the wild</a></li><li><a href="/articles/miragevpn-server.html">MirageVPN server</a></li><li><a href="/articles/miragevpn-testing.html">Testing MirageVPN against OpenVPN™</a></li><li><a href="/articles/2024-08-21-OpenVPN-and-MirageVPN.html">MirageVPN and OpenVPN</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-tar">
|
||||
<h3>
|
||||
<span>tar</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/gptar.html">GPTar</a></li><li><a href="/articles/gptar-update.html">GPTar (update)</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-testing">
|
||||
<h3>
|
||||
<span>testing</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/miragevpn-testing.html">Testing MirageVPN against OpenVPN™</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-unicode">
|
||||
<h3>
|
||||
<span>unicode</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/2024-02-03-python-str-repr.html">Python's `str.__repr__()`</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-unikernel">
|
||||
<h3>
|
||||
<span>unikernel</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/qubes-miragevpn.html">qubes-miragevpn, a MirageVPN client for QubesOS</a></li></ul>
|
||||
</div><div class="tag-box" id="tag-vpn">
|
||||
<h3>
|
||||
<span>vpn</span>
|
||||
</h3>
|
||||
<ul><li><a href="/articles/qubes-miragevpn.html">qubes-miragevpn, a MirageVPN client for QubesOS</a></li><li><a href="/articles/miragevpn-testing.html">Testing MirageVPN against OpenVPN™</a></li></ul>
|
||||
</div>
|
||||
</main>
|
||||
<footer>
|
||||
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||
<br />
|
||||
</footer>
|
||||
<script>hljs.highlightAll();</script>
|
||||
</body>
|
||||
</html>
|
Loading…
Reference in a new issue