forked from robur/blog.robur.coop
Pushed by YOCaml 2 from ec0dec16ef37517b8e979c093d7f2edeeed07482-dirty
This commit is contained in:
parent
a46df08b2c
commit
303bab8b63
4 changed files with 445 additions and 0 deletions
206
articles/2024-10-29-ptt.html
Normal file
206
articles/2024-10-29-ptt.html
Normal file
|
@ -0,0 +1,206 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>
|
||||||
|
Robur's blog - Postes, télégraphes et téléphones, next steps
|
||||||
|
</title>
|
||||||
|
<meta name="description" content="An update of our email stack">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||||
|
<script src="/js/hl.js"></script>
|
||||||
|
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<h1>blog.robur.coop</h1>
|
||||||
|
<blockquote>
|
||||||
|
The <strong>Robur</strong> cooperative blog.
|
||||||
|
</blockquote>
|
||||||
|
</header>
|
||||||
|
<main><a href="/index.html">Back to index</a>
|
||||||
|
|
||||||
|
<article>
|
||||||
|
<h1>Postes, télégraphes et téléphones, next steps</h1>
|
||||||
|
<ul class="tags-list"><li><a href="/tags.html#tag-SMTP">SMTP</a></li><li><a href="/tags.html#tag-emails">emails</a></li><li><a href="/tags.html#tag-mailing-lists">mailing-lists</a></li></ul><p>As you know from <a href="https://blog.robur.coop/articles/finances.html">our article on Robur's
|
||||||
|
finances</a>, we've just received
|
||||||
|
<a href="https://nlnet.nl/project/PTT">funding for our email project</a>. This project
|
||||||
|
started when I was doing my internship in Cambridge and it's great to see that
|
||||||
|
it's been able to evolve over time and remain functional. This article will
|
||||||
|
introduce you to the latest changes to <a href="https://github.com/mirage/ptt">our PTT
|
||||||
|
project</a> and how far we've got towards providing
|
||||||
|
an OCaml mailing list service.</p>
|
||||||
|
<h2 id="a-git-repository-or-a-simple-block-device-as-a-database"><a class="anchor" aria-hidden="true" href="#a-git-repository-or-a-simple-block-device-as-a-database"></a>A Git repository or a simple block device as a database?</h2>
|
||||||
|
<p>One issue that came up quickly in our latest experiments with our SMTP stack was
|
||||||
|
the database of users with an email address. Since we had decided to ‘break
|
||||||
|
down’ the various stages of an email submission to offer simple unikernels, we
|
||||||
|
ended up having to deploy 4 unikernels to have a service that worked.</p>
|
||||||
|
<ul>
|
||||||
|
<li>a unikernel for authentication</li>
|
||||||
|
<li>a unikernel DKIM-signing the incoming email</li>
|
||||||
|
<li>one unikernel as primary DNS server</li>
|
||||||
|
<li>one unikernel sending the signed email to its real destination</li>
|
||||||
|
</ul>
|
||||||
|
<p>And we're only talking here about the submission of an email, the reception
|
||||||
|
concerns another ‘pipe’.</p>
|
||||||
|
<p>The problem with such an architecture is that some unikernels need to have the
|
||||||
|
same data: the users. In this case, the first unikernel needs to know the user's
|
||||||
|
password in order to verify authentication. The final unikernel needs to know
|
||||||
|
the real destinations of the users.</p>
|
||||||
|
<p>Let's take the example of two users: foo@robur.coop and bar@robur.coop. The
|
||||||
|
first points to hannes@foo.org and the second to reynir@example.com.</p>
|
||||||
|
<p>If Hannes wants to send a message to bar@robur.coop under the identity of
|
||||||
|
foo@robur.coop, he will need to authenticate himself to our first unikernel.
|
||||||
|
This first unikernel must therefore:</p>
|
||||||
|
<ol>
|
||||||
|
<li>check that the user <code>foo</code> exists</li>
|
||||||
|
<li>the hashed password used by Hannes is the same as the one in the database</li>
|
||||||
|
</ol>
|
||||||
|
<p>Next, the email will be signed by our second unikernel. It will then forward the
|
||||||
|
email to the last unikernel, which will do the actual translation of the
|
||||||
|
recipients and DNS resolution. In other words:</p>
|
||||||
|
<ol>
|
||||||
|
<li>it will see that one (the only) recipient is bar@robur.coop</li>
|
||||||
|
<li>check that bar@robur.coop exists and obtain its real address</li>
|
||||||
|
<li>it will obtain reynir@example.com and perform DNS resolution on
|
||||||
|
<code>example.com</code> to find out the email server for this domain</li>
|
||||||
|
<li>finally send the email signed by foo@robur.coop to reynir@example.com!</li>
|
||||||
|
</ol>
|
||||||
|
<p>So the first and last unikernels need to have the same information about our
|
||||||
|
users. One for the passwords, the second for the real email addresses.</p>
|
||||||
|
<p>But as you know, we're talking about unikernels that exist independently of each
|
||||||
|
other. What's more, they can't share files and the possibility of them sharing
|
||||||
|
block-devices remains an open question (and a complex one where parallel access
|
||||||
|
may be involved). In short, the only way to ‘synchronise’ these unikernels in
|
||||||
|
relation to common data is with a Git repository.</p>
|
||||||
|
<p><a href="https://github.com/robur-coop/git-kv">Git</a> has the advantage of being widely used for our unikernels
|
||||||
|
(<a href="https://github.com/robur-coop/dns-primary-git/">primary-git</a>, <a href="https://github.com/dinosaure/pasteur">pasteur</a>, <a href="https://github.com/robur-coop/unipi">unipi</a> and
|
||||||
|
<a href="https://github.com/dinosaure/contruno">contruno</a>). The advantage is that you can track changes, modify
|
||||||
|
files and notify the unikernel to update itself (using nsupdate, a simple ping
|
||||||
|
or an http request to the unikernel).</p>
|
||||||
|
<p>The problem is that this requires certain skills. Even if it's ‘simple’ to set
|
||||||
|
up a Git server and then deploy our unikernels, we can restructure our
|
||||||
|
architecture and simplify the deployment of an SMTP stack!</p>
|
||||||
|
<h2 id="elit-and-oneffs"><a class="anchor" aria-hidden="true" href="#elit-and-oneffs"></a>Elit and OneFFS</h2>
|
||||||
|
<p>We have therefore decided to merge the email exchange service and email
|
||||||
|
submission into a unikernel so that this is the only user information requester.</p>
|
||||||
|
<p>So we decided to use <a href="https://github.com/robur-coop/oneffs">OneFFS</a> as the file system for our database,
|
||||||
|
which will be a plain JSON file. This is perhaps one of the advantages of
|
||||||
|
MirageOS, which is that you can decide exactly what you need to implement
|
||||||
|
specific objectives.</p>
|
||||||
|
<p>In this case, those with experience of Postfix, LDAP or MariaDB could confirm
|
||||||
|
that configuring an email service should be ‘simpler’ than implementing a
|
||||||
|
multitude of pipes between different applications and authentication methods.</p>
|
||||||
|
<p>The JSON file is therefore very simple and so is the creation of an OneFFS
|
||||||
|
image:</p>
|
||||||
|
<pre><code class="language-sh">$ cat >database.json<<EOF
|
||||||
|
> [ { "name": "din"
|
||||||
|
> , "password": "xxxxxx"
|
||||||
|
> , "mailboxes": [ "romain.calascibetta@gmail.com" ] } ]
|
||||||
|
> EOF
|
||||||
|
$ opam install oneffs
|
||||||
|
$ oneffs create -i database.json -o database.img
|
||||||
|
</code></pre>
|
||||||
|
<p>All you have to do is register this image as a block with <a href="https://github.com/robur-coop/albatross">albatross</a> and launch
|
||||||
|
our Elit unikernel with this block-device.</p>
|
||||||
|
<pre><code class="language-sh">$ albatross-client create-block --data=database.img database 1024
|
||||||
|
$ albatross-client create --net=service:br0 --block=database:database \
|
||||||
|
elit elit.hvt \
|
||||||
|
--arg=...
|
||||||
|
</code></pre>
|
||||||
|
<p>At this stage, and if we add our unikernel signing incoming emails, we have more
|
||||||
|
or less the same thing as what I've described in <a href="https://blog.osau.re/articles/smtp_1.html">my previous articles</a> on
|
||||||
|
<a href="https://blog.osau.re/articles/smtp_2.html">deploying</a> an <a href="https://blog.osau.re/articles/smtp_3.html">email service</a>.</p>
|
||||||
|
<h2 id="multiplex-receiving--sending-emails"><a class="anchor" aria-hidden="true" href="#multiplex-receiving--sending-emails"></a>Multiplex receiving & sending emails</h2>
|
||||||
|
<p>The PTT project is a toolkit for implementing SMTP servers. It gives developers
|
||||||
|
the choice of implementing their logic as they see fit:</p>
|
||||||
|
<ul>
|
||||||
|
<li>sign an email</li>
|
||||||
|
<li>resolve destinations according to a database</li>
|
||||||
|
<li>check SPF information</li>
|
||||||
|
<li>annotate the email as spam or not</li>
|
||||||
|
<li>etc.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Previously, PTT was split into 2 parts:</p>
|
||||||
|
<ol>
|
||||||
|
<li>management of incoming clients/emails</li>
|
||||||
|
<li>the logic to be applied to incoming emails and their delivery</li>
|
||||||
|
</ol>
|
||||||
|
<p>The second point was becoming increasingly complex, however, and errors in
|
||||||
|
sending emails are legion (DMARC non-alignment, the email is too big for the
|
||||||
|
destination, the destination doesn't exist, etc.). All the more so since, up to
|
||||||
|
now, PTT could only report these errors via the logs...</p>
|
||||||
|
<p>Hannes immediately mentioned the possibility of separating the logic of the
|
||||||
|
unikernel from the delivery. This will allow us to deal with temporary failures
|
||||||
|
(greylisting) as well. So a fundamental change was made:</p>
|
||||||
|
<ul>
|
||||||
|
<li>improve the <a href="https://github.com/mirage/colombe">sendmail</a> and <code>sendmail-lwt</code> packages (as well as proposing
|
||||||
|
<code>sendmail-miou</code>!) when sending or submitting an email</li>
|
||||||
|
<li>improve PTT so that there are now 3 distinct jobs: receiving, what to do with
|
||||||
|
incoming emails and sending emails</li>
|
||||||
|
</ul>
|
||||||
|
<p><img src="../images/smtp.jpg" alt="SMTP" ></p>
|
||||||
|
<p>This finally allows us to describe a clearer error management policy that is
|
||||||
|
independent of what we want to do with incoming emails. At this stage, we can
|
||||||
|
look for the <code>Return-Path</code> in emails that we haven't managed to send and notify
|
||||||
|
the senders!</p>
|
||||||
|
<p>All this is still in the experimental stage and practical cases are needed to
|
||||||
|
observe how we should handle errors and how others do.</p>
|
||||||
|
<h2 id="insights--next-goals"><a class="anchor" aria-hidden="true" href="#insights--next-goals"></a>Insights & Next goals</h2>
|
||||||
|
<p>We're already starting to have a bit of fun with email and we can start sending
|
||||||
|
and receiving emails right away.</p>
|
||||||
|
<p>We're also already seeing hacking attempts on our unikernel:</p>
|
||||||
|
<ul>
|
||||||
|
<li>people trying to authenticate themselves without <code>STARTTLS</code> (or with it,
|
||||||
|
depending on how clever the bot is)</li>
|
||||||
|
<li>people trying to send emails as non-existent users in our database</li>
|
||||||
|
<li>we're also seeing content that has nothing to do with SMTP</li>
|
||||||
|
</ul>
|
||||||
|
<p>Above all, this shows that, very early on, bots try to usurp the identity linked
|
||||||
|
to your server (in our case, osau.re) in order to send spam, authenticate
|
||||||
|
themselves or simply send ‘stuff’ and observe what happens. In this case, for
|
||||||
|
all the cases mentioned, Elit (and PTT) reacts well: in other words, it simply
|
||||||
|
cuts off the connection.</p>
|
||||||
|
<p>We were also able to observe how services such as gmail work. In addition, for
|
||||||
|
the purposes of a mailing list, email forwarding distorts DMARC verification
|
||||||
|
(specifically, SPF verification). The case is very simple:</p>
|
||||||
|
<p>foo@gmail.com tries to reply to robur@osau.re. robur@osau.re is a mailing list
|
||||||
|
to several addresses (one of them is bar@gmail.com). The unikernel will receive
|
||||||
|
the email and send it to bar@gmail.com. The problem is the alignment between
|
||||||
|
the <code>From</code> field (which corresponds to foo@gmail.com) and our osau.re server.
|
||||||
|
From gmail.com's point of view, there is a misalignment between these two
|
||||||
|
pieces of information and it therefore refuses to receive the email.</p>
|
||||||
|
<p>This is where our next objectives come in:</p>
|
||||||
|
<ul>
|
||||||
|
<li>finish our DMARC implementation</li>
|
||||||
|
<li>implement ARC so that our server notifies us that, on our side, the DMARC
|
||||||
|
check went well and that gmail.com should trust us on this.</li>
|
||||||
|
</ul>
|
||||||
|
<p>There is another way of solving the problem, perhaps a little more problematic,
|
||||||
|
modify the incoming email and in particular the <code>From</code> field. Although this
|
||||||
|
could be done quite simply with <a href="https://github.com/mirage/mrmime">mrmime</a>, it's better to concentrate on
|
||||||
|
DMARC and ARC so that we can send our emails as they are and never alter them
|
||||||
|
(especially as this will invalidate previous DKIM signatures!).</p>
|
||||||
|
<h2 id="conclusion"><a class="anchor" aria-hidden="true" href="#conclusion"></a>Conclusion</h2>
|
||||||
|
<p>It's always satisfying to see your projects working ‘more or less’ correctly.
|
||||||
|
This article will surely be the start of a series on the intricacies of email
|
||||||
|
and the difficulty of deploying such a service at home.</p>
|
||||||
|
<p>We hope that this NLnet-funded work will enable us to replace our current email
|
||||||
|
system with unikernels. We're already past the stage where we can, more or less
|
||||||
|
(without DMARC checking), send emails to each other, which is a big step!</p>
|
||||||
|
<p>So follow our work on our blog and if you like what we're producing (which
|
||||||
|
involves a whole bunch of protocols and formats - much more than just SMTP), you
|
||||||
|
can make <a href="https://robur.coop/Donate">a donation here</a>!</p>
|
||||||
|
|
||||||
|
</article>
|
||||||
|
|
||||||
|
</main>
|
||||||
|
<footer>
|
||||||
|
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||||
|
<br />
|
||||||
|
</footer>
|
||||||
|
<script>hljs.highlightAll();</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
62
articles/2024-12-04-github-sponsor.html
Normal file
62
articles/2024-12-04-github-sponsor.html
Normal file
|
@ -0,0 +1,62 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>
|
||||||
|
Robur's blog - Sponsor us via GitHub
|
||||||
|
</title>
|
||||||
|
<meta name="description" content="A new way to sponsor our cooperative">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||||
|
<script src="/js/hl.js"></script>
|
||||||
|
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<h1>blog.robur.coop</h1>
|
||||||
|
<blockquote>
|
||||||
|
The <strong>Robur</strong> cooperative blog.
|
||||||
|
</blockquote>
|
||||||
|
</header>
|
||||||
|
<main><a href="/index.html">Back to index</a>
|
||||||
|
|
||||||
|
<article>
|
||||||
|
<h1>Sponsor us via GitHub</h1>
|
||||||
|
<ul class="tags-list"><li><a href="/tags.html#tag-cooperative">cooperative</a></li><li><a href="/tags.html#tag-github">github</a></li></ul><p>We're delighted to announce the possibility of helping our cooperative through
|
||||||
|
the GitHub Sponsors platform. The link is available here:</p>
|
||||||
|
<p><a href="https://github.com/sponsors/robur-coop">https://github.com/sponsors/robur-coop</a></p>
|
||||||
|
<p>We would also like to reiterate the possibility of making a donation<sup><a href="#fn-preferable" id="ref-1-fn-preferable" role="doc-noteref" class="fn-label">[1]</a></sup> to our
|
||||||
|
cooperative via the IBAN of <a href="https://aenderwerk.de/">Änderwerk</a> available here (if you need
|
||||||
|
a tax-deductible donation receipt, please use <a href="https://aenderwerk.de/donate">this form</a>).</p>
|
||||||
|
<pre><code>Account holder: Änderwerk gGmbH
|
||||||
|
Subject: robur
|
||||||
|
IBAN: DE46 4306 0967 1289 8604 00
|
||||||
|
BIC: GENODEM1GLS
|
||||||
|
Bank: GLS Gemeinschaftsbank, Christstrasse 9, 44789 Bochum, Germany
|
||||||
|
</code></pre>
|
||||||
|
<p>More generally, you can refer to our <a href="https://blog.robur.coop/articles/finances.html">article</a> which explains our
|
||||||
|
funding since the creation of Robur and we would like to point out that,
|
||||||
|
despite our funding, part of our work remains unfunded: in particular with
|
||||||
|
regard to the maintenance of certain software as well as certain services made
|
||||||
|
available to our users.</p>
|
||||||
|
<p>We would therefore be delighted if users of our software and services could
|
||||||
|
finance our work according to their means. GitHub in particular offers an
|
||||||
|
easy-to-use platform for funding us (even if, in all transparency, it takes a
|
||||||
|
certain amount from each transaction).</p>
|
||||||
|
<section role="doc-endnotes"><ol>
|
||||||
|
<li id="fn-preferable">
|
||||||
|
<p>In fact, this method is preferable to us as this means it will go directly to us instead of through GitHub and Stripe who will take a small cut of the donation in fees.</p>
|
||||||
|
<span><a href="#ref-1-fn-preferable" role="doc-backlink" class="fn-label">↩︎︎</a></span></li></ol></section>
|
||||||
|
|
||||||
|
</article>
|
||||||
|
|
||||||
|
</main>
|
||||||
|
<footer>
|
||||||
|
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||||
|
<br />
|
||||||
|
</footer>
|
||||||
|
<script>hljs.highlightAll();</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
110
articles/gptar-update.html
Normal file
110
articles/gptar-update.html
Normal file
|
@ -0,0 +1,110 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>
|
||||||
|
Robur's blog - GPTar (update)
|
||||||
|
</title>
|
||||||
|
<meta name="description" content="libarchive vs hybrid GUID partition table and GNU tar volume header">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||||
|
<script src="/js/hl.js"></script>
|
||||||
|
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<h1>blog.robur.coop</h1>
|
||||||
|
<blockquote>
|
||||||
|
The <strong>Robur</strong> cooperative blog.
|
||||||
|
</blockquote>
|
||||||
|
</header>
|
||||||
|
<main><a href="/index.html">Back to index</a>
|
||||||
|
|
||||||
|
<article>
|
||||||
|
<h1>GPTar (update)</h1>
|
||||||
|
<ul class="tags-list"><li><a href="/tags.html#tag-OCaml">OCaml</a></li><li><a href="/tags.html#tag-gpt">gpt</a></li><li><a href="/tags.html#tag-tar">tar</a></li><li><a href="/tags.html#tag-mbr">mbr</a></li><li><a href="/tags.html#tag-persistent storage">persistent storage</a></li></ul><p>In a <a href="gptar.html">previous post</a> I describe how I craft a hybrid GUID partition table (GPT) and tar archive by exploiting that there are disjoint areas of a 512 byte <em>block</em> that are important to tar headers and <em>protective</em> master boot records used in GPT respectively.
|
||||||
|
I recommend reading it first if you haven't already for context.</p>
|
||||||
|
<p>After writing the above post I read an excellent and fun <em>and totally normal</em> article by Emily on how <a href="https://uni.horse/executable-tarballs.html">she created <strong>executable</strong> tar archives</a>.
|
||||||
|
Therein I learned a clever hack:
|
||||||
|
GNU tar has a tar extension for <em>volume headers</em>.
|
||||||
|
These are essentially labels for your tape archives when you're forced to split an archive across multiple tapes.
|
||||||
|
They can (seemingly) hold any text as label including shell scripts.
|
||||||
|
What's more is GNU tar and bsdtar <strong>does not</strong> extract these as files!
|
||||||
|
This is excellent, because I don't actually want to extract or list the GPT header when using GNU tar or bsdtar.
|
||||||
|
This prompted me to <a href="https://github.com/reynir/gptar/pull/1">use a different link indicator</a>.</p>
|
||||||
|
<p>This worked pretty great.
|
||||||
|
Listing the archive using GNU tar I still get <code>GPTAR</code>, but with verbose listing it's displayed as a <code>--Volume Header--</code>:</p>
|
||||||
|
<pre><code class="language-shell">$ tar -tvf disk.img
|
||||||
|
Vr-------- 0/0 16896 1970-01-01 01:00 GPTAR--Volume Header--
|
||||||
|
-rw-r--r-- 0/0 14 1970-01-01 01:00 test.txt
|
||||||
|
</code></pre>
|
||||||
|
<p>And more importantly the <code>GPTAR</code> entry is ignored when extracting:</p>
|
||||||
|
<pre><code class="language-shell">$ mkdir tmp
|
||||||
|
$ cd tmp/
|
||||||
|
$ tar -xf ../disk.img
|
||||||
|
$ ls
|
||||||
|
test.txt
|
||||||
|
</code></pre>
|
||||||
|
<h2 id="bsd-tar--libarchive"><a class="anchor" aria-hidden="true" href="#bsd-tar--libarchive"></a>BSD tar / libarchive</h2>
|
||||||
|
<p>Unfortunately, this broke bsdtar!</p>
|
||||||
|
<pre><code class="language-shell">$ bsdtar -tf disk.img
|
||||||
|
bsdtar: Damaged tar archive
|
||||||
|
bsdtar: Error exit delayed from previous errors.
|
||||||
|
</code></pre>
|
||||||
|
<p>This is annoying because we run FreeBSD on the host for <a href="https://opam.robur.coop">opam.robur.coop</a>, our instance of <a href="https://git.robur.coop/robur/opam-mirror/">opam-mirror</a>.
|
||||||
|
This Autumn we updated <a href="https://git.robur.coop/robur/opam-mirror/">opam-mirror</a> to use the hybrid GPT+tar GPTar <em>tartition table</em><sup><a href="#fn-tartition" id="ref-1-fn-tartition" role="doc-noteref" class="fn-label">[1]</a></sup> instead of hard coded or boot parameter specified disk offsets for the different partitions - which was extremely brittle!
|
||||||
|
So we were no longer able to inspect the contents of the tar partition from the host!
|
||||||
|
Unacceptable!
|
||||||
|
So I started to dig into libarchive where bsdtar comes from.
|
||||||
|
To my surprise, after building bsdtar from the git clone of the source code it ran perfectly fine!</p>
|
||||||
|
<pre><code class="language-shell">$ ./bsdtar -tf ../gptar/disk.img
|
||||||
|
test.txt
|
||||||
|
</code></pre>
|
||||||
|
<p>I eventually figure out <a href="https://github.com/libarchive/libarchive/pull/2127">this change</a> fixed it for me.
|
||||||
|
I got in touch with Emily to let her know that bsdtar recently fixed this (ab)use of GNU volume headers.
|
||||||
|
Her reply was basically "as of when I wrote the article, I was pretty sure bsdtar ignored it."
|
||||||
|
And indeed it did.
|
||||||
|
Examining the diff further revealed that it ignored the GNU volume header - just not "correctly" when the GNU volume header was abused to carry file content as I did:</p>
|
||||||
|
<pre><code class="language-diff"> /*
|
||||||
|
* Interpret 'V' GNU tar volume header.
|
||||||
|
*/
|
||||||
|
static int
|
||||||
|
header_volume(struct archive_read *a, struct tar *tar,
|
||||||
|
struct archive_entry *entry, const void *h, size_t *unconsumed)
|
||||||
|
{
|
||||||
|
- (void)h;
|
||||||
|
+ const struct archive_entry_header_ustar *header;
|
||||||
|
+ int64_t size, to_consume;
|
||||||
|
+
|
||||||
|
+ (void)a; /* UNUSED */
|
||||||
|
+ (void)tar; /* UNUSED */
|
||||||
|
+ (void)entry; /* UNUSED */
|
||||||
|
|
||||||
|
- /* Just skip this and read the next header. */
|
||||||
|
- return (tar_read_header(a, tar, entry, unconsumed));
|
||||||
|
+ header = (const struct archive_entry_header_ustar *)h;
|
||||||
|
+ size = tar_atol(header->size, sizeof(header->size));
|
||||||
|
+ to_consume = ((size + 511) & ~511);
|
||||||
|
+ *unconsumed += to_consume;
|
||||||
|
+ return (ARCHIVE_OK);
|
||||||
|
}
|
||||||
|
</code></pre>
|
||||||
|
<p>So thanks to the above change we can expect a release of libarchive supporting further flavors of abuse of GNU volume headers!
|
||||||
|
🥳</p>
|
||||||
|
<section role="doc-endnotes"><ol>
|
||||||
|
<li id="fn-tartition">
|
||||||
|
<p>Emily came up with the much better term "tartition table" than what I had come up with - "GPTar".</p>
|
||||||
|
<span><a href="#ref-1-fn-tartition" role="doc-backlink" class="fn-label">↩︎︎</a></span></li></ol></section>
|
||||||
|
|
||||||
|
</article>
|
||||||
|
|
||||||
|
</main>
|
||||||
|
<footer>
|
||||||
|
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||||
|
<br />
|
||||||
|
</footer>
|
||||||
|
<script>hljs.highlightAll();</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
67
articles/miragevpn-testing.html
Normal file
67
articles/miragevpn-testing.html
Normal file
|
@ -0,0 +1,67 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>
|
||||||
|
Robur's blog - Testing MirageVPN against OpenVPN™
|
||||||
|
</title>
|
||||||
|
<meta name="description" content="Some notes about how we test MirageVPN against OpenVPN™">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/hl.css">
|
||||||
|
<link type="text/css" rel="stylesheet" href="/css/style.css">
|
||||||
|
<script src="/js/hl.js"></script>
|
||||||
|
<link rel="alternate" type="application/rss+xml" href="/feed.xml" title="blog.robur.coop">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<h1>blog.robur.coop</h1>
|
||||||
|
<blockquote>
|
||||||
|
The <strong>Robur</strong> cooperative blog.
|
||||||
|
</blockquote>
|
||||||
|
</header>
|
||||||
|
<main><a href="/index.html">Back to index</a>
|
||||||
|
|
||||||
|
<article>
|
||||||
|
<h1>Testing MirageVPN against OpenVPN™</h1>
|
||||||
|
<ul class="tags-list"><li><a href="/tags.html#tag-OCaml">OCaml</a></li><li><a href="/tags.html#tag-MirageOS">MirageOS</a></li><li><a href="/tags.html#tag-cryptography">cryptography</a></li><li><a href="/tags.html#tag-security">security</a></li><li><a href="/tags.html#tag-testing">testing</a></li><li><a href="/tags.html#tag-vpn">vpn</a></li></ul><p>As our last milestone for the <a href="https://www.assure.ngi.eu/">EU NGI Assure</a> funded MirageVPN project (for now) we have been working on testing MirageVPN, our OpenVPN™-compatible VPN implementation against the upstream OpenVPN™.
|
||||||
|
During the development we have conducted many manual tests.
|
||||||
|
However, this scales poorly and it is easy to forget testing certain cases.
|
||||||
|
Therefore, we designed and implemented interoperability testing, driving the C implementation on the one side, and our OCaml implementation on the other side. The input for such a test is a configuration file that both implementations can use.
|
||||||
|
Thus we test establishment of the tunnel as well as the tunnel itself.</p>
|
||||||
|
<p>While conducting the tests, our instrumented binaries expose code coverage information. We use that to guide ourselves which other configurations are worth testing. Our goal is to achieve a high code coverage rate while using a small amount of different configurations. These interoperability tests are running fast enough, so they are executed on each commit by CI.</p>
|
||||||
|
<p>A nice property of this test setup is that it runs with an unmodified OpenVPN binary.
|
||||||
|
This means we can use an off-the-shelf OpenVPN binary from the package repository and does not entail further maintenance of an OpenVPN fork.
|
||||||
|
Testing against a future version of OpenVPN becomes trivial.
|
||||||
|
We do not just test a single part of our implementation but achieve an end-to-end test.
|
||||||
|
The same configuration files are used for both our implementation and the C implementation, and each configuration is used twice, once our implementation acts as the client, once as the server.</p>
|
||||||
|
<p>We added a flag to our client and our <a href="miragevpn-server">recently finished server</a> applications, <code>--test</code>, which make them to exit once a tunnel is established and an ICMP echo request from the client has been replied to by the server.
|
||||||
|
Our client and server can be run without a tun device which otherwise would require elevated privileges.
|
||||||
|
Unfortunately, OpenVPN requires privileges to at least configure a tun device.
|
||||||
|
Our MirageVPN implementation does IP packet parsing in userspace.
|
||||||
|
We test our protocol implementation, not the entire unikernel - but the unikernel code is a tiny layer on top of the purely functional protocol implementation.</p>
|
||||||
|
<p>We explored unit testing the packet decoding and decryption with our implementation and the C implementation.
|
||||||
|
Specifically, we encountered a packet whose message authentication code (MAC) was deemed invalid by the C implementation.
|
||||||
|
It helped us discover the MAC computation was correct but the packet encoding was truncated - both implementations agreed that the MAC was bad.
|
||||||
|
The test was very tedious to write and would not easily scale to cover a large portion of the code.
|
||||||
|
If of interest, take a look into our <a href="https://github.com/reynir/openvpn/tree/badmac-test">modifications to OpenVPN</a> and <a href="https://github.com/robur-coop/miragevpn/tree/badmac-test">modifications to MirageVPN</a>.</p>
|
||||||
|
<p>The end-to-end testing is in addition to our unit tests and fuzz testing; and to our <a href="miragevpn-performance.html">benchmarking</a> binary.</p>
|
||||||
|
<p>Our results are that with 4 configurations we achieve above 75% code coverage in MirageVPN.
|
||||||
|
While investigating the code coverage results, we found various pieces of code that were never executed, and we were able to remove them.
|
||||||
|
Code that does not exist is bug-free :D
|
||||||
|
With these tests in place future maintenance is less daunting as they will help us guard us from breaking the code.</p>
|
||||||
|
<p>At the moment we do not exercise the error paths very well in the code.
|
||||||
|
This is much less straightforward to test in this manner, and is important future work.
|
||||||
|
We plan to develop a client and server that injects faults at various stages of the protocol to test these error paths.
|
||||||
|
OpenVPN built with debugging enabled also comes with a <code>--gremlin</code> mode that injects faults, and would be interesting to investigate.</p>
|
||||||
|
|
||||||
|
</article>
|
||||||
|
|
||||||
|
</main>
|
||||||
|
<footer>
|
||||||
|
<a href="https://github.com/xhtmlboi/yocaml">Powered by <strong>YOCaml</strong></a>
|
||||||
|
<br />
|
||||||
|
</footer>
|
||||||
|
<script>hljs.highlightAll();</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
Loading…
Reference in a new issue