Robur Reproducible Builds
Published: 2021-11-16 (last updated: 2023-02-09)In 2021 we in Robur have been working towards easing deployment of reproducible mirage applications. The work has been funded by the European Union under the Next Generation Internet (NGI Pointer) initiative. The result is online.
+Robur Reproducible Builds
Published: 2021-11-16 (last updated: 2023-05-16)In 2021 we in Robur have been working towards easing deployment of reproducible mirage applications. The work has been funded by the European Union under the Next Generation Internet (NGI Pointer) initiative. The result is online.
The overall goal is to push MirageOS into production in a trustworthy way. We worked on reproducible builds for Opam packages and MirageOS - with the infrastructure being reproducible itself. Reproducible builds are crucial for supply chain security - everyone can reproduce the exact same binary (by using the same sources and environment), without reproducible builds we would not publish binaries.
-Reproducible builds are also great for fleet management: by inspecting the hash of the binary that is executed, we can figure out which versions of which libraries are in the unikernel - and suggest updates if newer builds are available or if a used library has a security flaw -- albatross-client-local update my-unikernel
is everything needed for an update.
Reproducible builds are also great for fleet management: by inspecting the hash of the binary that is executed, we can figure out which versions of which libraries are in the unikernel - and suggest updates if newer builds are available or if a used library has a security flaw -- albatross-client update my-unikernel
is everything needed for an update.
Several ready-to-use MirageOS unikernels are built on a daily basis - ranging from authoritative DNS servers (secondary, let's encrypt DNS solver), DNS-and-DHCP service (similar to dnsmasq), TLS reverse proxy, Unipi - a web server that delivers content from a git repository, DNS resolver, CalDAV server, and of course your own MirageOS unikernel.
Brief robur and MirageOS introduction
MirageOS is an operating system, developed in OCaml, which produces unikernels. A unikernel serves a single purpose and is a single process, i.e. only has the really needed dependencies. For example, an OpenVPN endpoint does neither include persistent storage (block device, file system) nor user management. MirageOS unikernels are developed in OCaml, a statically typed and type-safe programming language - which avoids common pitfalls from the grounds up (spatial and temporal memory safety issues).
@@ -28,16 +28,17 @@ $ pkg update $ pkg install solo5 albatrossFor other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (opam install solo5 albatross
). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.
There is no configuration needed. Start the albatross_console
and the albatross_daemon
service (via systemctl daemon-reload ; systemctl start albatross_daemon
on Linux or service albatross_daemon start
on FreeBSD). Executing albatross-client-local info
should return success (exit code 0) and no running unikernel. You may need to be in the albatross group, or change the permissions of the Unix domain socket (/run/albatross/util/vmmd.sock
on Linux, /var/run/albatross/util/vmmd.sock
on FreeBSD).
There is no configuration needed. Start the albatross_console
and the albatross_daemon
service (via systemctl daemon-reload ; systemctl start albatross_daemon
on Linux or service albatross_daemon start
on FreeBSD). Executing albatross-client info
should return success (exit code 0) and no running unikernel. You may need to be in the albatross group, or change the permissions of the Unix domain socket (/run/albatross/util/vmmd.sock
on Linux, /var/run/albatross/util/vmmd.sock
on FreeBSD).
To check that albatross works, get the latest hello world unikernel and run it:
$ wget https://builds.robur.coop/job/hello/build/latest/bin/hello.hvt
-$ albatross-client-local console my-hello-unikernel & # this is sent to the background since it waits and displays the console of the unikernel named "my-hello-unikernel"
-$ albatross-client-local create my-hello-unikernel hello.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
-$ albatross-client-local create --arg='--hello="Hello,\ my\ unikernel" my-hello-unikernel hello.hvt # executes the same unikernel, but passes the boot parameter "--hello"
-$ fg # back to albatross-client-local console
+$ albatross-client console my-hello-unikernel & # this is sent to the background since it waits and displays the console of the unikernel named "my-hello-unikernel"
+$ albatross-client create my-hello-unikernel hello.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
+$ albatross-client create --arg='--hello="Hello,\ my\ unikernel" my-hello-unikernel hello.hvt # executes the same unikernel, but passes the boot parameter "--hello"
+$ fg # back to albatross-client console
$ Ctrl-C # kill that process
-Voila, we have a working albatross installation. Albatross also supports a remote client (using a TLS handshake) albatross-client-bistro
, monitoring of unikernels (albatross_stat
and albatross_influx
services), and a TLS endpoint (via inetd with albatross-tls-inetd
).
Voila, we have a working albatross installation. Albatross also supports a remote client (using a TLS handshake) albatross-client --ca <ca.pem> --ca-key <ca.key> --server-ca <cacert.pem> --destination <myhost>
, monitoring of unikernels (albatross_stats
and albatross_influx
services), and a TLS endpoint (inetd/systemd: albatross-tls-endpoint
).
Please ensure to have albatross in version of at least 2.0.0 to follow this page.
Network for your unikernel
Next we want to setup networking for our unikernels. We use a so-called "bridge" interface for this, which is a virtual network switch where you connect "tap" interfaces (layer 2 ethernet devices). A MirageOS unikernel uses tap interfaces for communication. We give our bridge the name "service" (and for example for monitoring and management you may want to setup another bridge "management").
If you're using a network manager that is capable of setting up bridge interfaces, use that interface.
@@ -62,15 +63,15 @@ ifconfig_service="inet 10.0.42.1/24"Afterwards either restart your system or re-run the service scripts to have the bridge setup in your running system.
To check that the networking works, get the latest static website unikernel and run it:
$ wget https://builds.robur.coop/job/static-website/build/latest/bin/https.hvt
-$ albatross-client-local console my-website & # this is sent to the background since it waits and displays the console of the unikernel named "my-website"
-$ albatross-client-local create --net=service --arg='--ipv4=10.0.42.2/24' my-website https.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
+$ albatross-client console my-website & # this is sent to the background since it waits and displays the console of the unikernel named "my-website"
+$ albatross-client create --net=service --arg='--ipv4=10.0.42.2/24' my-website https.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
$ ping 10.0.42.2 # should receive answers
$ open http://10.0.42.2 # in your browser - also https://10.0.42.2 (you'll get a certificate warning)
$ wget http://10.0.42.2/ # should download the Hello Mirage world!
$ wget --no-check-certificate https://10.0.42.2/ # should also download the Hello Mirage world!
-$ fg # back to albatross-client-local console
+$ fg # back to albatross-client console
$ Ctrl-C # kill that process
-$ albatross-client-local destroy my-website # kills the unikernel
+$ albatross-client destroy my-website # kills the unikernel
When you reached this point, you have successfully launched a MirageOS unikernel, and are able to communicate from your computer with it. This uses the OCaml networking stack, and the host bridge interface.
Routing and Internet
@@ -110,9 +111,9 @@ On FreeBSD, addifconfig_service="addm em0"
to /etc
Unikernel execution
Let's test that your unikernels have access to the Internet by using the traceroute unikernel:
$ wget https://builds.robur.coop/job/traceroute/build/latest/bin/traceroute.hvt
-$ albatross-client-local console my-traceroute & # this is sent to the background since it waits and displays the console of the unikernel named "my-traceroute"
-$ albatross-client-local create --net=service --arg='--ipv4=10.0.42.2/24' --arg='--ipv4-gateway=10.0.42.1' my-traceroute traceroute.hvt # the IP configuration depends on your setup, use your public IP address and actual router IP if you've set it up.
-$ fg # back to albatross-client-local console
+$ albatross-client console my-traceroute & # this is sent to the background since it waits and displays the console of the unikernel named "my-traceroute"
+$ albatross-client create --net=service --arg='--ipv4=10.0.42.2/24' --arg='--ipv4-gateway=10.0.42.1' my-traceroute traceroute.hvt # the IP configuration depends on your setup, use your public IP address and actual router IP if you've set it up.
+$ fg # back to albatross-client console
$ Ctrl-C # kill that process
That's it. Albatross has more features, such as block devices, multiple bridges (for management, private networks, ...), restart if the unikernel exited with specific exit code, assignment of a unikernel to a specific CPU. It also has remote command execution and resource limits (you can allow your friends to execute a number of unikernels with limited memory and block storage accessing only some of your bridges). There is a daemon to collect metrics and report them to Grafana (via Telegraf and Influx). MirageOS unikernels also support IPv6, you're not limited to legacy IP.
diff --git a/atom b/atom
index ca7097d..d7c8c6e 100644
--- a/atom
+++ b/atom
@@ -1,6 +1,6 @@
-urn:uuid:8167ecfe-9676-11e7-8dc1-68f728e7bbbc robur 2023-02-10T09:38:32-00:00 2021-11-16T15:06:35-00:00 <p>In 2021 we in <a href="https://robur.coop/">Robur</a> have been working towards easing deployment of reproducible mirage applications. The work has been funded by the European Union under the <a href="https://pointer.ngi.eu/">Next Generation Internet (NGI Pointer) initiative</a>. The result is <a href="https://builds.robur.coop">online</a>.</p>
+urn:uuid:8167ecfe-9676-11e7-8dc1-68f728e7bbbc robur 2023-05-16T17:25:50-00:00 2021-11-16T15:06:35-00:00 <p>In 2021 we in <a href="https://robur.coop/">Robur</a> have been working towards easing deployment of reproducible mirage applications. The work has been funded by the European Union under the <a href="https://pointer.ngi.eu/">Next Generation Internet (NGI Pointer) initiative</a>. The result is <a href="https://builds.robur.coop">online</a>.</p>
<p>The overall goal is to push MirageOS into production in a trustworthy way. We worked on reproducible builds for <a href="https://opam.ocaml.org">Opam</a> packages and <a href="https://mirageos.org">MirageOS</a> - with the infrastructure being reproducible itself. Reproducible builds are crucial for supply chain security - everyone can reproduce the exact same binary (by using the same sources and environment), without reproducible builds we would not publish binaries.</p>
-<p>Reproducible builds are also great for fleet management: by inspecting the hash of the binary that is executed, we can figure out which versions of which libraries are in the unikernel - and suggest updates if newer builds are available or if a used library has a security flaw -- <code>albatross-client-local update my-unikernel</code> is everything needed for an update.</p>
+<p>Reproducible builds are also great for fleet management: by inspecting the hash of the binary that is executed, we can figure out which versions of which libraries are in the unikernel - and suggest updates if newer builds are available or if a used library has a security flaw -- <code>albatross-client update my-unikernel</code> is everything needed for an update.</p>
<p>Several ready-to-use MirageOS unikernels are built on a daily basis - ranging from <a href="https://builds.robur.coop/job/dns-primary-git/">authoritative DNS servers</a> (<a href="https://builds.robur.coop/job/dns-secondary/">secondary</a>, <a href="https://builds.robur.coop/job/dns-letsencrypt-secondary/">let's encrypt DNS solver</a>), <a href="https://builds.robur.coop/job/dnsvizor/">DNS-and-DHCP service (similar to dnsmasq)</a>, <a href="https://builds.robur.coop/job/tlstunnel/">TLS reverse proxy</a>, <a href="https://builds.robur.coop/job/unipi/">Unipi - a web server that delivers content from a git repository</a>, <a href="https://builds.robur.coop/job/dns-resolver/">DNS resolver</a>, <a href="https://builds.robur.coop/job/caldav/">CalDAV server</a>, and of course your own MirageOS unikernel.</p>
<h2>Brief robur and MirageOS introduction</h2>
<p><a href="https://mirageos.org">MirageOS</a> is an operating system, developed in OCaml, which produces unikernels. A unikernel serves a single purpose and is a single process, i.e. only has the really needed dependencies. For example, an OpenVPN endpoint does neither include persistent storage (block device, file system) nor user management. MirageOS unikernels are developed in <a href="https://ocaml.org">OCaml</a>, a statically typed and type-safe programming language - which avoids common pitfalls from the grounds up (spatial and temporal memory safety issues).</p>
@@ -27,16 +27,17 @@ $ pkg update
$ pkg install solo5 albatross
</code></pre>
<p>For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using <a href="https://opam.ocaml.org">opam</a> (<code>opam install solo5 albatross</code>). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.</p>
-<p>There is no configuration needed. Start the <code>albatross_console</code> and the <code>albatross_daemon</code> service (via <code>systemctl daemon-reload ; systemctl start albatross_daemon</code> on Linux or <code>service albatross_daemon start</code> on FreeBSD). Executing <code>albatross-client-local info </code> should return success (exit code 0) and no running unikernel. You may need to be in the albatross group, or change the permissions of the Unix domain socket (<code>/run/albatross/util/vmmd.sock</code> on Linux, <code>/var/run/albatross/util/vmmd.sock</code> on FreeBSD).</p>
+<p>There is no configuration needed. Start the <code>albatross_console</code> and the <code>albatross_daemon</code> service (via <code>systemctl daemon-reload ; systemctl start albatross_daemon</code> on Linux or <code>service albatross_daemon start</code> on FreeBSD). Executing <code>albatross-client info </code> should return success (exit code 0) and no running unikernel. You may need to be in the albatross group, or change the permissions of the Unix domain socket (<code>/run/albatross/util/vmmd.sock</code> on Linux, <code>/var/run/albatross/util/vmmd.sock</code> on FreeBSD).</p>
<p>To check that albatross works, get the latest hello world unikernel and run it:</p>
<pre><code>$ wget https://builds.robur.coop/job/hello/build/latest/bin/hello.hvt
-$ albatross-client-local console my-hello-unikernel & # this is sent to the background since it waits and displays the console of the unikernel named "my-hello-unikernel"
-$ albatross-client-local create my-hello-unikernel hello.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
-$ albatross-client-local create --arg='--hello="Hello,\ my\ unikernel" my-hello-unikernel hello.hvt # executes the same unikernel, but passes the boot parameter "--hello"
-$ fg # back to albatross-client-local console
+$ albatross-client console my-hello-unikernel & # this is sent to the background since it waits and displays the console of the unikernel named "my-hello-unikernel"
+$ albatross-client create my-hello-unikernel hello.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
+$ albatross-client create --arg='--hello="Hello,\ my\ unikernel" my-hello-unikernel hello.hvt # executes the same unikernel, but passes the boot parameter "--hello"
+$ fg # back to albatross-client console
$ Ctrl-C # kill that process
</code></pre>
-<p>Voila, we have a working albatross installation. Albatross also supports a remote client (using a TLS handshake) <code>albatross-client-bistro</code>, monitoring of unikernels (<code>albatross_stat</code> and <code>albatross_influx</code> services), and a TLS endpoint (via inetd with <code>albatross-tls-inetd</code>).</p>
+<p>Voila, we have a working albatross installation. Albatross also supports a remote client (using a TLS handshake) <code>albatross-client --ca <ca.pem> --ca-key <ca.key> --server-ca <cacert.pem> --destination <myhost></code>, monitoring of unikernels (<code>albatross_stats</code> and <code>albatross_influx</code> services), and a TLS endpoint (inetd/systemd: <code>albatross-tls-endpoint</code>).</p>
+<p>Please ensure to have albatross in version of at least 2.0.0 to follow this page.</p>
<h3>Network for your unikernel</h3>
<p>Next we want to setup networking for our unikernels. We use a so-called "bridge" interface for this, which is a virtual network switch where you connect "tap" interfaces (layer 2 ethernet devices). A MirageOS unikernel uses tap interfaces for communication. We give our bridge the name "service" (and for example for monitoring and management you may want to setup another bridge "management").</p>
<p>If you're using a network manager that is capable of setting up bridge interfaces, use that interface.</p>
@@ -61,15 +62,15 @@ ifconfig_service="inet 10.0.42.1/24"
<p>Afterwards either restart your system or re-run the service scripts to have the bridge setup in your running system.</p>
<p>To check that the networking works, get the latest static website unikernel and run it:</p>
<pre><code>$ wget https://builds.robur.coop/job/static-website/build/latest/bin/https.hvt
-$ albatross-client-local console my-website & # this is sent to the background since it waits and displays the console of the unikernel named "my-website"
-$ albatross-client-local create --net=service --arg='--ipv4=10.0.42.2/24' my-website https.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
+$ albatross-client console my-website & # this is sent to the background since it waits and displays the console of the unikernel named "my-website"
+$ albatross-client create --net=service --arg='--ipv4=10.0.42.2/24' my-website https.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
$ ping 10.0.42.2 # should receive answers
$ open http://10.0.42.2 # in your browser - also https://10.0.42.2 (you'll get a certificate warning)
$ wget http://10.0.42.2/ # should download the Hello Mirage world!
$ wget --no-check-certificate https://10.0.42.2/ # should also download the Hello Mirage world!
-$ fg # back to albatross-client-local console
+$ fg # back to albatross-client console
$ Ctrl-C # kill that process
-$ albatross-client-local destroy my-website # kills the unikernel
+$ albatross-client destroy my-website # kills the unikernel
</code></pre>
<p>When you reached this point, you have successfully launched a MirageOS unikernel, and are able to communicate from your computer with it. This uses the OCaml networking stack, and the host bridge interface.</p>
<h2>Routing and Internet</h2>
@@ -109,13 +110,13 @@ On FreeBSD, add <code>ifconfig_service="addm em0"</co
<h2>Unikernel execution</h2>
<p>Let's test that your unikernels have access to the Internet by using the <a href="https://hannes.robur.coop/Posts/Traceroute">traceroute</a> unikernel:</p>
<pre><code>$ wget https://builds.robur.coop/job/traceroute/build/latest/bin/traceroute.hvt
-$ albatross-client-local console my-traceroute & # this is sent to the background since it waits and displays the console of the unikernel named "my-traceroute"
-$ albatross-client-local create --net=service --arg='--ipv4=10.0.42.2/24' --arg='--ipv4-gateway=10.0.42.1' my-traceroute traceroute.hvt # the IP configuration depends on your setup, use your public IP address and actual router IP if you've set it up.
-$ fg # back to albatross-client-local console
+$ albatross-client console my-traceroute & # this is sent to the background since it waits and displays the console of the unikernel named "my-traceroute"
+$ albatross-client create --net=service --arg='--ipv4=10.0.42.2/24' --arg='--ipv4-gateway=10.0.42.1' my-traceroute traceroute.hvt # the IP configuration depends on your setup, use your public IP address and actual router IP if you've set it up.
+$ fg # back to albatross-client console
$ Ctrl-C # kill that process
</code></pre>
<p>That's it. Albatross has more features, such as block devices, multiple bridges (for management, private networks, ...), restart if the unikernel exited with specific exit code, assignment of a unikernel to a specific CPU. It also has remote command execution and resource limits (you can allow your friends to execute a number of unikernels with limited memory and block storage accessing only some of your bridges). There is a daemon to collect metrics and report them to Grafana (via Telegraf and Influx). MirageOS unikernels also support IPv6, you're not limited to legacy IP.</p>
- urn:uuid:a225bf44-9230-569f-8852-1b5d2132a749 Robur Reproducible Builds 2023-02-09T09:25:33-00:00 canopy 2019-09-10T19:40:14-00:00 <h1>MirageOS</h1>
+ urn:uuid:a225bf44-9230-569f-8852-1b5d2132a749 Robur Reproducible Builds 2023-05-16T17:25:50-00:00 canopy 2019-09-10T19:40:14-00:00 <h1>MirageOS</h1>
<p>MirageOS is a software suite to build custom-tailored operating systems from (mostly open source) small individual libraries. It has been developed since 2009 at the University of Cambridge, UK and is written in the programming language <a href="/Our%20Work/Technology-Employed#OCaml">OCaml</a>.</p>
<p>It compiles the necessary OCaml libraries into a unikernel - a small operating system, each built for a certain purpose. For each unikernel we can pick from hundreds of permissively licensed open source libraries which implement network protocols, storage on block devices, or interfaces to network devices via the hypervisor or host operating system. As we only put into each one exactly what is needed, each unikernel is fast; instantly booting and, as there is less code base, it is easier to maintain and keep secure.</p>
<p>As an example to see how lines of code compare, here are the number of lines of code needed for different elements of our <a href="/Our%20Work/Projects#Bitcoin%20Pinata">Bitcoin Pinata</a>, measured in thousands of lines of code:</p>