From 36a581e53e4d607bbee4cf218ccde0fdd5200241 Mon Sep 17 00:00:00 2001 From: The Robur Team Date: Tue, 22 Oct 2024 10:26:12 +0000 Subject: [PATCH] Pushed by YOCaml 2 --- articles/arguments.html | 479 ++++++++++++++++++++++++++++++++++++++++ atom.xml | 11 +- feed.xml | 9 +- index.html | 13 ++ rss1.xml | 6 + tags.html | 4 +- 6 files changed, 518 insertions(+), 4 deletions(-) create mode 100644 articles/arguments.html diff --git a/articles/arguments.html b/articles/arguments.html new file mode 100644 index 0000000..b1ed2f9 --- /dev/null +++ b/articles/arguments.html @@ -0,0 +1,479 @@ + + + + + + + + Robur's blogRuntime arguments in MirageOS + + + + + + + + +
+

blog.robur.coop

+
+ The Robur cooperative blog. +
+
+
Back to index + +
+

Runtime arguments in MirageOS

+

TL;DR: Passing runtime arguments around is tricky, and prone to change every other month.

+

Motivation

+

Sometimes, as an unikernel developer and also as operator, it's nice to have +some runtime arguments passed to an unikernel. Now, if you're into OCaml, +command-line parsing - together with error messages, man page generation, ... - +can be done by the amazing cmdliner +package from Daniel Bünzli.

+

MirageOS uses cmdliner for command line argument passing. This also enabled +us from the early days to have nice man pages for unikernels (see +my-unikernel-binary --help). There are two kinds +of arguments: those at configuration time (mirage configure), such as the +target to compile for, and those at runtime - when the unikernel is executed.

+

In Mirage 4.8.1 and 4.8.0 (released October 2024) there have been some changes +to command-line arguments, which were motivated by 4.5.0 (released April 2024) +and user feedback.

+

First of all, our current way to pass a custom runtime argument to a unikernel +(unikernel.ml):

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  let term = Arg.(value & opt string "Hello World!" doc) in
+  Mirage_runtime.register_arg term
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+          Logs.info (fun f -> f "%s" (hello ()));
+          Time.sleep_ns (Duration.of_sec 1) >>= fun () -> loop (n - 1)
+    in
+    loop 4
+end
+
+

We define the Cmdliner.Term.t +in line 6 (let term = ..) - which provides documentation ("How to say hello."), the option to +use (["hello"] - which is then translated to --hello=), that it is optional, +of type string (cmdliner allows you to convert the incoming strings to more +complex (or more narrow) data types, with decent error handling).

+

The defined argument is directly passed to Mirage_runtime.register_arg, +(in line 7) so our binding hello is of type unit -> string. +In line 14, the value of the runtime argument is used (hello ()) for printing +a log message.

+

The nice property is that it is all local in unikernel.ml, there are no other +parts involved. It is just a bunch of API calls. The downside is that hello () +should only be evaluated after the function start was called - since the +Mirage_runtime needs to parse and fill in the command line arguments. If you +call hello () earlier, you'll get an exception "Called too early. Please delay +this call to after the start function of the unikernel.". Also, since +Mirage_runtime needs to collect and evaluate the command line arguments, the +Mirage_runtime.register_arg may only be called at top-level, otherwise you'll +get another exception "The function register_arg was called to late. Please call +register_arg before the start function is executed (e.g. in a top-level binding).".

+

Another advantage is, having it all in unikernel.ml means adding and removing +arguments doesn't need another execution of mirage configure. Also, any +type can be used that the unikernel depends on - the config.ml is compiled only +with a small set of dependencies (mirage itself) - and we don't want to impose a +large dependency cone for mirage just because someone may like to use +X509.Key_type.t as argument type.

+

Earlier, before mirage 4.5.0, we had runtime and configure arguments mixed +together. And code was generated when mirage configure was executed to +deal with these arguments. The downsides included: we needed serialization for +all command-line arguments (at configure time you could fill the argument, which +was then serialized, and deserialized at runtime and used unless the argument +was provided explicitly), they had to appear in config.ml (which also means +changing any would need an execution of mirage configure), since they generated code +potential errors were in code that the developer didn't write (though we had +some __POS__ arguments to provide error locations in the developer code).

+

Related recent changes are:

+
    +
  • in mirage 4.8.1, the runtime arguments to configure the OCaml runtime system +(such as GC settings, randomization of hashtables, recording of backtraces) +are now provided using the cmdliner-stdlib +package.
    • +
  • in mirage 4.8.0, for git, dns-client, and happy-eyeballs devices the optional +arguments are generated by default - so they are always available and don't +need to be manually done by the unikernel developer.
    • +
+

Let's dive a bit deeper into the history.

+

History

+

In MirageOS, since the early stages (I'll go back to 2.7.0 (February 2016) where +functoria was introduced) used an embedded fork of cmdliner to handle command +line arguments.

+

Animated changes to the hello world unikernel

+

February 2016 (Mirage 2.7.0)

+

When looking into the MirageOS 2.x series, here's the code for our hello world +unikernel:

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    "Unikernel.Hello" (console @-> job)
+
+let () = register "hello-key" [main $ default_console]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (C: V1_LWT.CONSOLE) = struct
+  let start c =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        C.log c (Key_gen.hello ());
+        OS.Time.sleep 1.0 >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

As you can see, the cmdliner term was provided in config.ml, and in +unikernel.ml the expression Key_gen.hello () was used - Key_gen was +a module generated by the mirage configure invocation.

+

You can as well see that the term was wrapped in Key.create "hello" - where +this string was used as the identifier for the code generation.

+

As mentioned above, a change needed to be done in config.ml and a +mirage configure to take effect.

+

July 2016 (Mirage 2.9.1)

+

The OS.Time was functorized with a Time functor:

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    "Unikernel.Hello" (console @-> time @-> job)
+
+let () = register "hello-key" [main $ default_console $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (C: V1_LWT.CONSOLE) (Time : V1_LWT.TIME) = struct
+  let start c _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        C.log c (Key_gen.hello ());
+        Time.sleep 1.0 >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

February 2017 (Mirage pre3)

+

The Time signature changed, now the sleep_ns function sleeps in nanoseconds. +This avoids floating point numbers at the core of MirageOS. The helper package +duration is used to avoid manual conversions.

+

Also, the console signature changed - and log is now inside the Lwt monad.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (console @-> time @-> job)
+
+let () = register "hello-key" [main $ default_console $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (C: V1_LWT.CONSOLE) (Time : V1_LWT.TIME) = struct
+  let start c _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        C.log c (Key_gen.hello ()) >>= fun () ->
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

February 2017 (Mirage 3)

+

Another big change is that now console is not used anymore, but +logs.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time_lwt.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

January 2020 (Mirage 3.7.0)

+

The _lwt is dropped from the interfaces (we used to have Mirage_time and +Mirage_time_lwt - where the latter was instantiating the former with concrete +types: type 'a io = Lwt.t and type buffer = Cstruct.t -- in a cleanup +session we dropped the _lwt interfaces and opam packages. The reasoning was +that when we'll get around to move to another IO system, we'll move everything +at once anyways. No need to have lwt and something else (async, or nowadays +miou or eio) in a single unikernel.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  foreign
+    ~keys:[Key.abstract hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

October 2021 (Mirage 3.10)

+

Some renamings to fix warnings. Only config.ml changed.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt string "Hello World!" doc))
+
+let main =
+  main
+    ~keys:[key hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

June 2023 (Mirage 4.4)

+

The argument was moved to runtime.

+

config.ml

+
open Mirage
+
+let hello =
+  let doc = Key.Arg.info ~doc:"How to say hello." ["hello"] in
+  Key.(create "hello" Arg.(opt ~stage:`Run string "Hello World!" doc))
+
+let main =
+  main
+    ~keys:[key hello]
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (Key_gen.hello ());
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

March 2024 (Mirage 4.5)

+

The runtime argument is in config.ml refering to the argument as string +("Unikernel.hello"), and being passed to the start function as argument.

+

config.ml

+
open Mirage
+
+let runtime_args = [ runtime_arg ~pos:__POS__ "Unikernel.hello" ]
+
+let main =
+  main
+    ~runtime_args
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  Arg.(value & opt string "Hello World!" doc)
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time hello =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" hello);
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

October 2024 (Mirage 4.8)

+

Again, moved out of config.ml.

+

config.ml

+
open Mirage
+
+let main =
+  main
+    ~packages:[package "duration"]
+    "Unikernel.Hello" (time @-> job)
+
+let () = register "hello-key" [main $ default_time]
+
+

and unikernel.ml

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  Mirage_runtime.register_arg Arg.(value & opt string "Hello World!" doc)
+
+module Hello (Time : Mirage_time.S) = struct
+  let start _time =
+    let rec loop = function
+      | 0 -> Lwt.return_unit
+      | n ->
+        Logs.info (fun f -> f "%s" (hello ()));
+        Time.sleep_ns (Duration.of_sec 1) >>= fun () ->
+        loop (n-1)
+    in
+    loop 4
+end
+
+

2024 (Not yet released)

+

This is the future with time defunctorized. Read more in the discussion. +To delay the start function, a dep of noop is introduced.

+

config.ml

+
open Mirage
+
+let main =
+  main
+    ~packages:[package "duration"]
+    ~dep:[dep noop]
+    "Unikernel" job
+
+let () = register "hello-key" [main]
+
+

and unikernel.ml

+
open Lwt.Infix
+open Cmdliner
+
+let hello =
+  let doc = Arg.info ~doc:"How to say hello." [ "hello" ] in
+  Mirage_runtime.register_arg Arg.(value & opt string "Hello World!" doc)
+
+let start () =
+  let rec loop = function
+    | 0 -> Lwt.return_unit
+    | n ->
+      Logs.info (fun f -> f "%s" (hello ()));
+      Mirage_timer.sleep_ns (Duration.of_sec 1) >>= fun () ->
+      loop (n-1)
+  in
+  loop 4
+
+

Conclusion

+

The history of hello world shows that over time we slowly improve the developer +experience, and removing the boilerplate needed to get MirageOS unikernels up +and running. This is work over a decade including lots of other (here invisible) +improvements to the mirage utility.

+

Our current goal is to minimize the code generated by mirage, since code +generation has lots of issues (e.g. error locations, naming, binary size). It +is a long journey. At the same time, we are working on improving the performance +of MirageOS unikernels, developing unikernels that are useful in the real +world (VPN endpoint, DNSmasq replacement, ...), and also simplifying the +deployment of MirageOS unikernels.

+

If you're interested in MirageOS and using it in your domain, don't hesitate +to reach out to us (via eMail: team@robur.coop) - we're keen to deploy MirageOS +and find more domains where it is useful. If you can spare a dime, we're a +registered non-profit in Germany - and can provide tax-deductable receipts for +donations (more information).

+ +
+ +
+ + + + diff --git a/atom.xml b/atom.xml index bdcc89d..6a1959c 100644 --- a/atom.xml +++ b/atom.xml @@ -3,12 +3,21 @@ https://blog.robur.coop/atom.xml The Robur's blog YOCaml - 2024-10-21T00:00:00Z + 2024-10-22T00:00:00Z The Robur Team + + https://blog.robur.coop//articles/arguments.html + Runtime arguments in MirageOS + 2024-10-22T00:00:00Z + The history of runtime arguments to a MirageOS unikernel + + + + https://blog.robur.coop//articles/finances.html How has robur financially been doing since 2018? diff --git a/feed.xml b/feed.xml index 7fd855a..5a6cc45 100644 --- a/feed.xml +++ b/feed.xml @@ -5,9 +5,16 @@ https://blog.robur.coop/ - Mon, 21 Oct 2024 00:00:00 GMT + Tue, 22 Oct 2024 00:00:00 GMT https://www.rssboard.org/rss-specification YOCaml + + Runtime arguments in MirageOS + https://blog.robur.coop//articles/arguments.html + + https://blog.robur.coop//articles/arguments.html + Tue, 22 Oct 2024 00:00:00 GMT + How has robur financially been doing since 2018? https://blog.robur.coop//articles/finances.html diff --git a/index.html b/index.html index a5e7ac8..578fdba 100644 --- a/index.html +++ b/index.html @@ -25,6 +25,19 @@

Essays and ramblings

  1. +
    + + +
    +
    + 2024-10-22 + Runtime arguments in MirageOS
    +

    The history of runtime arguments to a MirageOS unikernel

    +
    + +
    +
    +
  2. diff --git a/rss1.xml b/rss1.xml index b4cb97f..3ddc3f3 100644 --- a/rss1.xml +++ b/rss1.xml @@ -6,6 +6,7 @@ + @@ -21,6 +22,11 @@ + + Runtime arguments in MirageOS + https://blog.robur.coop//articles/arguments.html + + How has robur financially been doing since 2018? https://blog.robur.coop//articles/finances.html diff --git a/tags.html b/tags.html index d54ee36..8d24d36 100644 --- a/tags.html +++ b/tags.html @@ -41,7 +41,7 @@

    MirageOS

    -     +

    OpenVPN