132 lines
7.7 KiB
Text
132 lines
7.7 KiB
Text
---
|
|
title: Operating systems
|
|
author: hannes
|
|
tags: overview, operating system, mirageos
|
|
abstract: Basics of OS and MirageOS
|
|
---
|
|
|
|
## Remark about this site
|
|
|
|
Sorry to be late with this entry, but I had to fix some issues:
|
|
|
|
- this website is based on [Canopy](https://github.com/Engil/Canopy), the content is stored as markdown in a [git repository](https://github.com/hannesm/hannes.nqsb.io)
|
|
- it was running in a FreeBSD jail, but when I compiled too much the underlying zfs file system didn't feel happy (and is now hanging in kernel space in a read)
|
|
- no remote power switch (borrowed to a friend 3 weeks ago), nobody was willing to go to the data centre and reboot
|
|
- I wanted to move it anyways to a host where I can deploy Xen guest VMs
|
|
- turns out the Xen compilation and deployment mode needed some love:
|
|
- I ported a newer [bin_prot](https://github.com/hannesm/bin_prot/tree/113.33.00+xen) to xen
|
|
- I wrote a clean patch to [serve via TLS](https://github.com/Engil/Canopy/pull/15) (including [HSTS header](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security) and redirecting HTTP (moved permanently) to HTTPS)
|
|
- I found a memory leak in the [mirage-http](https://github.com/mirage/mirage-http/pull/23) library
|
|
- I was travelling
|
|
- good news: it now works on Xen, and there is [an atom feed](https://hannes.nqsb.io/atom)
|
|
- life of an "eat your own dogfood" full stack engineer ;)
|
|
|
|
## What is an operating system?
|
|
|
|
Wikipedia says: "An operating system (OS) is system software that manages
|
|
computer hardware and software resources and provides common services for
|
|
computer programs." Great. In other terms, it is an abstraction layer.
|
|
Applications don't need to deal with the low-level bits (device drivers) of the
|
|
computer.
|
|
|
|
But if we look at the landscape of deployed operating systems, there is a lot
|
|
more going on than abstracting devices: usually this includes process management (scheduler),
|
|
memory management (virtual memory), [C
|
|
library](https://en.wikipedia.org/wiki/C_standard_library), user management
|
|
(including access control), persistent storage (file system), network stack,
|
|
etc. all being part of the kernel, and executed in kernel space. A
|
|
counterexample is [Minix](http://www.minix3.org/), which consists of a tiny
|
|
microkernel, and executes the above mentioned services as user-space processes.
|
|
|
|
The kernel has full access to the hardware, runs in
|
|
[ring 0](https://en.wikipedia.org/wiki/Protection_ring) and thus an issue in the
|
|
kernel is devastating for the entire system. Since developers are not perfect,
|
|
there will always be bugs in code. Since we are (or at least I am) interested
|
|
in robust systems, every piece of code running in ring 0 is of concern to us.
|
|
|
|
This is the pre-virtualisation world, now there is on top of that a
|
|
[hypervisor](https://en.wikipedia.org/wiki/Hypervisor), which runs in ring -1.
|
|
The hypervisor gives access to memory and hardware to virtual machines, and
|
|
schedules virtual machines on processors.
|
|
|
|
|
|
|
|
This ominous "cloud" uses hypervisors on huge amount of physical machines, and
|
|
executes off-the-shelf operating systems as virtual machines on top. Accounting
|
|
is then done by resource usage (time, bandwidth, storage).
|
|
|
|
## From scratch
|
|
|
|
Ok, now we have hypervisors which already deals with memory and scheduling. Why
|
|
should we have the very same functionality again in the virtual machine?
|
|
|
|
Additionally, earlier in my live (back in 2005 at the Dutch hacker camp "What
|
|
the hack") I proposed (together with Andreas Bogk) to [phase out UNIX before
|
|
2038-01-19](https://berlin.ccc.de/~hannes/wth.pdf) (this is when `time_t`
|
|
overflows, unless promoted to 64 bit), and replace it with Dylan. A [random
|
|
comment](http://www.citizen428.net/blog/2005/08/03/what-the-hack-recap/) about
|
|
our talk on the Internet is "the proposal that rewritting an entire OS in a
|
|
language with obscure sytanx was somewhat original. However, I now somewhat feel
|
|
a strange urge to spend some time on Dylan, which is really weird..."
|
|
|
|
Being without funding back then, we didn't get far (hugest success was a
|
|
[TCP/IP](https://github.com/dylan-hackers/network-night-vision/) stack in
|
|
Dylan), and as mentioned earlier I went into formal methods and mechanised
|
|
proofs of full functional correctness properties...
|
|
|
|
A bit more than two years ago, David pointed me to
|
|
[MirageOS](https://mirage.io), an operating system from scratch in the
|
|
functional and statically typed language [OCaml](https://ocaml.org). Since then
|
|
I spend nearly every day on OCaml libraries (with varying success on being happy
|
|
with my code). There are also more than two people caring about MirageOS.
|
|
|
|
The idea is pretty straightforward: use a hypervisor, and its hardware
|
|
abstractions (virtualised input/output and network device), and execute the
|
|
OCaml runtime directly on it. No C library included (since May 2015, see [this
|
|
thread](http://lists.xenproject.org/archives/html/mirageos-devel/2014-05/msg00070.html)).
|
|
This OCaml-based virtual machine runs in kernel space (this is bad, but
|
|
[this article](https://matildah.github.io/posts/2016-01-30-unikernel-security.html) shows
|
|
why it isn't too bad) for now, and
|
|
consists of the required libraries only (this website is 16MB in size, which
|
|
includes the static CSS and JavaScript (bootstrap, jquery, fonts), HTTP, TLS, git, TCP/IP libraries,
|
|
and I didn't even bother to strip it). The memory management in MirageOS is
|
|
straightforward: the hypervisor provides the OCaml runtime with a chunk of memory, which
|
|
immediately takes all of it.
|
|
|
|
This is much simpler to configure and deploy than a UNIX operating system:
|
|
There is no virtual memory, no process management, no file
|
|
system, no user management in the image.
|
|
|
|
At compile time (which is configuration time), I hardcode the TLS keys, remote
|
|
git repository, which IP and ports to use, and deployment is a `xl create
|
|
canopy.xl` (which contains the name of the image, the name of the bridge
|
|
interface, and how much memory it may consume).
|
|
|
|
The full command line for this website is: `mirage configure --no-opam --xen -i Posts -n "full stack engineer" -r https://github.com/hannesm/hannes.nqsb.io.git --dhcp false --net direct --ip 198.167.222.205 --netmask 255.255.255.0 --gateways 198.167.222.1 --tls 443 --port 80` :)
|
|
|
|
Instead of running on a multi-purpose operating system, this website uses a
|
|
bunch of libraries, which are compiled and statically
|
|
linked into the virtual machine image.
|
|
|
|
MirageOS uses the module system of OCaml to define how interfaces should be, thus an
|
|
application developer does not need to care whether they are using the TCP/IP
|
|
stack written in OCaml, or the sockets API of a UNIX operating system. This
|
|
also allows to compile and debug your library on UNIX using off-the-shelf tools
|
|
before deploying it as a virtual machine (NB: this is a lie, since there is code
|
|
which is only executed when running on Xen, and this code can be buggy) ;).
|
|
|
|
Most of the MirageOS ecosystem is developed under MIT/ISC/BSD license, which
|
|
allows everybody to use it for whichever project they want.
|
|
|
|
Did I mention that by using less code the attack vectors shrink immediately? In
|
|
addition to that, using a memory safe programming language where the developer
|
|
does not need to care about allocations and bounds checks, immediately removes
|
|
several classes of security problems (namely spatial and temporal memory
|
|
issues). There is enough left, such as logical issues, and there is no access
|
|
control (that's fine for this website, the content is "protected" by GitHub's
|
|
access control).
|
|
|
|
I hope I gave some insight into what the purpose of an operating systems is, and
|
|
how MirageOS fits into the picture. I'm interested in feedback, either via
|
|
[twitter](https://twitter.com/h4nnes) or as an issue on the [data repository on
|
|
GitHub](https://github.com/hannesm/hannes.nqsb.io/issues).
|