RunCVM Container Runtime

Introduction

RunCVM (Run Container Virtual Machine) is an experimental open-source Docker container runtime for Linux, created by Struan Bartlett at NewsNow Labs, that makes launching standard containerised workloads and system workloads (e.g. Systemd, Docker, even OpenWrt) in VMs as easy as launching a container.

View on Asciinema

Quick start

Install:

curl -s -o - https://raw.githubusercontent.com/newsnowlabs/runcvm/main/runcvm-scripts/runcvm-install-runtime.sh | sudo sh

Now launch an nginx VM listening on port 8080:

docker run --runtime=runcvm --name nginx1 --rm -p 8080:80 nginx

Launch a MariaDB VM, with 2 cpus and 2G memory, listening on port 3306:

docker run --runtime=runcvm --name mariadb1 --rm -p 3306:3306 --cpus 2 --memory 2G --env=MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 mariadb

Launch a vanilla ubuntu VM, with interactive terminal:

docker run --runtime=runcvm --name ubuntu1 --rm -it ubuntu

Gain another interactive console on ubuntu1:

docker exec -it ubuntu1 bash

Launch with VNC:

docker run --runtime=runcvm --name ubuntu2 --env=RUNCVM_DISPLAY_MODE=vnc -p 5900:5900 ubuntu

Launch a VM with 1G memory and a 1G ext4-formatted backing file mounted at /var/lib/docker and stored in the underlying container's filesystem:

docker run -it --runtime=runcvm --init --memory=1G --env=RUNCVM_DISKS=/disks/docker,/var/lib/docker,ext4,1G <docker-image>

Launch a VM with 2G memory and a 5G ext4-formatted backing file mounted at /var/lib/docker and stored in a dedicated Docker volume on the host:

docker run -it --runtime=runcvm --init --memory=2G --mount=type=volume,src=runcvm-disks,dst=/disks --env=RUNCVM_DISKS=/disks/docker,/var/lib/docker,ext4,5G <docker-image>

Launch a 3-node Docker Swarm on a network with 9000 MTU and, on the swarm, an http global service:

git clone https://github.com/newsnowlabs/runcvm.git && \
cd runcvm/tests/50-http-docker-swarm && \
NODES=3 MTU=9000 ./test

Graphical workloads with VNC

Launch Ubuntu:

cat <<EOF | docker build --tag=ubuntu-x -
FROM ubuntu:jammy
ENV DEBIAN_FRONTEND=noninteractive
RUN apt update && apt -y install --install-recommends apt-utils kmod wget iproute2 systemd ca-certificates curl gnupg udev dbus lightdm
ENTRYPOINT ["/lib/systemd/systemd"]
ENV RUNCVM_DISPLAY_MODE=vnc
EOF
docker run -d --runtime=runcvm -m 2g --name=vnc -p 5900:5900 ubuntu-x

System workloads

Docker+Sysbox runtime demo - Launch Ubuntu running Systemd and Docker with Sysbox runtime; then within it run an Alpine _Sysbox_ container; and, _within that_ install dockerd and run a container from the 'hello-world' image:

cat <<EOF | docker build --tag=ubuntu-docker-sysbox -
FROM ubuntu:jammy
RUN apt update && apt -y install apt-utils kmod wget iproute2 systemd \
    ca-certificates curl gnupg udev dbus && \
    curl -fsSL https://get.docker.com | bash
RUN wget -O /tmp/sysbox.deb \
    https://downloads.nestybox.com/sysbox/releases/v0.6.2/sysbox-ce_0.6.2-0.linux_amd64.deb && \
    apt -y install /tmp/sysbox.deb
ENTRYPOINT ["/lib/systemd/systemd"]
ENV RUNCVM_DISKS='/disks/docker,/var/lib/docker,ext4,1G;/disks/sysbox,/var/lib/sysbox,ext4,1G'
VOLUME /disks
EOF
docker run -d --runtime=runcvm -m 2g --name=ubuntu-docker-sysbox ubuntu-docker-sysbox
docker exec ubuntu-docker-sysbox bash -c "docker run --rm --runtime=sysbox-runc alpine ash -x -c 'apk add docker; dockerd &>/dev/null & sleep 5; docker run --rm hello-world'"
docker rm -fv ubuntu-docker-sysbox

• Watch on Asciinema

cat <<EOF | docker build --tag=ubuntu-docker-runcvm -
FROM ubuntu:jammy
RUN apt update && apt -y install apt-utils kmod wget iproute2 systemd \
    ca-certificates curl gnupg udev dbus && \
    curl -fsSL https://get.docker.com | bash
COPY --from=newsnowlabs/runcvm:latest /opt /opt/
RUN rm -f /etc/init.d/docker && \
    bash /opt/runcvm/scripts/runcvm-install-runtime.sh --no-dockerd && \
    echo kvm_intel >>/etc/modules
ENTRYPOINT ["/lib/systemd/systemd"]
ENV RUNCVM_DISKS='/disks/docker,/var/lib/docker,ext4,1G'
VOLUME /disks
EOF
docker run -d --runtime=runcvm -m 2g --name=ubuntu-docker-runcvm ubuntu-docker-runcvm
docker exec ubuntu-docker-runcvm bash -c "docker run --rm --runtime=runcvm alpine ash -x -c 'apk add docker; dockerd &>/dev/null & sleep 5; docker run --rm hello-world'"
docker rm -fv ubuntu-docker-runcvm

Docker+GVisor runtime demo - Launch Ubuntu running Systemd and Docker with GVisor runtime; then within it run the 'hello-world' image in a _GVisor_ container:

cat <<EOF | docker build --tag=ubuntu-docker-gvisor -
FROM ubuntu:jammy
RUN apt update && apt -y install apt-utils kmod wget iproute2 systemd \
    ca-certificates curl gnupg udev dbus jq && \
    curl -fsSL https://get.docker.com | bash
RUN curl -fsSL https://gvisor.dev/archive.key | gpg --dearmor -o /usr/share/keyrings/gvisor-archive-keyring.gpg
RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases release main" >/etc/apt/sources.list.d/gvisor.list && \
    apt update && \
    apt-get install -y runsc
RUN [ ! -f /etc/docker/daemon.json ] && echo '{}' > /etc/docker/daemon.json; cat /etc/docker/daemon.json | jq '.runtimes.runsc.path="/usr/bin/runsc"' | tee /etc/docker/daemon.json
ENTRYPOINT ["/lib/systemd/systemd"]
ENV RUNCVM_DISKS='/disks/docker,/var/lib/docker,ext4,1G'
VOLUME /disks
EOF
docker run -d --runtime=runsc -m 2g --name=ubuntu-docker-gvisor ubuntu-docker-gvisor
docker exec ubuntu-docker-gvisor bash -c "docker run --rm --runtime=runsc hello-world"
docker rm -fv ubuntu-docker-gvisor

Launch OpenWrt - with port forward to LuCI web UI on port 10080:

docker import --change='ENTRYPOINT ["/sbin/init"]' https://archive.openwrt.org/releases/23.05.2/targets/x86/generic/openwrt-23.05.2-x86-generic-rootfs.tar.gz openwrt-23.05.2 && \
docker network create --subnet 172.128.0.0/24 runcvm-openwrt && \
echo -e "config interface 'loopback'\n\toption device 'lo'\n\toption proto 'static'\n\toption ipaddr '127.0.0.1'\n\toption netmask '255.0.0.0'\n\nconfig device\n\toption name 'br-lan'\n\toption type 'bridge'\n\tlist ports 'eth0'\n\nconfig interface 'lan'\n\toption device 'br-lan'\n\toption proto 'static'\n\toption ipaddr '172.128.0.5'\n\toption netmask '255.255.255.0'\n\toption gateway '172.128.0.1'\n" >/tmp/runcvm-openwrt-network && \
docker run -it --rm --runtime=runcvm --name=openwrt --network=runcvm-openwrt --ip=172.128.0.5 -v /tmp/runcvm-openwrt-network:/etc/config/network -p 10080:80 openwrt-23.05.2

• Watch on Asciinema

RunCVM-in-Portainer walk-through

RunCVM was born out of difficulties experienced using the Docker and Podman CLIs to launch Kata Containers v2, and a belief that launching containerised workloads in VMs using Docker needn't be so complicated.

Motivations included: efforts to re-add OCI CLI commands for docker/podman to Kata v2 to support Docker & Podman; other Kata issues #3358, #1123, #1133, #3038; #5321; #6861; Podman issues #8579 and #17070; and Kubernetes issue #40114; though please note, since authoring RunCVM some of these issues may have been resolved.

Like Kata, RunCVM aims to be a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualisation technology.

However, while Kata aims to launch standard container images inside a restricted-privileges namespace inside a VM running a single fixed and heavily customised kernel and Linux distribution optimised for this purpose, RunCVM intentionally aims to launch container _or VM_ images as the _VM's root filesystem_ using stock or bespoke Linux kernels, the upshot being RunCVM's can run VM workloads that Kata's security and kernel model would explicitly prevent.

For example:

• RunCVM can launch system images expecting to interface directly with hardware, like OpenWRT
• RunCVM can launch VMs nested inside a RunCVM VM - i.e. an 'inner' RunCVM Container/VM guest can be launched by Docker running within an 'outer' RunCVM Container/VM guest (assuming the host supports nested VMs) - in this sense, RunCVM is 'reentrant'.

• Compatible with docker run (with experimental support for podman run).
• Uses a lightweight 'wrapper-runtime' technology that subverts the behaviour of the standard container runtime runc to cause a VM to be launched within the container (making its code footprint and external dependencies extremely small, and its internals extremely simple and easy to understand and tailor for specific purposes).
• Highly portable among Linux distributions and development platforms providing KVM. Can even be installed on GitHub Codespaces!
• Written, using off-the-shelf open-source components, almost entirely in shell script for simplicity, portability and ease of development.

RunCVM makes some trade-offs in return for this simplicity. See the full list of features and limitations.

Contents

• Introduction
• Quick start
• Motivation
• Licence
• Project aims
• Project ambitions
• Applications for RunCVM
• How RunCVM works
• System requirements
• Supported distributions
• Installation
• Upgrading
• Features and Limitations
• RunCVM vs Kata comparison
• Kernel selection
• Option reference
• Advanced usage
• Developing
• Building
• Testing
• Contributing
• Support
• Uninstallation
• Legals

Licence

RunCVM is free and open-source, licensed under the Apache Licence, Version 2.0. See the LICENSE file for details.

Project aims

• Run any standard container workload in a VM using docker run with no need to customise images or the command line (except adding --runtime=runcvm)
• Run unusual container workloads, like dockerd and systemd that will not run in standard container runtimes
• Maintain a similar experience within a RunCVM VM as within a container: process table, network interfaces, stdio, exit code handling should be broadly similar to maximise compatibility
• Container start/stop/kill semantics respected, where possible providing clean VM shutdown on stop
• VM console accessible as one would expect using docker run -it, docker start -ai and docker attach (and so on), generally good support for other docker container subcommands
• Efficient container startup, by using virtiofs to serve a container's filesystem directly to a VM (instead of unpacking an image into a backing file)
• Improved security compared to the standard container runtime, and as much security as possible without compromising the simplicity of the implementation
• Command-line and image-embedded options for customising the a container's VM specifications, devices, kernel
• Intelligent kernel selection, according to the distribution used in the image being launched
• No external dependencies, except for Docker/Podman and relevant Linux kernel modules (kvm and tun)
• Support multiple Docker network interfaces attached to a created (but not yet running) container using docker run --network= and docker network connect
• Dual-stack (IPv4 and IPv6) networking, where the host and Docker are already configured to support it

Project ambitions

• Support for booting VM with a file-backed disk root fs generated from the container image, instead of only virtiofs root
• Support running foreign-architecture VMs by using QEMU dynamic CPU emulation for the entire VM (instead of the approach used by https://github.com/multiarch/qemu-user-static which uses dynamic CPU emulation for each individual binary)
• Support for QEMU microvm or potentially Amazon Firecracker
• More natural console support with independent stdout and stderr channels for docker run -it
• Improve VM boot time and other behaviours using custom kernel
• Support for specific hardware e.g. graphics display served via VNC

Applications for RunCVM

The main applications for RunCVM are:

1. Running and testing applications that:

1. Running existing container workloads with increased security
2. Testing container workloads that are already intended to launch in VM environments, such as on fly.io
3. Developing any of the above applications in Dockside (see RunCVM and Dockside)

How RunCVM works

RunCVM's 'wrapper' runtime, runcvm-runtime, receives container create commands triggered by docker run/create commands, modifies the configuration of the requested container in such a way that the created container will launch a VM that boots from the container's filesystem, and then passes the request on to the standard container runtime (runc) to actually create and start the container.

For a deep dive into RunCVM's internals, see the section on Developing RunCVM.

System requirements

RunCVM should run on any amd64 (x86_64) hardware (or VM) running Linux Kernel >= 5.10, and that supports KVM and Docker. So if your host can already run KVM VMs and Docker then it should run RunCVM.

RunCVM has no other host dependencies, apart from Docker (or experimentally, Podman) and the kvm and tun kernel modules. RunCVM comes packaged with all binaries and libraries it needs to run (including its own QEMU binary).

RunCVM is tested on x86-64 hosts with KVM running Debian 12 (Bookworm) and Ubuntu 24.04 LTS (Noble Numbat), with Docker Engine, both Docker's own packages and Ubuntu's docker.io. The integration tests can also run where KVM is unavailable, under software emulation (see Testing).

rp_filter sysctl settings

For RunCVM to support Docker DNS within Container/VMs, the following condition on /proc/sys/net/ipv4/conf/ must be met:

• the max of all/rp_filter and /rp_filter should be 0 ('No Source Validation') or 2 (Loose mode as defined in RFC3704 Loose Reverse Path)

This means that:

• if all/rp_filter will be set to 0, then /rp_filter must be set to 0 or 2

• if all/rp_filter will be set to 1, then /rp_filter must be set to 2

• if all/rp_filter will be set to 2, then no further action is needed

• the Debian default is 0;
• the Ubuntu default is 2;
• the Google Cloud Debian image has default 1 and rp_filter settings in /etc/sysctl.d/60-gce-network-security.conf must be modified or overridden to support RunCVM.

This requirement concerns IPv4 only. A dual-stack Container/VM's DNS requests and docker exec connections still use IPv4, and Linux has no IPv6 equivalent of rp_filter, so IPv6 needs no further host configuration.

Supported distributions

RunCVM aims to support each distribution's latest release and one or two before it (for Ubuntu, its LTS releases), while they remain supported upstream, as the image a Container/VM runs. It bundles a kernel for each of these distributions, built from the release shown, which a Container/VM boots unless its image has a kernel of its own (see Kernel auto-detection). The table lists the image releases that the integration tests (tests/25-images-kernels) boot:

| Distribution | Image releases tested | RunCVM's kernel (RUNCVM_KERNEL) | Kernel built from | Kernel series | Module compression | Image releases whose own modprobe can load the kernel's modules | |---|---|---|---|---|---|---| | Debian | 12 (Bookworm) | debian | Debian 12 (Bookworm) | 6.1 | none | all | | Ubuntu | 24.04 LTS (Noble Numbat) | ubuntu | Ubuntu 22.04 LTS (Jammy Jellyfish) | 5.15 | none | all | | Oracle Linux | none (the kernel is tested with an Alpine Linux image) | ol | Oracle Linux 9 | 5.14 | xz | 7 and later (9 and 10 once kmod is installed) | | Alpine Linux | the latest (the alpine image) | alpine | Alpine Linux 3.19 (linux-virt) | 6.6 | gzip | all | | OpenWrt | none (the kernel is tested with an Alpine Linux image) | openwrt | Alpine Linux 3.19 (linux-virt) | 6.6 | none | all |

Each kernel comes from the release in the "Kernel built from" column, which stays fixed for a given RunCVM version: its series (as shown) does not change, and each build of RunCVM takes that release's newest kernel update (e.g. Debian 12's newest 6.1 kernel).

The last column concerns the workload's own module loading, e.g. modprobe, or dockerd loading overlay or br_netfilter, in the VM. It reflects which compressed formats each release's kmod (or, for Alpine Linux, BusyBox) can read: for example, Debian 9 (Stretch), Ubuntu 16.04 LTS (Xenial Xerus) and Ubuntu 18.04 LTS (Bionic Beaver) read only uncompressed modules, Debian 10 (Buster), Debian 11 (Bullseye) and Ubuntu 20.04 LTS (Focal Fossa) also read xz, and later releases also read zstd. Oracle Linux 9 and 10 images include no kmod until it is installed. RunCVM itself loads the modules it needs, while the VM boots, with its own bundled BusyBox modprobe, which reads uncompressed, gzip and xz modules, whatever the image.

Other releases of these distributions, including the older releases RunCVM aims to support, may work, but are not yet tested.

Other distributions

RunCVM bundles no kernel for other distributions. How it treats some popular ones, as named by their images' /etc/os-release:

| Distribution | ID | ID_LIKE | Status | |---|---|---|---| | Rocky Linux | rocky | rhel centos fedora | Refused, unless launched with --env=RUNCVM_KERNEL= (e.g. ol) | | AlmaLinux | almalinux | rhel centos fedora | Refused, unless launched with RUNCVM_KERNEL | | CentOS Stream | centos | rhel fedora | Refused, unless launched with RUNCVM_KERNEL | | Red Hat Universal Base Image (UBI) | rhel | fedora | Refused, unless launched with RUNCVM_KERNEL | | Amazon Linux | amzn | fedora | Refused, unless launched with RUNCVM_KERNEL | | Fedora | fedora | | Refused, unless launched with RUNCVM_KERNEL | | openSUSE Leap, openSUSE Tumbleweed | opensuse-leap, opensuse-tumbleweed | suse opensuse | Refused, unless launched with RUNCVM_KERNEL | | Arch Linux | arch | | Refused, unless launched with RUNCVM_KERNEL | | Kali Linux | kali | debian | Refused, unless launched with RUNCVM_KERNEL (e.g. debian) | | Wolfi (Chainguard) | wolfi | | Refused, unless launched with RUNCVM_KERNEL | | Distroless (Debian-based, e.g. gcr.io/distroless/base-debian12) | debian | | Debian's kernel, as for Debian | | BusyBox, and images built FROM scratch | (no /etc/os-release) | | Debian's kernel |

A Container/VM launched with RUNCVM_KERNEL boots the named kernel, whatever the image.

Installation

Run:

curl -s -o - https://raw.githubusercontent.com/newsnowlabs/runcvm/main/runcvm-scripts/runcvm-install-runtime.sh | sudo sh

This will:

• Install the RunCVM software package to /opt/runcvm (installation elsewhere is currently unsupported)
• For Docker support:

• For Podman support (experimental)

• Check your system network device rp_filter settings, and amend them if necessary

docker run --runtime=runcvm --rm -it hello-world

Install on Google Cloud

Create an image that will allow instances to have VMX capability:

gcloud compute images create debian-12-vmx --source-image-project=debian-cloud --source-image-family=debian-12   --licenses="https://compute.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"

Now launch a VM, install Docker and RunCVM:

cat >/tmp/startup-script.sh <<EOF
#!/bin/bash
apt update && apt -y install apt-utils kmod wget iproute2 systemd \
    ca-certificates curl gnupg udev dbus jq && \
    mkdir -p /etc/docker && echo '{"userland-proxy": false}' >/etc/docker/daemon.json && \
    curl -fsSL https://get.docker.com | bash && \
    curl -s -o - https://raw.githubusercontent.com/newsnowlabs/runcvm/main/runcvm-scripts/runcvm-install-runtime.sh | sudo REPO=newsnowlabs/runcvm:latest sh
EOF
gcloud compute instances create runcvm-vmx-test --zone=us-central1-a --machine-type=n2-highmem-2 --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default --metadata-from-file=startup-script=/tmp/startup-script.sh --no-restart-on-failure --maintenance-policy=TERMINATE --provisioning-model=SPOT --instance-termination-action=STOP  --no-service-account --no-scopes --create-disk=auto-delete=yes,boot=yes,image=debian-12-vmx,mode=rw,size=50,type=pd-ssd --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --labels=goog-ec-src=vm_add-gcloud --reservation-affinity=any

Upgrading

To upgrade, follow this procedure:

1. Stop all RunCVM containers.
2. Run /opt/runcvm/scripts/runcvm-install-runtime.sh (or rerun the installation command - which runs the same script)
3. Start any RunCVM containers.

Features and limitations

In the below summary of RunCVM's current main features and limitations, [+] is used to indicate an area of compatibility with standard container runtimes and [-] is used indicate a feature of standard container runtimes that is unsupported.

N.B. docker run and docker exec options not listed below are unsupported and their effect, if used, is unspecified.

• docker run

$ docker network create --scope=local testnet >/dev/null && docker run --name=test --rm -it --runtime=kata --network=testnet --entrypoint=/bin/ash alpine -c 'for n in test google.com 8.8.8.8; do echo "ping $n ..."; ping -q -c 8 -i 0.5 $n; done'; docker network rm testnet >/dev/null
    ping test ...
    ping: bad address 'test'
    ping google.com ...
    ping: bad address 'google.com'
    ping 8.8.8.8 ...
    PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
    8 packets transmitted, 8 packets received, 0% packet loss
    round-trip min/avg/max = 0.911/1.716/3.123 ms
    
    $ docker network create --scope=local testnet >/dev/null && docker run --name=test --rm -it --runtime=runcvm --network=testnet --entrypoint=/bin/ash alpine -c 'for n in test google.com 8.8.8.8; do echo "ping $n ..."; ping -q -c 8 -i 0.5 $n; done'; docker network rm testnet >/dev/null
    ping test ...
    PING test (172.25.8.2): 56 data bytes
--- test ping statistics ---
    8 packets transmitted, 8 packets received, 0% packet loss
    round-trip min/avg/max = 0.033/0.085/0.137 ms
    ping google.com ...
    PING google.com (172.217.16.238): 56 data bytes
--- google.com ping statistics ---
    8 packets transmitted, 8 packets received, 0% packet loss
    round-trip min/avg/max = 8.221/8.398/9.017 ms
    ping 8.8.8.8 ...
    PING 8.8.8.8 (8.8.8.8): 56 data bytes
--- 8.8.8.8 ping statistics ---
    8 packets transmitted, 8 packets received, 0% packet loss
    round-trip min/avg/max = 1.074/1.491/1.801 ms

$ docker run --rm --runtime=kata --name=test -m 2g --env=RUNCVM_KERNEL_DEBUG=1 -it alpine ash -c 'free -h'
                total        used        free      shared  buff/cache   available
    Mem:           3.9G       94.4M        3.8G           0        3.7M        3.8G
    Swap:             0           0           0
    $ docker run --rm --runtime=kata --name=test -m 3g --env=RUNCVM_KERNEL_DEBUG=1 -it alpine ash -c 'free -h'
                total        used        free      shared  buff/cache   available
    Mem:           4.9G      107.0M        4.8G           0        3.9M        4.8G
    Swap:             0           0           0
    $ docker run --rm --runtime=kata --name=test -m 0g --env=RUNCVM_KERNEL_DEBUG=1 -it alpine ash -c 'free -h'
                total        used        free      shared  buff/cache   available
    Mem:           1.9G       58.8M        1.9G           0        3.4M        1.9G
    Swap:             0           0           0

$ find /opt/runcvm/kernels/ -maxdepth 2 | sed 's!^/opt/runcvm/kernels/!!; /^$/d'
debian
debian/latest
debian/5.10.0-16-amd64
alpine
alpine/latest
alpine/5.15.59-0-virt
ubuntu
ubuntu/latest
ubuntu/5.15.0-43-generic
ol
ol/5.14.0-70.22.1.0.1.el9_0.x86_64
ol/latest

docker run --rm --runtime=runcvm --env=RUNCVM_KERNEL=ol hello-world

docker run -it --runtime=runcvm --init --mount=type=volume,src=runcvm-disks,dst=/disks --env=RUNCVM_DISKS=/disks/root,/,ext4,40G <docker-image>