nixkube / nix + kube / nix-csi / nix-nri
Mount /nix into Kubernetes pods using the CSI ephemeral volumes or NRI(Node Resource Interface). Volumes share lifetime with Pods and are embedded into the Podspec.
Deploying nixkube
Stick your pubkeys in ./keys and they will be imported into the module system then run the following command and you'll have nixkube deployed.
nix run --file . kubenixEval.deploymentScript -- --yes --prune
If you'd rather mangle YAML yourself you can use
nix build --file . easykubenix.manifestYAMLFile
and stuff the result into Kustomize, a blender or your Kubernetes cluster
Which commit to deploy
Deploy the tested branch, or a release. CI moves tested to a commit of
main once every test of it passed, and by then its images are on ghcr.io.
main itself can name an image that CI has not pushed yet.
The manifest names each image by content: the init image by a tag from its
store hash, and scratch by digest. A release pins the init image by
digest too. To run an image this checkout did not build, set
nixkube.images.nix (for example to ghcr.io/nixidae/nixkube/nix:main)
and nixkube.imagePullPolicy = "Always". To publish your own build of the
init image, run nix run --file . nixImage.pushArch.x86_64-linux; it skips
a tag that already holds the same image.
Deploying workloads
nixkube supports two methods for injecting Nix stores into pods:
CSI Ephemeral Volumes (Explicit)
Request Nix stores explicitly via CSI volumeAttributes. Specify one or more:
storePath- Direct nix store path (highest priority)flakeRef- Flake reference to buildnixExpr- Nix expression to evaluate and build
apiVersion: v1
kind: Pod
metadata:
name: hello-csi
spec:
containers:
- name: hello
image: nixos/nix:latest
volumeMounts:
- name: nix
mountPath: /nix
volumes:
- name: nix
ephemeral:
volumeClaimTemplate:
spec:
accessModes: ["ReadOnlyMany"]
storageClassName: ephemeral-storage
resources:
requests:
storage: 1Gi
csi:
driver: nixkube
volumeAttributes:
x86_64-linux: /nix/store/hello-......
aarch64-linux: /nix/store/hello-......
flakeRef: github:nixos/nixpkgs/nixos-unstable#hello
nixExpr: |
let
nixpkgs = builtins.fetchTree {
type = "github";
owner = "nixos";
repo = "nixpkgs";
ref = "nixos-unstable";
};
pkgs = import nixpkgs { };
in
pkgs.hello
The first successful option by priority wins.
Tip: For command arrays and environment variables, use lib.getExe to reference executables without managing full store paths:
command = [ (lib.getExe pkgs.hello) ]
env = {
name = "HELLO_CONFIG";
value = pkgs.hello-config;
}
NRI Plugin (Automatic via Annotations)
Use pod annotations to automatically inject Nix stores without explicit volume requests:
apiVersion: v1
kind: Pod
metadata:
name: hello-nri
annotations:
nix-nri/pod: |
/nix/store/hello-......
nix-nri/pod@aarch64-linux: |
/nix/store/hello-aarch64-......
spec:
containers:
- name: hello
image: nixos/nix:latest
# /nix is automatically mounted by NRI plugin
Runtime compatibility
What each mount path gives a pod, by container runtime (CRI) and OCI runtime
(RuntimeClass). ro and rw are the NRI modes: nixkube/pod-rw asks for a
writable /nix.
| | runc | crun | youki | gVisor (runsc) | Kata Containers | |---|---|---|---|---|---| | containerd | CSI, NRI ro/rw | CSI, NRI ro/rw | CSI, NRI ro/rw | CSI; NRI refused¹ | CSI, NRI ro², ³ | | CRI-O | CSI, NRI ro/rw | CSI, NRI ro/rw | CSI, NRI ro/rw | no container runs⁴ | CSI, NRI ro², ³ |
- gVisor keeps the container's rootfs out of the node's reach. The
- A VM gets /nix as a read-only virtio-fs share, and waits in the VM until
nixkube/pod-path mounts are
refused with the reason: the tree is hardlinks into the node's store.
Set nixkube.nri.vmRuntimeHandlers if your Kata handler is not kata.
The wait is a static busybox that NRI puts in front of the container's
args, because Kata checks the command exists before any OCI hook runs;
crictl inspect shows the changed args. A container that also mounts
something under /nix fails with EROFS, because its mount point would be
inside the read-only share.
- Needs Kata 4.0.0 or later. Earlier guest kernels have a use-after-free
- Measured with CRI-O 1.36.5 and runsc 20260406, on a plain busybox pod
CRI-O with Kata needs skip_mount_home = "true" in containers-storage's
overlay options, or the VM gets an empty rootfs. CRI-O refuses a Kata pod on
the host network ("Host networking requested, not supported by runtime");
containerd accepts one. CRI-O creates a container when an NRI plugin returns
an error, without the plugin's changes.
A pod with hostUsers: false gets both mounts under CRI-O. Under
containerd (2.3.4 to 2.4.1 and main) no user-namespaced container starts
from an image with zero layers, such as a FROM scratch image that
nixkube fills: "mkdirat rootfs/proc: permission denied", with or without
nixkube. Lillecarl/containerd#1 has the cause and #2 a fix. An image with
one layer, even one without /proc, is not affected.
Tested by nix/uml on Kubernetes 1.37, containerd 2.3.4, CRI-O 1.36.5,
runc 1.4.3, crun 1.29.1, youki 0.7.0, gVisor 20260406 and Kata 4.2.0 on
nested QEMU. umlMatrix. is each combination as an
attribute, by hand and not in CI:
nix run --file . umlMatrix.uml-crio.driver -- --out ./o
Examples:
DocumentationThis project is made possible by