Slurm Bridge

Run [Slurm] as a [Kubernetes] scheduler. A [Slinky] project.

Table of Contents

• Slurm Bridge

Overview

[Slurm] and [Kubernetes] are workload managers originally designed for different kinds of workloads. Kubernetes excels at scheduling workloads that run for an indefinite amount of time, with potentially vague resource requirements, on a single node, with loose policy, but can scale its resource pool infinitely to meet demand; Slurm excels at quickly scheduling workloads that run for a finite amount of time, with well-defined resource requirements and topology, on multiple nodes, with strict policy, and a known resource pool.

This project enables the best of both workload managers. It contains a [Kubernetes] scheduler to manage select workloads from Kubernetes, which allows for co-location of Kubernetes and Slurm workloads within the same cluster. This means the same hardware can be used to run both traditional HPC and cloud-like workloads, reducing operating costs. On hybrid nodes, the hardware is shared over time: a physical node must not run native Slurm user workloads and Slurm-bridge-managed Kubernetes user workloads simultaneously. Kubernetes and Slurm system daemons are expected to remain co-located on those nodes.

Using slurm-bridge, workloads can be submitted from within a Kubernetes context as a Pod, PodGroup, Job, JobSet, or LeaderWorkerSet and from a Slurm context using salloc or sbatch. Workloads submitted via Slurm will execute as they would in a Slurm-only environment, using slurmd. Workloads submitted from Kubernetes will have their resource requirements translated into a representative Slurm job by slurm-bridge. That job will serve as an external job and will be scheduled by the Slurm controller. Upon resource allocation to a K8s workload by the Slurm controller, slurm-bridge will bind the workload's pod(s) to the allocated node(s). At that point, the kubelet will launch and run the pod the same as it would within a standard Kubernetes instance.

!"Slurm Bridge Architecture"

For additional architectural notes, see the [architecture] docs.

Features

Slurm

Slurm is a full featured HPC workload manager. To highlight a few features:

• [Priority][slurm-priority]: assigns priorities to jobs upon submission and

• [Preemption][slurm-preempt]: stop one or more low-priority jobs to let a

• [QoS][slurm-qos]: sets of policies affecting scheduling priority,

• [Fairshare][slurm-fairshare]: distribute resources equitably among users

Compatibility

Each Slurm Bridge release series supports all Kubernetes minor versions that are supported upstream when it is released, starting with Kubernetes 1.35. That set is recorded for each release series. Support for newer Kubernetes minors may be backported in a patch release after end-to-end validation; these backports are optional.

| Release | Kubernetes 1.35 | Kubernetes 1.36 | Kubernetes 1.37 | | :------ | :-------------: | :-------------: | :-------------: | | 1.3.X | ✓ | ✓ | ✓ | | 1.2.X | ✓ | ✓ | — | | 1.1.X | ✓ | ✓ | — | | 1.0.X | ✓ | ✓ | — |

✓ means supported; — means unsupported. X denotes the patch number within a release series. Use the latest release in each series. Backported support is marked with the first release that supports it.

| Release | Minimum Slurm (Data Parser) | | :------ | :-------------------------------------------------: | | 1.3.X | [25.11][slurm-25.11] ([v0.0.44][slurm-data_parser]) | | 1.2.X | [25.11][slurm-25.11] ([v0.0.44][slurm-data_parser]) | | 1.1.X | [25.11][slurm-25.11] ([v0.0.44][slurm-data_parser]) | | 1.0.X | [25.11][slurm-25.11] ([v0.0.44][slurm-data_parser]) |

Upgrade support

Within a release series, intermediate patches may be skipped: 1.2.0 can upgrade directly to any newer 1.2.X. Between series sharing the same first version number, upgrade one series at a time: 1.1 → 1.2 → 1.3. See the upgrade guide for preparation and checks.

Limitations

• Bridge jobs use exclusive, whole-node allocations by default. Workloads that

• Supports [DRA Driver CPU][dra-driver-cpu] for CPUs, plus indexed GPU and

• NVIDIA GPU backend selection is resource-name based:

• Native cpu requests do not activate CPU DRA. Pods must explicitly request

• Native CPU requests reserve CPU capacity in Slurm, but do not constrain the

CPU DRA requires Kubernetes 1.36 or newer and uses one shared device per socket. Native CPU scheduling and GPU/NIC DRA remain supported on Kubernetes 1.35. See Workload for CPU DRA configuration and migration requirements.

Installation

Create a secret for slurm-bridge to communicate with Slurm.

export $(scontrol token username=slurm lifespan=infinite)
kubectl create namespace slurm
kubectl create namespace slurm-bridge
kubectl create secret generic slurm-bridge-token --namespace=slurm --from-literal="SLURM_JWT=$SLURM_JWT" --type=Opaque

Install the slurm-bridge scheduler after configuring the quickstart prerequisites, including cert-manager for the default webhook certificates. The default configuration requires the built-in Workload and PodGroup APIs. On Kubernetes 1.35, add --set scheduler.featureGates.SlurmBridgeGenericWorkload=false to the command below; see Built-in PodGroup for other cluster versions.

helm install slurm-bridge oci://ghcr.io/slinkyproject/charts/slurm-bridge \
  --namespace=slurm --create-namespace

For additional instructions, see the [quickstart] guide.

Documentation

Follow the upgrade policy when updating an existing installation. Upgrades stop all bridge components before applying changes.

Project documentation is located in the [docs] directory of this repository.

[Slinky documentation][slinky-docs] is hosted on the web.

Support and Development

Feature requests, code contributions, and bug reports are welcome!

Github/Gitlab submitted issues and PRs/MRs are handled on a best effort basis.

The SchedMD official issue tracker is at .

To schedule a demo or simply to reach out, please [contact SchedMD][contact-schedmd].

License

Copyright (C) SchedMD LLC.

Licensed under the Apache License, Version 2.0 you may not use project except in compliance with the license.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

[architecture]: ./docs/architecture.md [contact-schedmd]: https://www.schedmd.com/slurm-resources/contact-schedmd/ [docs]: https://github.com/SlinkyProject/slurm-bridge/tree/main/docs [dra-driver-cpu]: https://github.com/kubernetes-sigs/dra-driver-cpu [dra-driver-nvidia-gpu]: https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu [dra-example-driver]: https://github.com/kubernetes-sigs/dra-example-driver [dranet]: https://github.com/kubernetes-sigs/dranet [kubernetes]: https://kubernetes.io/ [nvidia-dra-driver]: https://github.com/NVIDIA/k8s-dra-driver-gpu [quickstart]: ./docs/quickstart.md [slinky]: https://slinky.ai/ [slinky-docs]: https://slinky.schedmd.com/ [slurm]: https://slurm.schedmd.com/overview.html [slurm-25.11]: https://www.schedmd.com/slurm-version-25-11-0-is-now-available/ [slurm-data_parser]: https://slurm.schedmd.com/rest_clients.html#data_parser_lifecycle [slurm-fairshare]: https://slurm.schedmd.com/fair_tree.html [slurm-preempt]: https://slurm.schedmd.com/preempt.html [slurm-priority]: https://slurm.schedmd.com/priority_multifactor.html [slurm-qos]: https://slurm.schedmd.com/qos.html