Cluster API Provider Metal3 for Managed Bare Metal Hardware

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management. The API itself is shared across multiple cloud providers. Cluster API Provider Metal3 is one of the providers for Cluster API and enables users to deploy a Cluster API based cluster on top of bare metal infrastructure using Metal3.

Compatibility with Cluster API

| CAPM3 version | Cluster API version | CAPM3 Release |  CAPI Release | | ------------- | ------------------- | ------------- | -------------- | | v1beta1 | v1beta1 | v1.1.X |  v1.1.X        | | v1beta1 | v1beta1 | v1.2.X |  v1.2.X        | | v1beta1 | v1beta1 | v1.3.X |  v1.3.X        | | v1beta1 | v1beta1 | v1.4.X |  v1.4.X        | | v1beta1 | v1beta1 | v1.5.X |  v1.5.X        | | v1beta1 | v1beta1 | v1.6.X |  v1.6.X        | | v1beta1 | v1beta1 | v1.7.X |  v1.7.X        | | v1beta1 | v1beta1 | v1.8.X |  v1.8.X        | | v1beta1 | v1beta1 | v1.9.X |  v1.9.X        | | v1beta1 | v1beta1 | v1.10.X |  v1.10.X       | | v1beta1 | v1beta1 | v1.11.X |  v1.11.X       | | v1beta1 | v1beta2 | v1.12.X |  v1.12.X       | | v1beta2 | v1beta2 | v1.13.X |  v1.13.X       | | v1beta2 | v1beta2 | v1.14.X |  v1.14.X       |

Deploying the metal3 provider

The recommended method is using Clusterctl.

Starting from v0.5.0 release of Cluster API Provider Metal3, Baremetal Operator is decoupled from Cluster API Provider Metal3 deployments when deployed via clusterctl. For this reason, Baremetal Operator will not be installed when initializing the Metal3 provider with clusterctl, and its CRDs and controller need to be manually installed. Example flow of installing Metal3 provider:

1. Install Cluster API core, bootstrap and control-plane providers. This will

clusterctl init --core cluster-api:v1.13.0 \
       --bootstrap kubeadm:v1.13.0 \
       --control-plane kubeadm:v1.13.0 -v5

1. Install Metal3 provider. This will install the latest version of Cluster API

clusterctl init --infrastructure metal3

You can also specify the provider version by appending a version tag to the provider name as follows:

clusterctl init --infrastructure metal3:v1.13.0

1. Deploy Baremetal Operator manifests and CRDs. You need to install

git clone https://github.com/metal3-io/baremetal-operator.git
   kubectl create namespace baremetal-operator-system
   cd baremetal-operator
   kustomize build config/default | kubectl apply -f -

1. Install Ironic. There are a couple of ways to do it.

Please refer to the getting-started for more info.

Pivoting ⚠️

Starting from v0.5.0 release of Cluster API Provider Metal3, Baremetal Operator is decoupled from Cluster API Provider Metal3 deployments when deployed via clusterctl. For that reason, when performing clusterctl move, custom objects outside of the Cluster API chain or not part of CAPM3 will not be pivoted to a target cluster. An example of those objects is BareMetalHost, or user created ConfigMaps and Secrets which are reconciled by Baremetal Operator. To ensure that those objects are also pivoted as part of clusterctl move, the clusterctl.cluster.x-k8s.io label needs to be set on the BareMetalHost CRD before pivoting. If there are other CRDs that also need to be pivoted to the target cluster, the same label needs to be set on them.

All the other objects owned by BareMetalHost, such as Secret and ConfigMap, don't require this label to be set, because they hold ownerReferences to BareMetalHost, and that is good enough for clusterctl to move the entire hierarchy of the BareMetalHost object.

CAPM3 also automatically sets the clusterctl.cluster.x-k8s.io/block-move annotation on a claimed BareMetalHost until its pause and status annotations are applied, to prevent it from being moved mid-reconcile. This annotation blocks the entire clusterctl move operation while present on any object. A BareMetalHost stuck in this state can stall the whole move, not just its own cluster.

Development Environment

There are multiple ways to set up a development environment:

• Using Tilt
• Other management cluster
• See metal3-dev-env for an

API

See the API Documentation for details about the objects used with this Cluster API provider. You can also see the cluster deployment workflow for the outline of the deployment process.

Architecture

The architecture with the components involved can be found in the docs.

E2E test

To trigger e2e tests on a PR, use the following phrases:

Integration tests

• /test metal3-ubuntu-e2e-integration-test-main runs integration e2e

• /test metal3-centos-e2e-integration-test-main runs integration e2e

Release-1.14 branch:

• /test metal3-ubuntu-e2e-integration-test-release-1-14 runs integration e2e

• /test metal3-centos-e2e-integration-test-release-1-14 runs integration e2e

Release-1.13 branch:

• /test metal3-ubuntu-e2e-integration-test-release-1-13 runs integration e2e

• /test metal3-centos-e2e-integration-test-release-1-13 runs integration e2e

Release-1.12 branch:

• /test metal3-ubuntu-e2e-integration-test-release-1-12 runs integration e2e

• /test metal3-centos-e2e-integration-test-release-1-12 runs integration e2e

Basic tests

Unlike integration tests, basic tests focus on the target cluster creation without involving pivoting from the bootstrap cluster. To run basic tests use:

• /test metal3-ubuntu-e2e-basic-test-main runs basic e2e tests with main

• /test metal3-centos-e2e-basic-test-release-1-14 runs basic e2e tests on

• /test metal3-centos-e2e-basic-test-release-1-13 runs basic e2e tests on

• /test metal3-centos-e2e-basic-test-release-1-12 runs basic e2e tests on

Feature tests

On main branch:

• /test metal3-ubuntu-e2e-feature-test-main-pivoting runs e2e pivot based

• /test metal3-ubuntu-e2e-feature-test-main-remediation runs e2e remediation

• /test metal3-ubuntu-e2e-feature-test-main-features runs e2e non pivot

• /test metal3-centos-e2e-feature-test-main-pivoting runs e2e pivot based

• /test metal3-centos-e2e-feature-test-main-remediation runs e2e remediation

• /test metal3-centos-e2e-feature-test-main-features runs e2e non pivot based

Release-1.14 branch:

• /test metal3-ubuntu-e2e-feature-test-release-1-14-pivoting runs e2e pivot

• /test metal3-ubuntu-e2e-feature-test-release-1-14-remediation runs e2e

• /test metal3-ubuntu-e2e-feature-test-release-1-14-features runs e2e non

• /test metal3-centos-e2e-feature-test-release-1-14-pivoting runs e2e pivot

• /test metal3-centos-e2e-feature-test-release-1-14-remediation runs e2e

• /test metal3-centos-e2e-feature-test-release-1-14-features runs e2e non

Release-1.13 branch:

• /test metal3-ubuntu-e2e-feature-test-release-1-13-pivoting runs e2e pivot

• /test metal3-ubuntu-e2e-feature-test-release-1-13-remediation runs e2e

• /test metal3-ubuntu-e2e-feature-test-release-1-13-features runs e2e non

• /test metal3-centos-e2e-feature-test-release-1-13-pivoting runs e2e pivot

• /test metal3-centos-e2e-feature-test-release-1-13-remediation runs e2e

• /test metal3-centos-e2e-feature-test-release-1-13-features runs e2e non

Release-1.12 branch:

• /test metal3-ubuntu-e2e-feature-test-release-1-12-pivoting runs e2e pivot

• /test metal3-ubuntu-e2e-feature-test-release-1-12-remediation runs e2e

• /test metal3-ubuntu-e2e-feature-test-release-1-12-features runs e2e non

• /test metal3-centos-e2e-feature-test-release-1-12-pivoting runs e2e pivot

• /test metal3-centos-e2e-feature-test-release-1-12-remediation runs e2e

• /test metal3-centos-e2e-feature-test-release-1-12-features runs e2e non

Upgrade tests

Clusterctl upgrade tests

CAPM3 tests upgrade from all supported releases to the current one. We run upgrade test on main branch from different releases:

• /test metal3-e2e-clusterctl-upgrade-test-main runs e2e clusterctl

• /test metal3-e2e-clusterctl-upgrade-test-release-1-14 runs e2e clusterctl

• /test metal3-e2e-clusterctl-upgrade-test-release-1-13 runs e2e clusterctl

• /test metal3-e2e-clusterctl-upgrade-test-release-1-12 runs e2e clusterctl

K8s upgrade tests

CAPM3 tests upgrading Kubernetes between the last 3 releases. The trigger takes the format: /test metal3-e2e---upgrade-test-

• /test metal3-e2e-1-35-1-36-upgrade-test-main
• /test metal3-e2e-1-36-1-37-upgrade-test-release-1-14
• /test metal3-e2e-1-35-1-36-upgrade-test-release-1-13
• /test metal3-e2e-1-34-1-35-upgrade-test-release-1-12

• Triggers follow the pattern: /test metal3--e2e--test-