• Usage

• Developer Guide

Usage

Prerequisites

Before installing the OpenDataHub Operator, ensure your environment meets the following requirements:

Platform Requirements

• OpenShift: Version 4.19 or higher

External Operators (Prerequisites)

The following external operators are required or recommended depending on different use case. These must be installed separately before ODH operator:

| Operator | Purpose | Required For | Installation | |----------|---------|--------------|--------------| | OpenShift Cert Manager Operator | Certificate management | Gateway with TLS, KServe with Istio | Optional - Install from OperatorHub for llm-d, MaaS and Kueue | | Red Hat build of Kueue Operator | Advanced job queueing and resource management | Distributed workloads, job scheduling with Kueue component | Optional - Install from OperatorHub for Kueue integration | | Red Hat Connectivity Link Operator | Manage the lifecycle of the Kuadrant system | Optional - Install from OperatorHub | | LeaderWorkerSet (LWS) Operator | Leader-worker set controller for distributed training | Advanced distributed training workflows | Optional - Install from OperatorHub for Kueue integration | | OpenTelemetry Operator | Distributed tracing and telemetry | Monitoring and observability | Optional - for advanced monitoring | | Tempo Operator | Distributed tracing backend | Tracing infrastructure | Optional - for trace storage | | Cluster Observability Operator | Enhanced cluster monitoring | Production monitoring | Optional - for comprehensive observability | | Perses Operator | Dashboard and visualization for metrics | Monitoring dashboards and visualizations | Optional - Install from OperatorHub for enhanced metric visualization | | JobSet Operator | Providing JobSet controller | Trainer component | Optional - Install from OperatorHub for Trainer component integration |

ODH Component Operators (Managed by ODH)

The following components are automatically integrated by the ODH operator based on your DataScienceCluster configuration:

| Component | Operator/Repository | Purpose | Management State | |-----------|-------------------|---------|------------------| | KServe | opendatahub-io/kserve | Model serving platform | Optional | | Ray | opendatahub-io/kuberay | Distributed computing framework | Optional | | Trainer | opendatahub-io/trainer | ML training job management | Optional | | Feast Operator | opendatahub-io/feast | Feature store for ML | Optional | | Model Registry Operator | opendatahub-io/model-registry | Model versioning and registry | Optional | | TrustyAI | opendatahub-io/trustyai-service-operator | AI explainability and governance | Optional | | Dashboard | opendatahub-io/odh-dashboard | Web UI for ODH management | Optional | | Workbenches | opendatahub-io/notebooks | Jupyter notebook environments | Optional | | AI Pipelines | opendatahub-io/data-science-pipelines-operator | ML pipeline orchestration | Optional |

Note: These component controller do not need to be installed separately.

Optional Components

• GPU Support: For GPU workloads and metrics:

• Monitoring Stack:

Namespace Requirements

• One namespace will be designated as the applicationsNamespace in DSCI (default: opendatahub)
• Only one namespace in the cluster can have the label opendatahub.io/application-namespace: true if user prefer overwrite default namespace
• The operator itself runs in a separate namespace (default: opendatahub-operator in ODH)

Resource Requirements

• Sufficient cluster resources to run enabled components
• For production deployments, ensure adequate CPU, memory, and storage based on your workload requirements
• GPU nodes (if using GPU-accelerated workloads)

Installation

• The latest version of operator can be installed from the community-operators catalog on OperatorHub.

Please note that the latest releases are made in the Fast channel.

• It can also be built and installed from source manually, see the Developer guide for further instructions.

cat <<EOF | oc create -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: opendatahub-operator
        namespace: openshift-operators
      spec:
        channel: fast
        name: opendatahub-operator
        source: community-operators
        sourceNamespace: openshift-marketplace
      EOF

2. Create DSCInitialization CR manually. You can also use operator to create default DSCI CR by removing env variable DISABLE_DSC_CONFIG from CSV or changing the value to "false", followed by restarting the operator pod.

3. Create DataScienceCluster CR to enable components

Configuration

ODH operator can be configured both through flags and environment variables, here a list of the available one:

| Env variable | Corresponding flag | Description | Default value | |------------------------------------------------------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------| | ODH_MANAGER_METRICS_BIND_ADDRESS | --metrics-bind-address | The address the metric endpoint binds to. | :8080 | | ODH_MANAGER_HEALTH_PROBE_BIND_ADDRESS | --health-probe-bind-address | The address the probe endpoint binds to. | :8081 | | ODH_MANAGER_LEADER_ELECT | --leader-elect | Enable leader election for controller manager. | false | | ODH_MANAGER_LOG_MODE | --log-mode | Log mode ('', prod, devel), default to ''. See Log mode values for details. | | | ODH_MANAGER_PPROF_BIND_ADDRESS or PPROF_BIND_ADDRESS | --pprof-bind-address | The address that pprof binds to. | | | ZAP_DEVEL | --zap-devel | Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn)
Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) | false | | ZAP_ENCODER | --zap-encoder | Zap log encoding (one of 'json' or 'console') | | | ZAP_LOG_LEVEL | --zap-log-level | Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error' | info | | ZAP_STACKTRACE_LEVEL | --zap-stacktrace-level | Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic'). | | | ZAP_TIME_ENCODING | --zap-time-encoding | Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). | | | OPERATOR_NAMESPACE | --operator-namespace | The namespace the operator is deployed in. Falls back to in-cluster service account namespace. | (auto-detected) | | DISABLE_DSC_CONFIG | --disable-dsc-config | Disable automatic creation of default DSCInitialization CR. | false | | DEFAULT_MANIFESTS_PATH | --default-manifests-path | Base path for component manifests. | | | ODH_PLATFORM_TYPE | --platform-type | Platform type override (OpenDataHub, ManagedRHOAI, SelfManagedRHOAI, XKS). Auto-detects if empty. | (auto-detected) |

If both env variables and flags are set for the same configuration, flags values will be used.

Log mode values

| log-mode | zap-stacktrace-level | zap-log-level | zap-encoder | Comments | |-------------|----------------------|---------------|-------------|-----------------------------------------------| | devel | WARN | INFO | Console | lowest level, using epoch time | | development | WARN | INFO | Console | same as devel | | "" | ERROR | INFO | JSON | default option | | prod | ERROR | INFO | JSON | highest level, using human readable timestamp | | production | ERROR | INFO | JSON | same as prod |

Use custom application namespace

• For new cluster (i.e. a cluster that has not been used for ODH or RHOAI), using "namespace A" as an example of the targeted application namespace. Follow below steps before install ODH operator:

• For cases in which ODH is already running in the cluster:

Use custom workbench namespace

The workbench namespace is configurable. By default, workbenches are deployed to the opendatahub application namespace (or rhods-notebooks for downstream). You can configure a custom workbench namespace when enabling the workbenches component.

Important notes:

• The workbenchNamespace field is immutable - it can only be set once when workbenches are first enabled
• The namespace must follow Kubernetes naming conventions (lowercase alphanumeric characters or '-', max 63 characters)

In your DataScienceCluster CR, specify the workbenchNamespace field:

apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
  name: default-dsc
spec:
  components:
    workbenches:
      managementState: Managed
      workbenchNamespace: my-custom-workbench-namespace

Developer Guide

Pre-requisites

• Go version go1.24
• operator-sdk version can be updated to v1.37.0

Download manifests

The manifest-tools download command (in cmd/manifest-tools/) fetches manifests from remote git repositories. It reads manifests-config.yaml for component definitions and their manifest locations.

Structure of manifests-config.yaml

Each component is defined under the components section with per-platform (odh/rhoai) entries containing repo, ref, and sourcePath fields.

Workflow

1. The tool shallow-clones each component’s repository at the specified ref (supports branch@sha tracking format).
2. It copies the content from sourcePath to the local opt/manifests// folder.

Local Storage

The tool uses opt/manifests and opt/charts folders to host all required manifests, sourced from each component’s repository.

Adding New Components

To include a new component, add a new entry in the components section of manifests-config.yaml.

Customizing Manifests Source

You can override a component’s source using the --component flag:

make get-manifests
or with overrides:
go run -C ./cmd/manifest-tools main.go download --component dashboard=maistra:odh-dashboard:test-manifests:manifests

If the component key does not exist in manifests-config.yaml, the command will fail.

##### for local development

make get-manifests

This first cleanup your local opt/manifests folder. Ensure back up before run this command if you have local changes of manifests want to reuse later.

##### for build operator image

make image-build

By default, building an image without any local changes(as a clean build) This is what the production build system is doing.

In order to build an image with local opt/manifests folder set USE_LOCAL make variable to true e.g make image-build USE_LOCAL=true"

Build Image

• Custom operator image can be built using your local repository

make image IMG=quay.io/<username>/opendatahub-operator:<custom-tag>

The default image used is quay.io/opendatahub/opendatahub-operator:dev-0.0.1 when not supply argument for make image

• To build multi-arch image, set environment variable PLATFORM

export PLATFORM=linux/amd64,linux/arm64,linux/ppc64le,linux/s390x
  make image

• Once the image is created, the operator can be deployed either directly, or through OLM. For each deployment method a

export KUBECONFIG=<path to kubeconfig>

• The operator can be built in ODH or RHOAI mode (build defaults to ODH mode); to build in RHOAI mode, use

make image ODH_PLATFORM_TYPE=rhoai

Deployment

##### Deployment Methods Comparison

| Method | Use Case | OLM Required | Appears in OpenShift Console | |--------|----------|--------------|------------------------------| | OLM Deployment | Production-like, catalog-based | ✅ Yes | ✅ Yes (Operators section) | | Direct Deployment (make deploy) | Development, testing | ❌ No | ❌ No |

Deploying operator locally

• Define operator namespace

export OPERATOR_NAMESPACE=<namespace-to-install-operator>

• Deploy the created image in your cluster using following command:

make deploy IMG=quay.io/<username>/opendatahub-operator:<custom-tag> OPERATOR_NAMESPACE=<namespace-to-install-operator>

• To remove resources created during installation use:

make undeploy

• make deploy auto-applies digest-pinned image overrides from manifests-config.yaml to config/manager/manager.yaml (or config/rhoai/manager/manager.yaml when ODH_PLATFORM_TYPE=rhoai). To skip: SKIP_IMAGE_OVERRIDES=1 make deploy.

Image Overrides

Component operator images can be pinned by digest for reproducible testing. Image digests are configured in manifests-config.yaml under the imageOverrides section. The top-level buildConfig section pins the exact ODH-Build-Config and RHOAI-Build-Config commits used as release image sources. ODH overrides resolve from the ODH CSV (with params.env as an ODH-only source); RHOAI overrides resolve independently from the RHOAI CSV.

For e2e clusters, registry.redhat.io/rhoai/ CSV references are rewritten to their quay.io/rhoai/ mirrors while retaining the production digest. Other registry.redhat.io namespaces are not rewritten. hack/update-rhai-images.sh remains a local helper for updating downloaded params.env files and is not part of the e2e override flow.

Note: For OLM deployments, image overrides are applied after operator-sdk run bundle creates the Subscription. This is safe because the DSC starts with all components set to Removed, so no component images are pulled before the overrides are applied.

# Resolve digests (run after updating manifest SHAs)
make resolve-image-digests
For make deploy (auto-applied, skip with SKIP_IMAGE_OVERRIDES=1)
make deploy IMG=...
For OLM deploy (manual)
operator-sdk run bundle ...
make apply-image-overrides-olm
For E2E with OLM overrides (auto-applied, skip with SKIP_IMAGE_OVERRIDES=1)
make e2e-test

Deploying operator using OLM

Detailed Deployement instructions

• OpenDataHub and RHOAI OLM Deployment

• To create a new bundle in defined operator namespace, run following command:

export OPERATOR_NAMESPACE=<namespace-to-install-operator>
  make bundle

Note : Skip the above step if you want to run the existing operator bundle.

• Build Bundle Image:

make bundle-build bundle-push BUNDLE_IMG=quay.io/<username>/opendatahub-operator-bundle:<VERSION>

• RHOAI bundle can be built using ODH_PLATFORM_TYPE=rhoai with any of the bundle make targets.

• Run the Bundle on a cluster:

operator-sdk run bundle quay.io/<username>/opendatahub-operator-bundle:<VERSION> --namespace $OPERATOR_NAMESPACE --decompression-image quay.io/project-codeflare/busybox:1.36

• Understanding Catalog Generation:

Important Note: Users must provide the old version bundle images as a comma-separated list, in ascending order, to generate an upgradeable catalog image. For example:

make catalog-build catalog-push -e CATALOG_IMG=quay.io/<username>/opendatahub-operator-index:<target_version> \
    BUNDLE_IMGS=quay.io/<username>/opendatahub-operator-bundle:v2.26.0,\
    quay.io/<username>/opendatahub-operator-bundle:v2.27.0,\
    quay.io/<username>/opendatahub-operator-bundle:v2.28.0

This creates the following upgrade path:

v2.26.0 -> v2.27.0 -> v2.28.0 -> target_version

For more details on the File-Based Catalog format, see the FBC documentation.

• Build Catalog Image:

make catalog-build catalog-push -e CATALOG_IMG=quay.io/<username>/opendatahub-operator-index:<target_version> BUNDLE_IMGS=<list-of-comma-separated-bundle-images>

Cloud Manager (CCM)

The Cloud Manager (CCM) is a separate controller that reconciles KubernetesEngine resources for supported cloud providers. It deploys configured chart dependencies and reports their health in the KubernetesEngine status.

For new xKS dependencies managed by CCM, use the chart dependency onboarding checklist.

Supported Providers

| Provider | CRD | Description | |----------|-----|-------------| | AWS | AWSKubernetesEngine | Reconciles dependencies for AWS clusters | | Azure | AzureKubernetesEngine | Reconciles dependencies for Azure AKS clusters | | CoreWeave | CoreWeaveKubernetesEngine | Reconciles dependencies for CoreWeave clusters |

CCM can manage chart dependencies such as Gateway API, LeaderWorkerSet (LWS), Sail Operator, and RHCL. For xKS Helm installations, cert-manager is a Helm subchart of rhai-on-xks-chart rather than a CCM dependency.

CCM Deployment

Build the cloud manager binary:

make build-ccm

Deploy a cloud manager to the cluster (e.g., Azure):

make deploy-ccm-azure IMG=quay.io/<username>/opendatahub-operator:<custom-tag>

Deploy with local image pull policy (for development):

make deploy-ccm-local-azure IMG=quay.io/<username>/opendatahub-operator:<custom-tag>

Run a cloud manager locally:

make run-ccm-azure

Install only the CRDs:

make install-ccm-azure

Undeploy / Uninstall:

make undeploy-ccm-azure
make uninstall-ccm-azure

Replace azure in these targets with another provider listed under Supported Providers.

CCM Configuration

The cloud manager is configured via environment variables or command-line flags:

| Environment Variable | Flag | Description | |---------------------|------|-------------| | RHAI_OPERATOR_NAMESPACE | --rhai-operator-namespace | The namespace where the operator is deployed (required) | | DEFAULT_CHARTS_PATH | --default-charts-path | The base directory for locally-bundled Helm charts (required) |

When running locally via make run-ccm-, DEFAULT_CHARTS_PATH defaults to opt/charts. In production deployments, it is set to /opt/charts in the manager manifests.

Example AzureKubernetesEngine CR:

apiVersion: infrastructure.opendatahub.io/v1alpha1
kind: AzureKubernetesEngine
metadata:
  name: default-azurekubernetesengine
spec:
  dependencies:
    gatewayAPI:
      managementPolicy: Managed
    lws:
      managementPolicy: Managed
    sailOperator:
      managementPolicy: Managed

Example CoreWeaveKubernetesEngine CR:

apiVersion: infrastructure.opendatahub.io/v1alpha1
kind: CoreWeaveKubernetesEngine
metadata:
  name: default-coreweavekubernetesengine
spec:
  dependencies:
    gatewayAPI:
      managementPolicy: Managed
    lws:
      managementPolicy: Managed
    sailOperator:
      managementPolicy: Managed

RHAII Mode

RHAII (Red Hat AI Inference) is a deployment mode that runs a subset of the operator focused exclusively on KServe. This is useful when you only need model serving capabilities without the full Open Data Hub stack.

In RHAII mode, the operator deploys only the KServe component CRD and its associated webhooks (connection-isvc and connection-llmisvc mutation webhooks).

Prerequisites

RHAII mode requires cert-manager to be available in the cluster. In an XKS deployment, the XKS chart manages cert-manager installation. When deploying the Cloud Manager outside that chart, install cert-manager before deploying the RHAII operator.

Supported Providers

Tests are also performed with a KinD instance.

| Provider | Description | |----------|-------------| | Azure | Azure Kubernetes Engine cloud provider | | CoreWeave | CoreWeave Kubernetes Engine cloud provider |

RHAII Deployment

Deploy in RHAII mode:

make deploy-rhaii IMG=quay.io/<username>/opendatahub-operator:<custom-tag>

Deploy with local image pull policy (for development):

make deploy-rhaii-local IMG=quay.io/<username>/opendatahub-operator:<custom-tag>

Undeploy:

make undeploy-rhaii

Example KServe CR for RHAII:

apiVersion: components.platform.opendatahub.io/v1alpha1
kind: Kserve
metadata:
  name: default-kserve
spec: {}

WVA compatibility note:

The spec.wva field is deprecated and preserved only for backward compatibility. Workload Variant Autoscaler (WVA) is no longer supported, and the operator treats WVA as removed regardless of whether an existing resource previously set it to Managed.

Test with customized manifests

There are 2 ways to test your changes with modification:

1. Using custom manifests in OLM operator. See custom manifest for more details.

1. Build operator image with local manifests, running make image-build USE_LOCAL=true

Update API docs

Whenever a new api is added or a new field is added to the CRD, please make sure to run the command:

make api-docs

Change logging level at runtime

Log level can be changed at runtime by DSCI devFlags by setting .spec.devFlags.logLevel. It accepts the same values as --zap-log-level flag or ZAP_LOG_LEVEL env variable. See example :

apiVersion: dscinitialization.opendatahub.io/v2
kind: DSCInitialization
metadata:
  name: default-dsci
spec:
  devFlags:
    logLevel: debug
  ...

Example DSCInitialization

1. Default DSCI configuration

kind: DSCInitialization
apiVersion: dscinitialization.opendatahub.io/v2
metadata:
  name: default-dsci
spec:
  applicationsNamespace: opendatahub
  monitoring:
    managementState: Managed
    namespace: opendatahub
    metrics:
      replicas: 2
      storage:
        retention: 90d
        size: 5Gi
    traces:
      storage:
        backend: pv
        size: 5Gi
        retention: 2160h
  trustedCABundle:
    customCABundle: ''
    managementState: Managed

1. DSCI with custom application and monitoring namespace

kind: DSCInitialization
apiVersion: dscinitialization.opendatahub.io/v2
metadata:
  name: default-dsci
spec:
  applicationsNamespace: my-custom-namespace
  monitoring:
    managementState: Managed
    namespace: my-custom-namespace
  trustedCABundle:
    managementState: Managed

Note: Before applying DSCI with a custom application namespace, ensure you have created the namespace and labeled it with opendatahub.io/application-namespace: true. See Use custom application namespace for complete setup instructions.

Apply these examples with modifications for your usage.

Example DataScienceCluster

When the operator is installed successfully in the cluster, a user can create a DataScienceCluster CR to enable ODH components. At a given time, ODH supports only one instance of the CR, which can be updated to get custom list of components.

1. Enable all components

apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
  name: default-dsc
spec:
  components:
    aigateway:
      managementState: "Managed"
      batchGateway:
        managementState: "Managed"
    dashboard:
      managementState: Managed
    aipipelines:
      managementState: Managed
    kserve:
      managementState: Managed
      nim:
        managementState: Managed
      rawDeploymentServiceConfig: Headed
    kueue:
      managementState: Removed
    modelregistry:
      managementState: Managed
    ray:
      managementState: Managed
    trainer:
      managementState: Managed
    trustyai:
      managementState: Managed
    workbenches:
      managementState: Managed
    feastoperator:
      managementState: Managed
    llamastackoperator:
      managementState: Removed
    ogx:
      managementState: Removed
    mlflowoperator:
      managementState: Managed
    sparkoperator:
      managementState: Managed
    mcplifecycleoperator:
      managementState: Removed

The mlflowoperator DSC field still acts as the user-facing on/off switch, but internally it now drives the MLflow module handler flow rather than the legacy in-tree component reconciler.

1. Enable only Dashboard and Workbenches

apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
  name: example
spec:
  components:
    dashboard:
      managementState: Managed
    workbenches:
      managementState: Managed

1. Enable Workbenches with custom namespace

apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
  name: example
spec:
  components:
    workbenches:
      managementState: Managed
      workbenchNamespace: my-workbench-namespace

Note: The workbenchNamespace field once set, it cannot be changed (immutable).

Example GatewayConfig

The GatewayConfig custom resource is used to configure gateway settings for OpenDataHub, including OIDC authentication and ingress gateway configuration. This CR is cluster-scoped and must be named default-gateway.

Automatic Creation: The GatewayConfig CR is automatically created when a DSCInitialization CR is applied to the cluster. Users typically don't need to create this CR manually unless they want to configure OIDC authentication mode or customize ingress gateway settings.

In OcpRoute mode, the operator applies a 60s timeout to the default shared Gateway Route when the default IngressController's server timeout is shorter. Administrators can override this through spec.ocpRoute.serverTimeout. See Configure the shared Gateway Route timeout for examples and how this setting interacts with component HTTPRoute timeouts.

For legacy Dashboard and gateway URLs, see Route cleanup and legacy URL redirects, including the RHOAI 3.6+ deprecation page and how to disable or re-enable redirects through GatewayConfig.

Here's an example of the default GatewayConfig CR (automatically created):

apiVersion: services.platform.opendatahub.io/v1alpha1
kind: GatewayConfig
metadata:
  name: default-gateway
spec:
  cookie: {}  # Uses defaults: expire: 24h, refresh: 1h
  certificate:
    type: OpenshiftDefaultIngress

For a setup with custom certificate (Provided type):

apiVersion: services.platform.opendatahub.io/v1alpha1
kind: GatewayConfig
metadata:
  name: default-gateway
spec:
  cookie: {}  # Uses defaults: expire: 24h, refresh: 1h
  domain: "*.apps.example.com"
  certificate:
    type: Provided
    secretName: custom-tls-cert  # Secret must already exist in openshift-ingress namespace with tls.crt and tls.key

For an advanced example with OIDC authentication:

apiVersion: services.platform.opendatahub.io/v1alpha1
kind: GatewayConfig
metadata:
  name: default-gateway
spec:
  oidc:
    issuerURL: "https://keycloak.example.com/auth/realms/opendatahub"
    clientID: "opendatahub-client"
    clientSecretRef:
      name: oidc-client-secret
      key: client-secret
  cookie:
    expire: 24h  # Default: 24h
    refresh: 1h  # Default: 1h
  domain: "*.apps.example.com"
  authTimeout: 10s  # Default: 5s
  certificate:
    type: SelfSigned

For an advanced example using cluster domain with custom subdomain:

apiVersion: services.platform.opendatahub.io/v1alpha1
kind: GatewayConfig
metadata:
  name: default-gateway
spec:
  cookie: {}  # Uses defaults
  domain: apps.example.com
  certificate:
    type: SelfSigned

This will create a gateway rh-ai.apps.example.com (using default subdomain)

For an advanced example with custom subdomain:

apiVersion: services.platform.opendatahub.io/v1alpha1
kind: GatewayConfig
metadata:
  name: default-gateway
spec:
  cookie: {}  # Uses defaults
  domain: apps.cluster.example.com
  subdomain: custom-gateway
  certificate:
    type: SelfSigned

This will use the cluster's default domain with your custom subdomain: custom-gateway.apps.cluster.example.com

The operator creates a NetworkPolicy whenever it deploys kube-auth-proxy. On upgrade, a previously stored spec.networkPolicy.ingress.enabled: false no longer suppresses policy creation; the upgraded controller creates the policy during reconciliation. The stored GatewayConfig needs no manual migration, but remove the obsolete spec.networkPolicy field from manifests before applying them again.

Important Notes:

• The GatewayConfig name must be exactly default-gateway
• This is a cluster-scoped resource
• Automatic creation: This CR is automatically created after DSCInitialization CR is applied
• Manual configuration needed: Only configure this CR manually if you want to enable OIDC authentication mode or customize ingress gateway settings
• OIDC configuration is optional and only needed when cluster is in OIDC authentication mode
• Certificate types can be OpenshiftDefaultIngress, SelfSigned, or Provided
• If subdomain is not specified or is empty, the default value rh-ai is used.
• If domain is not specified, the cluster's default domain is used.
• NetworkPolicy is enabled by default to secure kube-auth-proxy traffic. It allows only Gateway pods to reach the authentication port (TCP 8443). kube-auth-proxy serves metrics on TCP 9000, but nothing scrapes that port today: the OpenShift monitoring stack only scrapes what a PodMonitor tells it to scrape (defaulting to port 8080), and kube-auth-proxy has no PodMonitor. The monitoring ingress rules are therefore omitted to keep the policy least-privileged; they can be reintroduced together with a PodMonitor when kube-auth-proxy becomes a metrics scrape target.

Run functional Tests

The functional tests are writted based on ginkgo and gomega. In order to run the tests, the user needs to setup the envtest which provides a mocked kubernetes cluster. A detailed explanation on how to configure envtest is provided here.

To run the test on individual controllers, change directory into the contorller's folder and run

ginkgo -v

This provides detailed logs of the test spec.

Note: When running tests for each controller, make sure to add the BinaryAssetsDirectory attribute in the envtest.Environment in the suite_test.go file. The value should point to the path where the envtest binaries are installed.

In order to run tests for all the controllers, we can use the make command

make unit-test

Run e2e Tests

A user can run the e2e tests in the same namespace as the operator. To deploy opendatahub-operator refer to this section. The following environment variables must be set when running locally:

export KUBECONFIG=/path/to/kubeconfig

Ensure when testing RHODS operator in dev mode, no ODH CSV exists Once the above variables are set, run the following:

make e2e-test

Configuring e2e Tests

| Configuration Env var | Description | Default value | |------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------| | E2E_TEST_OPERATOR_NAMESPACE | Namespace where the ODH operator is deployed. | opendatahub-operator-system | | E2E_TEST_APPLICATIONS_NAMESPACE | Namespace where the ODH applications are deployed. | opendatahub | | E2E_TEST_WORKBENCHES_NAMESPACE | Namespace where the workbenches are deployed. | opendatahub | | E2E_TEST_DSC_MONITORING_NAMESPACE | Namespace where the ODH monitoring is deployed. | opendatahub | | E2E_TEST_DEPENDANT_OPERATORS_MANAGEMENT | To configure the execution of tests related to the Dependant Operators management, this is useful to run e2e tests for an operator running out of the cluster i.e. for debugging purposes | true | | E2E_TEST_DSC_MANAGEMENT | To configure the execution of DSCI/DSC management tests, this is useful to run e2e tests for an operator running out of the cluster i.e. for debugging purposes | true | | E2E_TEST_DSC_VALIDATION | To configure the execution of DSCI/DSC validation tests, this is useful to run e2e tests for an operator running in xKS cluster | true | | E2E_TEST_OPERATOR_CONTROLLER | To configure the execution of tests related to the Operator POD, this is useful to run e2e tests for an operator running out of the cluster i.e. for debugging purposes | true | | E2E_TEST_OPERATOR_RESILIENCE | To configure the execution of operator resilience tests, useful for testing operator fault tolerance scenarios | true | | E2E_TEST_WEBHOOK | To configure the execution of tests related to the Operator WebHooks, this is useful to run e2e tests for an operator running out of the cluster i.e. for debugging purposes | true | | E2E_TEST_CONVERSION_WEBHOOK | Run conversion webhook scenarios. Defaults to true in full and targeted runs; set to false to disable them. DSC scenarios delete default-dsc. | true | | E2E_TEST_CONVERSION_WEBHOOK_DSC | Include DSC conversion scenarios when the conversion suite runs. Set false to run only Platform conversion scenarios. | true | | E2E_TEST_CONVERSION_WEBHOOK_PLATFORM | Include Platform conversion scenarios when the conversion suite runs. Set false to run only DSC conversion scenarios. | true | | E2E_TEST_DELETION_POLICY | Specify when to delete DataScienceCluster, DSCInitialization, and controllers. Valid options are: always, on-failure, and never. | always | | E2E_TEST_COMPONENTS | Enable testing of individual components specified by --test-component flag | true | | E2E_TEST_COMPONENT | A space separated configuration to control which component should be tested, by default all component specific test are executed | all components | | E2E_TEST_SERVICES | Enable testing of individual services specified by --test-service flag | true | | E2E_TEST_SERVICE | A space separated configuration to control which services should be tested, by default all service specific test are executed | all services | | E2E_AUTO_RESOLVE | When true, make e2e-test derives E2E_TEST_COMPONENT/E2E_TEST_SERVICE automatically from the PR's changed files (see Path-based test scoping), instead of running the full suite. The resolver always runs and logs its decision regardless of this flag, so its accuracy can be checked in job logs before enabling it. | false | | E2E_TEST_OPERATOR_V2TOV3UPGRADE | To configure the execution of V2 to V3 upgrade tests, useful for testing V2 to V3 upgrade scenarios | true | | E2E_TEST_CLEAN_UP_PREVIOUS_RESOURCES | To configure the cleaning-up of the previous resources in the cluster. This flag is quite useful to test custom scenarios, as well as running the tests after upgrade, to keep the previous DSC and test it. | true | | E2E_TEST_BACKUP_AND_RESTORE_DSCI_AND_DSC | To back up DSCI/DSC at the beginning of the whole test suite and restore them at the end. Useful for testing custom scenarios, including removal of DSCI/DSC during the tests while keeping cluster in the original shape. | false | | E2E_TEST_CIRCUIT_BREAKER | Enable circuit breaker to halt tests on repeated infrastructure failures. | true | | E2E_TEST_CIRCUIT_BREAKER_THRESHOLD | Consecutive infrastructure failures before health-checking for infrastructure problems. | 3 | | E2E_TEST_TAG | Tag to run tests for. Options: All, Smoke, Tier1. | All | | E2E_TEST_KUEUE_OPERATOR_CHANNEL | Kueue subscription channel. | stable-v1.5 | | E2E_TEST_TIMEOUT | Timeout for the containerized E2E test binary, including conversion webhook scenarios. | 110m | | E2E_TEST_FLAGS | Alternatively the above configurations can be passed to e2e-tests as flags using this env var (see flags table below) | | | SKIP_IMAGE_OVERRIDES | Skip auto-applying digest-pinned image overrides during make deploy and make e2e-test. Set to any value to disable. | (unset) |

Alternatively the above configurations can be passed to e2e-tests as flags by setting up E2E_TEST_FLAGS variable. Following table lists all the available flags:

| Flag | Description | Default value | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------| | --operator-namespace | Namespace where the ODH operator is deployed. | opendatahub-operator-system | | --applications-namespace | Namespace where the ODH applications are deployed. | opendatahub | | --workbenches-namespace | Namespace where the workbenches are deployed.

... (README truncated for length)