OpenShift Operator

Operators are controllers which implement and manage Custom Resources with custom reconciliation logic. Operators extend Kubernetes to manage custom resources similar as Kubernetes built-in resources. Operators are of two types, Cluster Operators and OLM Operators. The Cluster Operators (Platform Operators) are used to manage the OpenShift platform itself such as managing the cluster and upgrading cluster etc. The OLM Operators are managed by Operator Lifecycle Manager (OLM).

Operators can be developed using Helm, Go and Ansible. Helm operator only support Level 2 maturity level with capabilities Basic Install and Seamless. The Helm-based operator wraps existing Helm chart into Operator for seamless installation/upgrades. It creates a simple CRD containing the desirable parameters to be passed to Helm Chart as part of the spec for custom resource. We can configure the OpenShift platform versions supported by the Operator.

Operator Lifecycle Manager (OLM)

Operator Lifecycle Manager (OLM) is Kubernetes native application and it helps to install and manage the life cycle of the operators. OLM allows to Package, Distribute and Manage Operators.
Kubernetes Operators are application specific controllers that automate the deployment, configuration and management of complex applications on Kubernetes. OLM provides a declarative way to install and manage Operators using Kubernetes cluster resource definitions (CRDs). OLM is installed by default in OpenShift cluster.

OLM is comprised of the catalog-operator, olm-operator and packageserver. The status of OLM pods can be checked using the command oc get pods -n openshift-operator-lifecycle-manager.

• OLM Operator is responsible for deploying applications defined by CSV resources after the required resources specified in the CSV are present in the cluster. It watches for the ClusterServiceVersion (CSVs) to appear in the watched namespaces. It checks whether all the owned and required CRDs specified in CSV are available in the cluster and deploys the Operator once all CSV resources are available.

• Catalog Operator is responsible for resolving and installing CSVs and the required resources they specify. It is also responsible for watching CatalogSources for updates to packages in channels and upgrading them (optionally automatically) to the latest available versions. Catalog Operator installs CSV and all the resources specified in CSV to the cluster, so that OLM Operator can then deploy the Operator. The catalog operator updates the catalog sources and watches the channels of subscribed operators. It uses subscriptions to query catalogs for the best version of the Operator. Catalog Operator creates all of the resources in the Install Plan to build the required resources for new and upgraded CSV.

·       PackageServer is API provider which Serves the OLM APIs (e.g., via OperatorHub). It shows all the packages available across Catalogs. The below command lists all the Operators available to be installed from all the catalog sources configured to the cluster.

kubectl get packagemanifest

Custom Resource Definition

Custom Resource Definition (CRD) allows users to extend the Kubernetes API by defining new schema and structure for new custom resource types. This includes specifying its API group, kind, plural and singular names, and the fields within its specification (spec) and status. CRD provides the object definition of the parameters which will be passed to underlying Helm Chart of the Operator.

ClusterServiceVersion

ClusterServiceVersion (CSV) is a Kubernetes Custom Resource (CR) that defines the metadata, deployment details, and lifecycle management information of the Operator for the Operator Lifecycle Manager (OLM) to install/manage operator and its dependencies. CSV is used to install the operator, manage its lifecycle such as upgrade, uninstall, etc and display the information about it in the OpenShift Console. OLM reads the CSV to ensure the Operator version to install, the permissions required to install the Operator from a catalog and the CRD it needs to watch for Operator state. CSV contains the details of the CRDs, namely the owned CRDs of the Operator and the required CRDs for the Operator. The required CRDs needs to be installed in the cluster before installing the operator. Each new version of the operator will be a new CSV.

Install Strategy is another section of CSV which tells OLM operator what to provision to get the operator running, that is the OLM operator follows when all resources exists in the cluster.

The Operator Framework's suggested-namespace annotation in CSV is used to ensure that customer creates specified namespace during operator installation.

The InstallMode in CSV specifies scope of the Operator i.e which namespaces it can watch and manage custom resources (CRs) within the cluster. Install modes determine whether the Operator: runs in a single namespace or its own namespace, watches multiple namespaces, or manages resources cluster-wide. The InstallModes allows to declare supported deployment scopes, ensuring security boundaries and whether OLM can validate if installation is compatible with the chosen OperatorGroup.

The OpenShift operator supports OwnNamespace install mode and only runs on specific namespace. If another operand instance is created with an existing operand then the second operand gets installed after the current operand is uninstalled.

kubectl get csv -n emprovise-system

kubectl describe csv -n emprovise-system

Subscription

Subscription is an OLM resource which tells OpenShift, the operator to be installed or to be updated from a given catalog source and channel. It acts like a contract between the OperatorSource / CatalogSource (where Operator packages live), and the ClusterServiceVersion. OLM will be monitoring the channels when there is subscription present and new CSVs will be applied to keep the operator updated.

Once we register or subscribe to channel then OLM watches for the channel for new versions to be released. All subscriptions by default are configured for automatic approval. Update the installPlanApprovalfield in the Subscription to manual for manual approval to install/upgrade the operator. To manually approve, update the field approved to true in install plan for operator installation to proceed.

Subscription is way to tell OLM that we want to have operator from specific channel in a package within a catalog to be installed and maintained on the cluster.

kubectl get subscription  -n emprovise-system kube-demo-subscription -o yaml

kubectl describe subscription -n emprovise-system

CatalogSource

CatalogSource (Catalog Registry) is a library of operators which contains information about available operators, their versions and instructions to install them. It is a repository of metadata that OLM uses to discover and install Operator along with its dependencies. It is designed to automate the tedious tasks to install and update the operators. Catalog source is configured to pull an image every 30 minutes and pull down latest catalog image when the update becomes available. This keeps channel updated with latest CSVs.

kubectl get catalogsource -n emprovise-system kube-demo-catalog -o yaml

OperatorGroup

OperatorGroup defines the scope of visibility for one or more Operators by providing multitenancy configuration. OperatorGroup assigns target namespaces and generates required RBAC access for member Operators. It ensures there is no conflict in APIs supported by each member managed by the same group. There must be only single OperatorGroup created per namespace. The Operator should have the Install Mode configured in its CSV that supports the OperatorGroup's target namespaces in order to become the member of the OperatorGroup. Member CSVs of an OperatorGroup gets the olm group annotations in generated CSV for the bundle.

oc get operatorgroups -n emprovise-system -o yaml

Channels

Channel represents a stream of versions within an Operator Catalog that groups one or more versions of an Operator’s ClusterServiceVersions (CSVs) to represent a logical upgrade path. Each channel defines a sequence (or upgrade path) of Operator versions that OLM can follow when installing or upgrading an Operator.

KubeDemo Operator supports two channels stable which is default, and alpha channel in Operator Bundle/Catalog. OLM will be monitoring the channels when there is subscription present and new CSVs will be applied to keep the operator upto date.
Add catalog source to the cluster and have OLM can watch a specific channel in a package from the catalog (service) to detect and apply updates to our operator.

If new updates available in the channel which it is subscribed to OLM would automatically will create install plan which defines all the resources required to install/upgrade the new version of operator. Install Plans get created in same namespace as the operator when new version is available.

Install Plan

Install plan contain the details of all the resources which needs to be created in order to automatically install or upgrade a CSV. Install plans gets generated for each new CSV being applied to the cluster.

kubectl get installplan -n emprovise-system

kubectl describe installplan -n emprovise-system

Operator Controller

Operator Controller is the core execution engine of the Operator deployed on OpenShift cluster. It houses operational logic extending the Kubernetes API to manage complex stateful applications. It executes the Kubernetes Control Loop (or "reconciliation loop"), which continuously monitors the custom resources, comparing the desired state to the actual state and takes the remediating action. The Operator Controller’s reconciliation loop function is a continuous, three-step cycle:

·       Watch: It watches for changes (events) on the Custom Resources (CRs) defined by the Operator's Custom Resource Definition (CRD).

·       Analyze: When a change is detected (e.g., a user creates, updates, or deletes a CR), the Operator Controller compares the new desired state (defined in the CR) with the current actual state of the application in the cluster (e.g., existing Pods, Services, ConfigMaps).

·       Act (Reconcile): If a difference is found, the Operator Controller executes the necessary steps to reconcile the state. This usually involves issuing standard Kubernetes API calls to create, update, or delete underlying resources (like Deployments, StatefulSets, or Secrets) to match the user's defined goal.

Operator Controller is referred as Controller Manager in OpenShift Operator repository.

Metrics Service

Operator bundle resource exposes metrics endpoint (on port 8443) as service from the Controller manager which can be used by Prometheus or Kubernetes Metric Server for scraping metrics and monitoring operator health.

RBAC Permissions

RBAC (Role-Based Access Control) resources determine the access for Operator’s controller and the end user access (via OLM-generated roles) to the KubeDemo custom resource. OLM installs and manages these RBAC objects automatically when the ClusterServiceVersion (CSV) is applied from your Operator bundle.

The Operator SDK auto-generates RBAC roles for end users to inspect KubeDemo CRs safely, providing minimal read-only access to observe but not modify the KubeDemo custom resources.

SQLite Based Catalog

SQLite based catalog is the legacy catalog storage for operators. It was part of legacy implementation of catalog images and deprecated. It is NOT recommended to enabled SQLite based catalog by setting environment variable CATALOG_MODE to sqlite.

File Based Catalog

File-Based Catalog is the declarative YAML/JSON format replacing the SQLite database format in Catalog image which was hard to: maintain, version control, and review.

Each document defines a single catalog entry (like an Operator package, a version, or a channel). The catalog contains Operator version entries for each channel (alpha and stable channel), Operator bundle image URLs, and Custom Resource Definitions (CRDs) description and CRD example. These entries are built into an image (the catalog image) that OLM uses to discover and install Operators. The KubeDemo Operator by default uses File based catalog with environment variable CATALOG_MODE to fbc.

Catalog Template

Catalog templates are used to simplify a view of a catalog and allow easier manipulation of catalogs. They are reusable YAML “blueprints” that define how catalog entries (packages, channels, bundles) should be structured or generated. Catalog templates are of two types Basic Template and Semantic Template identified by a top-level key schema: olm.template.basic or schema: olm.template.semanticrespectively. Templates uses YAML parameters to simplify catalog authoring and reduce duplication. Template is similar to "macros that expand into actual catalog entries (packages, channels, bundles).

Basic Template

Basic template provides a simple and explicit way to define catalog entries (packages, channels, and bundles) — all in one file. It’s typically used for small or manually curated catalogs, or when you want direct control.

Semver Template

Semver template is an advanced version of the basic template that introduces semantic version awareness and automated upgrade graph generation. It’s designed for larger catalogs or Operators with multiple versions, where you want OLM to auto-generate channel graphs based on semantic version rules. This template supports channel names Candidate, Fast, and Stable, in order of increasing channel stability.

Since currently we don’t anticipate frequent Helm chart and Operator releases, prefer manually defined and managed channels with operator versions, KubeDemo Operator uses Basic template to generate FBC.

Catalog Basic Template Management

The Basic Template for File-based Catalog (FBC) is used to manage the upgrade graph of the Operator with version entries for supported channels alpha and stable.

The previous catalog/basic.yaml is either converted from previous catalog/catalog.yaml downloaded from previous catalog image for generate new catalog FBC. If no previous catalog exists then empty catalog file copied from config/catalog/basic-template to catalog/basic.yaml. The new operator bundle version entry is added for alpha channel (Default Fast Channel) along with olm.bundle Bundle image URL to the basic template. Finally the basic template is converted to full FBC catalog/catalog.yaml which is packaged in new catalog image.

The below custom scripts and Makefile commands are used to manage the Basic Template and to convert to full form of File-based catalog.

·       Convert catalog shell script converts full File-based catalog to basic template catalog. The make extract-catalog command leverages the /scripts/convert_catalog.sh script for basic template conversion.

./scripts/convert_catalog.sh catalog/catalog.yaml catalog/basic_template.yaml

·       Update the basic template catalog by adding new operator bundle version to the specified channel.

./scripts/update-catalog.sh catalog/basic.yaml quay.io/community-operator-pipeline-prod/kube-demo:0.0.6 alpha

·       Below command reads the basic catalog template file from input variable INPUT_CATALOG_TEMPLATE and finds the latest operator version for FAST_CHANNEL set to alpha channel.

make latest-version-from-catalog

·       The generate catalog command adds new operator entry (using update-catalog.sh) to the existing catalog/basic.yaml catalog basic template if present or to an empty basic template catalog copied from config/catalog/basic-template.yaml. It then generates the full form of file based catalog in catalog/catalog.yaml and validates the catalog. Finally it generates the Dockerfile to build catalog image.

make catalog-generate

Operator Images

The Operator image packages the KubeDemo Helm chart and the "watches.yaml" configuration in the opt/helm path to install the KubeDemo Helm chart.

The make bundle command generates the Operator Bundle files which has Manifests directory consisting of ClusterServiceVersion, Operator's CRD, RBAC permissions, Metadata directory which has annotations.yaml used to generate the Bundle Docker image and Tests directory containing scorecard test execution configuration. The Operator image and the Operator Bundle files are used to publish Community Operator in production.

The Bundle and Catalog docker images are only used for local operator testing or integration testing. We use the command oc apply -k config/olm to install the KubeDemo operator on OpenShift cluster. The Bundle image contains the Operator Bundle files generated from make bundle command added to the manifests/, metadata/ and /tests/scorecard respectively in the container.

The Catalog docker image contains the full FBC catalog generated added inside the /configs directory in the container.

Operator images only supports Linux ARM64 and Linux AMD64 systems.

Operator SDK

The OpenShift Operator project was initialized using the operator-sdk utility's initialize command.
The Operator SDK is toolkit part of Operator Framework which provides the tools to build, test, and package Operators. It supports below commands to create, delete and test Operators.

Create operator project directory and initialize the project.

operator-sdk init --domain emprovise.com --plugins helm

Create a simple Kube-Demo API using Helm’s built-in chart boilerplate (from helm create):

operator-sdk create api --group demo --version v1alpha1 --kind KubeDemo

The Operator SDK’s install operator command creates temporary CatalogSource and Subscription to install kube-demo operator similar to new customized kubectl apply -k config/olm command. It is used for local testing to run the operator bundle.

operator-sdk run bundle "${V1CS_OPERATOR_BASE_IMG_URL}/kube-demo-bundle:v${OPERATOR_VERSION}" --namespace emprovise-system

The operator upgrade command upgrades the installed operator by deleting the old subscription and creating new operator’s subscription:

operator-sdk run bundle-upgrade "${V1CS_OPERATOR_BASE_IMG_URL}/kube-demo-bundle:v${OPERATOR_VERSION}" --namespace emprovise-system

The Operator SDK’s cleanup command deletes the Operator Subscription, CatalogSource and ClusterServiceVersion. It is similar to executing kubectl delete -k config/olm which is used currently for more customized install/uninstall.

operator-sdk cleanup kube-demo --namespace emprovise-system

Operator Package Manager (OPM)

Operator Package Manager (OPM) is a CLI tool used to build, manage, and inspect Operator catalogs images used by OLM to list the available Operator version and upgrade path. OPM manages catalogs that list Operators and their bundles. OLM uses CatalogSources to discover Operators which are served from index images built using OPM. OPM supports Sqlite catalogs using opm index add, opm index inspect, opm serve commands and FBC catalogs using below commands.

Render and view manifests inside a bundle image.

opm render docker.io/emprovise/kube-demo-bundle:v0.0.1

Convert full FBC to Basic template catalog.

opm alpha convert-template basic -o yaml catalog/catalog.yaml -o yaml

Validate full FBC catalog.

opm validate ./catalog

Helm Idempotency

The Operator Helm chart should be Idempotent, i.e. every repeated install/update of same Helm chart should produce exact same resources. This is required to prevent the Controller Manager from repeatedly reconciling the Helm chart installation. The controller manager runs a dry run upgrade every few minutes, and if resources are found to have drifted (ie. secrets are randomly defined), then the controller manager will repeatedly reinstall Helm Chart.

Note: The mode to allow checking existing secrets using lookup uses the --dryrun=server option of Helm internally within the Operator controller. This is enabled as dryRunOption: server in openshift-operator/watches.yaml

 

OpenShift Operator Community Release Process

Community Operator Release requires to create a forked repository of Community Operators Production which will be used to create community operator pull requests.

Community Operator Release with Auto-Release FBC

·       Execute the OpenShift Operator Release workflow from actions repository by Manually triggering the workflow using Run workflow and specifying the New Operator release versions. This will create a new Operator image (along with Bundle and Catalog images which are used for internal testing) pushed into kube-demo Integration ECR and downloadable Operator Bundle artifact (Manifests, Annotations and Tests directories) from the workflow.

·       Create new directory named as new operator version number e.g. 0.0.2 in the forked Community Operators Production repository inside the operators/kube-demo directory and copy all the Operator Bundle artifact files as its contents. Also add the release-config.yaml file with the replaces field containing the previous (existing) operator version e.g. v0.0.1 as below. The release-config.yamlautomates the catalog update using the File-Based-Catalog auto-release and avoids manually creating a second PR with catalog changes.

---

catalog_templates:

  - template_name: basic.yaml

    channels: [stable, alpha]

    replaces: kube-demo.v0.0.1

·       The release-config.yaml for auto-release of File-Based-Catalog using basic template not just supports replacing version but can also specify the list of versions or range of versions to skip with below fields.

o   replaces - the bundle that the new bundle replaces in the update graph.

o   skips - a list of bundles that should be skipped in the update graph.

o   skipRange - a range of bundles that should be skipped in the update graph.

·       Push the above new operator directory changes as commit to the forked community operator prod repository on new branch e.g. kube-demo-0.0.2. Create signed commit using the command git commit -S -m "release operator kube-demo (0.0.2)" --author=<name_email> for DCO check. Create a new Pull request in the Community Operators Production repository using the forked community operator changes.

·       Once the Community Operator Prod pull request is created, it triggers the operator-hosted-pipeline which executes all validation checks such as pre-flight checks and other static-tests. On successful completion of validation checks/tests the pull request automatically gets merged if it is authorized by approved reviewers list in CI yaml. This will trigger the operator-release-pipeline workflow which creates the operator bundle image using the provided bundle files for the new operator version. The merged pull request will have the newly release bundle image URL in Release Info as below.

·       It triggers to automatically create new Pull Request for Catalog update which will update the basic.yaml file containing the operator’s basic template catalog entries. The Community Operator release workflow then adds/sync index for new Operator bundle version to the existing Community Operator Catalog.

NOTE: Once the Operator version (manifest files) is added to Community-Operators-Prod repository we cannot make any changes for existing operator version. If you want to update bundle files for existing Operator version, you first need to delete the operator and update the corresponding catalog entries using below make catalogs command. Then add the same operator with updated bundle files similar to adding new operator version.

Community Operator Release with Manual-Release FBC

·       Manual-Release FBC for community operator is used when we have to:

o   Add existing operator version to another channel (alpha or stable).

o   The operator-hosted-pipeline workflow for automated Catalog update pull request fails, in which case we manually need to update the basic template and full FBC catalogs (for v4.12 to v4.20).

o   Delete the released operator version which requires updating the catalogs.

·       Execute the OpenShift Operator Release workflow passing the new operator version which publishes the operator image in public ECR and outputs bundle artifact from the workflow.

·       Create new directory named as new operator version number e.g. 0.0.2 in the forked Community Operators Production repository inside the operators/kube-demo directory and copy all the Operator Bundle artifact files as its contents.

·       Push the above new operator directory changes as commit to the forked community operator prod repository on new branch, using signed commit for e.g. git commit -S -m "release operator kube-demo (0.0.2)" --author=<name_email>.

·       Create a new Pull request in the Community Operators Production repository using the above forked community operator changes. After the pull request is merged we get the newly released bundle image URL in Release Info from the Merged Operator Pull Request.

·       Manually add the catalog entry for the desired channel (namely alpha and/or stable) and olm.bundle schema entry with the bundle image URL from the previous pipeline execution in basic.yaml in catalog-templates. For example in the below snippet of catalog basic template, we add entry to alpha channel with replaces kube-demo.v0.0.1 for name kube-demo.v0.0.2. We also add newly released bundle image URL quay.io/community-operator-pipeline-prod/kube-demo:0.0.2 for olm.bundle schema entry as below.

---

schema: olm.template.basic

entries:

  - schema: olm.channel

    name: alpha

    package: kube-demo

    entries:

      - name: kube-demo.v0.0.1

      - name: kube-demo.v0.0.2

        replaces: kube-demo.v0.0.1

  - schema: olm.bundle

    image: quay.io/community-operator-pipeline-prod/kube-demo:0.0.1

  - schema: olm.bundle

    image: quay.io/community-operator-pipeline-prod/kube-demo:0.0.2

·       Update the full FBC catalogs for kube-demo in catalogs directory, by generating full FBC for all the supported OpenShift versions (v4.12-v4.20) using the updated basic template (basic.yaml) from the below command. The below command uses kube-demo Makefile and render_catalogs.sh which requires basic template name to be basic.yaml

make catalogs

·       Create new signed commit with the updated basic.yaml and all the support Openshift’s full FBC catalogs for kube-demo operator. Push new commit to the forked Community Operators Production repository with new branch e.g. kube-demo-0.0.2-fbc.

·       Create new pull request for Catalog Update which gets auto-merged (if PR created by approved reviewers list) after validating all the checks from operator-hosted-pipeline workflow. The merged catalog update pull request executes the operator-release-pipeline workflow to release all the catalog updates to community OperatorHub.