Altinity ClickHouse® Operator at Red Hat’s OperatorHub.io

by Alexander Zaitsev

CoreOS introduced operators for Kubernetes in 2016. A Kubernetes Operator is an application that integrates into Kuberentes and manages other applications. It significantly simplifies deployment and management of cloud native apps by leveraging application specific operational knowledge. Since then operators have been developed for many applications in Kubernetes, including some databases . As a result, operators became a successful pattern for managing cloud native applications.

This success led to a problem. If you need to run a particular Kubernetes application, how do you find and load the appropriate operator?

In 2018 Red Hat acquired CoreOS and shortly after OperatorHub.io was launched. OperatorHub.io is a ‘marketplace’ for operators: a standardized way to build, discover and install operators. The concept looks promising, and we submitted the Altinity ClickHouse operator to OperatorHub.io. In this article, we will review what it means and how users can benefit from the operator registry.

OperatorHub.io overview

Red Hat uses OperatorHub.io as part of the OpenShift 4.x platform, which is based on Kubernetes ( https://www.openshift.com ). OperatorHub.io is not just a simple catalog. It requires and enforces operators to implement the Operator Lifecycle Manager API (OLM, see https://github.com/operator-framework/operator-lifecycle-manager ).

OLM extends Kubernetes to enable management of operators in a declarative way. It is operator of operators. Under OLM, each operator becomes a managed resource. Operators can be added to the catalog for particular Kubernetes cluster, and become available for users to install easily by name using standardised resource definitions and kubectl utility.

Here is a diagram showing relevant components.

OLM imposes some important requirements on the operators it manages:

• An operator and the resources it operates must be restricted to one namespace due to difficulties to enforce RBAC in multi-tenant environments.
• Operator configuration should be represented as Kubernetes as CRD (Custom Resource Definition)
• The operator needs to clearly specify its dependencies for resolution.

There are some other requirements and considerations. You can read the details about the OLM philosophy in this article: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/philosophy.md

Important note: There is an Operator SDK providing a standardised toolset for operator developers that is somewhat limited. Fortunately OLM does not require you to use it.

Installing Operator Lifecycle Manager (OLM)

Let’s take a look on how Altinity clickhouse-operator can be installed from OperatorHub.io. The OLM model implies that administrators are responsible for making operators to be available for end users. So we have to install OLM and add the ClickHouse operator to the OLM catalog. OperatorHub.io provides good instructions on how to do so. Just navigate to https://operatorhub.io/operator/clickhouse and click “Install”.

You will see step-by-step installation instructions. Let’s look at and discuss them in more detail. We assume that the Kubernetes cluster is available, and kubectl is already properly configured for access.

The first step is to run an install script to deploy OLM.

$ curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.12.0/install.sh | bash -s 0.12.0

This takes a while and installs all necessary resources into two namespaces: olm and operators. Let’s see if there is anything interesting inside. In particular, we can find that there is a catalog of community operators:

# kubectl get catsrc --all-namespaces

Another interesting resource is packagemanifest. It can be used in order to find the name of appropriate operators:

# kubectl get packagemanifests -n olm

‘kubectl describe packagemanifest clickhouse -n olm’ shows additional info that matches information on OperatorHub.io itself.

Installing clickhouse-operator using OLM

Now let’s Install clickhouse-operator from operatorhub.io with the help of OLM.

OperatorHub.io provides a ready-to-use manifest that does all the necessary preparation.

# kubectl create -f https://operatorhub.io/install/clickhouse.yaml

Once this manifest is applied clickhouse-operator will be installed in the “my-clickhouse” namespace and will be usable from this namespace only. OLM will make sure that clickhouse-operator has the latest version using Subscription resource mechanism. Namespace and subscription settings can be obviously changed, and we can look inside clickhouse.yaml to see how we can do it:

apiVersion: v1
kind: Namespace
metadata:
  name: my-clickhouse
---
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: operatorgroup
  namespace: my-clickhouse
spec:
  targetNamespaces:
  - my-clickhouse
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-clickhouse
  namespace: my-clickhouse
spec:
  channel: latest
  name: clickhouse
  source: operatorhubio-catalog
  sourceNamespace: olm

So, if clickhouse-operator needs to be installed in another namespace, it can be done with a new Subscription resource. Note also the subscription channel source — it is the same community catalog that we observed above. In general, installing an operator looks very simple with OLM. The manifest is much simpler than the one without OLM. (See https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md or https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/manifests/operator/clickhouse-operator-install.yaml for examples.)

After applying the above manifest, we can check that the operator is available by examining yet another OLM custom resource ClusterServiceVersion:

# kubectl get csv -n my-clickhouse

We should see something like the following:

Now we have clickhouse-operator up and running in the “my-clickhouse” namespace. ClickHouse clusters must be created within the same namespace. If one wants to have ClickHouse in a different namespace, another operator needs to be installed there as described above.

With clickhouse-operator up and running, we can now proceed to install ClickHouse clusters. There are many examples available at the clickhouse-operator github.com page (link to https://github.com/Altinity/clickhouse-operator/tree/master/docs/examples). But we can craft a special one for OLM installations. olm-ch-cluster.yaml 3 shard cluster with persistent storage:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: olm-ch-cluster
spec:
  defaults:
    templates: 
      volumeClaimTemplate: default 
  configuration:
    clusters:
      - name: default
        layout:
          shardsCount: 3
  templates:
    volumeClaimTemplates:
      - name: default
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 500Mi

Let’s apply this manifest and check ClickHouseInstallation is up and running:

# kubectl apply -f olm-ch-cluster.yaml -n my-clickhouse
clickhouseinstallation.clickhouse.altinity.com/olm-ch-cluster created

# kubectl get chi -o wide -n my-clickhouse

Conclusion

OperatorHub.io provides a transparent way to manage different operators using OLM and make them accessible to kubernetes cluster users. It requires you to learn some new custom resources but is not very complicated. We have only recently begun to use it and anticipate there may be pitfalls, but the approach looks promising. Having clickhouse-operator at OperatarHub.io makes it easy to find and download by ClickHouse users.

Note, however, that clickhouse-operator has a lot of configuration capabilities that allow you to customize its behavior and simplify clickhouse deployment. Those are not used when operator is installed via OLM due to OLM limitations. For advanced configuration capabilities clickhouse-operator needs to be installed from Altinity github. See the Quick Start Guide and the Operator Configuration Guide for more details.

The ClickHouse operator is an important step for ClickHouse. Kubernetes popularity is increasing rapidly and managing databases in Kubernetes is transitioning from a challenging exercise to a mainstream practice. Several companies are already using ClickHouse operator in test and production environments. They are providing valuable feedback and helping us to improve the operator. Please join! ClickHouse is fun, and ClickHouse in Kubernetes is fun²!

Originally published at https://www.altinity.com on October 18, 2019.