New to KubeDB? Please start here.

Visualize ClickHouse Metrics with Grafana Dashboard

KubeDB exposes ClickHouse metrics through a sidecar exporter. Once Prometheus scrapes those metrics, you can visualize them in Grafana using a pre-built KubeDB dashboard. This tutorial walks through the full setup: deploying the monitoring stack, enabling monitoring on a ClickHouse instance, and importing the Grafana dashboard.

Before You Begin

• You need a Kubernetes cluster with kubectl configured. If you do not already have a cluster, you can create one by using kind.

• KubeDB must be installed in your cluster with kubedb-metrics enabled. Follow the setup guide here and make sure to include the flag below during installation:

  --set kubedb-metrics.enabled=true
  

  kubedb-metrics creates MetricsConfiguration objects for each database type, which Panopticon (Step 2) uses to expose metrics to Prometheus.

• To keep monitoring resources isolated, we use a separate monitoring namespace and deploy the database in the demo namespace.

  $ kubectl create ns monitoring
  namespace/monitoring created
  
  $ kubectl create ns demo
  namespace/demo created
  

Note: YAML files used in this tutorial are stored in docs/examples/clickhouse/monitoring folder in GitHub repository kubedb/docs.

Configuration

These two steps — deploying kube-prometheus-stack and installing Panopticon — are shared prerequisites for all KubeDB database monitoring guides. If you have already completed them in another guide, skip to Step 1.

Step 1: Deploy kube-prometheus-stack

kube-prometheus-stack installs Prometheus, Prometheus Operator, Alertmanager, and Grafana together. This is the recommended way to get the full monitoring stack on Kubernetes.

Add the prometheus-community Helm repo and install:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

$ helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set grafana.image.tag=7.5.5

Wait for all pods to be ready:

$ kubectl get pods -n monitoring
NAME                                                   READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2     Running   0          2m
prometheus-grafana-xxxx                                3/3     Running   0          2m
prometheus-kube-prometheus-operator-xxxx               1/1     Running   0          2m
prometheus-kube-prometheus-prometheus-0                2/2     Running   0          2m
prometheus-kube-state-metrics-xxxx                     1/1     Running   0          2m

Find the serviceMonitorSelector label that Prometheus uses to pick up ServiceMonitor objects. You will need this label when enabling monitoring on the ClickHouse instance.

$ kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'
{"matchLabels":{"release":"prometheus"}}

The label is release: prometheus.

Step 2: Install Panopticon

Panopticon is the Appscode operator that reads MetricsConfiguration objects created by kubedb-metrics and exposes them to Prometheus. It must be installed before enabling kubedb-metrics.

$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update

$ helm upgrade --install panopticon appscode/panopticon \
  --version v2026.4.30 \
  --namespace kubeops --create-namespace \
  --set monitoring.enabled=true \
  --set monitoring.agent=prometheus.io/operator \
  --set monitoring.serviceMonitor.labels.release=prometheus \
  --set-file license=/path/to/kubedb-license.txt \
  --wait --timeout 5m0s

Verify panopticon is running:

$ kubectl get pods -n kubeops
NAME                          READY   STATUS    RESTARTS   AGE
panopticon-xxxx               1/1     Running   0          1m

Setup

Step 1: Deploy ClickHouse with Monitoring Enabled

Below is the ClickHouse object with monitoring configured to use Prometheus Operator.

apiVersion: kubedb.com/v1alpha2
kind: ClickHouse
metadata:
  name: ch-grafana-demo
  namespace: demo
spec:
  version: "26.2.6"
  deletionPolicy: WipeOut
  storage:
    storageClassName: "local-path"
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
  monitor:
    agent: prometheus.io/operator
    prometheus:
      serviceMonitor:
        labels:
          release: prometheus
        interval: 10s

Here,

• monitor.agent: prometheus.io/operator tells KubeDB to create a ServiceMonitor for this instance.
• monitor.prometheus.serviceMonitor.labels must match the serviceMonitorSelector label of your Prometheus (release: prometheus).
• monitor.prometheus.serviceMonitor.interval sets the scrape interval to 10 seconds.

Create the ClickHouse instance:

$ kubectl create -f https://github.com/kubedb/docs/raw/v2026.6.19/docs/examples/clickhouse/monitoring/coreos-prom-clickhouse.yaml
clickhouse.kubedb.com/ch-grafana-demo created

Wait for it to be Ready:

$ kubectl get clickhouse -n demo ch-grafana-demo
NAME              VERSION   STATUS   AGE
ch-grafana-demo   26.2.6    Ready    2m

KubeDB creates a stats service named {clickhouse-name}-stats for the exporter:

$ kubectl get svc -n demo --selector="app.kubernetes.io/instance=ch-grafana-demo"
NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
ch-grafana-demo         ClusterIP   10.96.10.1     <none>        9000/TCP    2m
ch-grafana-demo-stats   ClusterIP   10.96.10.2     <none>        9363/TCP    2m

KubeDB also creates a ServiceMonitor in the demo namespace:

$ kubectl get servicemonitor -n demo
NAME                    AGE
ch-grafana-demo-stats   2m

Verify it carries the correct label:

$ kubectl get servicemonitor -n demo ch-grafana-demo-stats -o jsonpath='{.metadata.labels}'
{"release":"prometheus", ...}

Step 2: Verify Prometheus is Scraping

Port-forward the Prometheus pod:

$ kubectl port-forward -n monitoring \
  prometheus-prometheus-kube-prometheus-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Open http://localhost:9090/targets in your browser. Look for an entry whose service label matches ch-grafana-demo-stats. Its state should be UP.

If the target is missing, check that the ServiceMonitor label (release: prometheus) matches the Prometheus serviceMonitorSelector.

Step 3: Access Grafana

Port-forward the Grafana service:

$ kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
Forwarding from 127.0.0.1:3000 -> 80

Open http://localhost:3000. The username is admin. Retrieve the auto-generated password from the secret:

$ kubectl get secret -n monitoring prometheus-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d

After a successful login you will see the Grafana home page:

Step 4: Configure Prometheus as a Data Source

If you installed Grafana via kube-prometheus-stack, Prometheus is already configured as the default data source — skip to Step 5.

For a standalone Grafana installation:

1. Go to Connections → Data sources → Add new data source.

2. Select Prometheus.

3. Set the URL to your Prometheus service:

   http://prometheus-operated.monitoring.svc:9090
   

4. Click Save & test. You should see Data source is working.

Step 5: Import KubeDB ClickHouse Dashboard

The KubeDB ClickHouse dashboards are distributed as JSON files. Each JSON file is a complete dashboard definition — panels, queries, variables, and layout — that Grafana loads in one shot. Without importing, you would have to build every panel and write every PromQL query by hand. Importing lets you skip that entirely.

Three dashboards are available. Download all three JSON files from the appscode/grafana-dashboards repository (clickhouse/ folder):

Import steps (repeat for each of the three files):

1. In Grafana, click the + icon in the left sidebar.
2. Select Import from the menu.
3. Click Upload JSON file and select one of the downloaded .json files.
4. In the Prometheus dropdown that appears, select your Prometheus data source.
5. Click Import.

The import page looks like this — click Upload dashboard JSON file to select the file:

After importing all three files, they will appear under Dashboards in the left sidebar.

Step 6: Explore the Dashboard

After opening a dashboard, you will see dropdown filters at the top. These control which data is shown across all panels — change them to focus on a specific instance without editing any queries.

Once you set these, all panels update automatically. Below is what each dashboard shows:

KubeDB / ClickHouse / Summary — start here for a cluster health overview:

• Queries per Second — total query throughput across the cluster
• Memory Usage — current memory consumption vs. configured limits
• Merged Parts — background merge operations per second (health indicator)
• Inserted Rows / Bytes — data ingestion rate
• Replication Queue — pending replicated operations (should be near zero in healthy clusters)
• CPU / Storage — resource consumption vs. requests and limits

KubeDB / ClickHouse / Pod — drill into a specific pod:

• Queries — queries executed on this specific pod
• Memory — per-pod memory usage over time
• CPU — per-pod CPU usage
• Merge Operations — background merges on this pod
• Replicated Fetches — data fetched from other replicas

KubeDB / ClickHouse / Database — service health and query-level metrics, grouped into three collapsible sections:

General — service reachability and cluster role:

• Service Status / Service Uptime — whether the exporter endpoint is UP, and how long the pod has been running
• Healthy Pods Count — number of ClickHouse pods currently healthy
• Cluster Status — this pod’s role (Primary or replica)
• Current QPS / Active ClickHouse Connections — live query rate and open client connections
• ClickHouse Metrics reads vs writes — read vs. write operation rate
• ClickHouse network received vs sent — network throughput in/out

Service Health and Queries Overview — cache efficiency and query success rates:

• ClickHouse Opened File Cache Hits / File Cache Miss Rate — file cache effectiveness
• Max Parts Per Partition — highest number of active data parts in any partition (high counts may need compaction)
• Percent of Failed SELECTs / INSERTs, logged errors — error rates (should stay at 0)
• ClickHouse Cluster Query Rate — total queries per second across the cluster
• Queries with MEMORY_LIMIT_EXCEEDED, Failed Queries, Failed SELECT Queries, Failed Asynchronous INSERT Queries — failure counters (should stay at 0)
• SELECT Queries, Synchronous / Asynchronous INSERT Queries — query throughput by type

Cleaning up

# Remove the ClickHouse instance
kubectl delete clickhouse -n demo ch-grafana-demo

# Remove namespaces
kubectl delete ns demo

# Uninstall monitoring stack (optional)
helm uninstall prometheus -n monitoring
helm uninstall panopticon -n kubeops
kubectl delete ns monitoring kubeops

Next Steps

• Monitor your ClickHouse database with KubeDB using built-in Prometheus.
• Monitor your ClickHouse database with KubeDB using Prometheus Operator.
• Want to hack on KubeDB? Check our contribution guidelines.