adding a second Ceph cluster to MicroK8s

[duffrecords]

I have two Ceph clusters (one dedicated to provisioning block devices to MicroK8s and another that is one big CephFS volume for documents and other media). The first one has already been added to my MicroK8s cluster and that added both the CephFS and RBD plugins, the latter of which is all I actually needed for that purpose. Now I want to add the second one so I can optionally mount some of its directories within my containers but I haven't found much documentation about how to add multiple Ceph clusters. I did read that they need to be in separate namespaces. Are there any Ceph CSI resources that should be shared between both clusters or do I need a completely separate Ceph CSI deployment for the second cluster?

[Madhu-1]

@duffrecords you can add one more cluster with a different clusterID

ceph-csi/deploy/csi-config-map-sample.yaml

Line 70 in a32ba13

"clusterID": "<cluster-id>",

and use that clusterID in the storageclass to create PVC from. the clusterID is an arbitrary value to point to which the ceph cluster is to be considered to provision the pvc.

[duffrecords]

So is it just a matter of editing the existing ConfigMap and creating a StorageClass and then the existing csi-cephfsplugin pods that are running on each node will use that information, i.e. without having to deploy a second set of pods in a different namespace?

[Madhu-1]

Yes thats correct, a single CSI driver can work with multiple ceph cluster.

[duffrecords]

Thanks! It was actually simpler than I thought. For anyone else trying to do this, here's what I did.

First, I completely removed the resources I had previously created with MicroK8s' rook-ceph addon because I couldn't really tell what was happening under the hood. Then I used the RBD and CephFS plugin templates from this project. The only part I had to modify was the kubelet path because MicroK8s is installed as a snap and, although there are symbolic links to provide the typical /var/lib/kubelet, the containers can't resolve those to their targets on the host system. So basically everywhere that path was referenced in the templates, I changed it to /var/snap/microk8s/common/var/lib/kubelet. Then I created a CSI config map like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
  namespace: default
data:
  config.json: |-
    [
      {
        "clusterID": "b0f91435-1b5f-4a8b-994f-c1639e319834",
        "monitors": [
          "10.0.0.19:6789",
          "10.0.0.20:6789",
          "10.0.0.21:6789"
        ]
      },
      {
        "clusterID": "ec12af30-dd15-47d5-8483-ebfde0522a6b",
        "monitors": [
          "10.0.0.9:6789",
          "10.0.0.14:6789",
          "10.0.0.18:6789"
        ],
        "cephFS": {
          "subvolumeGroup": ""
        }
      }
    ]

I realized the rest of the parameters from the sample file weren't necessary for my setup so the config map actually ended up being very minimal. Then I created the secrets and storage classes for each cluster and now it's working!

[julienlau]

it may be a good idea to have an example of this in the repo