Can we consider to add "fsid" as a filter variable alongside with "cluster"?

[Deleted user]

Guys, in my work environment, I have multiple ceph clusters. Because I'm new to Ceph, when I installed these clusters, I didn't modified their cluster names. As a result, they all have the same cluster name, so I cannot use "cluster" to distinguish them. By far, there seems no easy way to modify ceph cluster name, too.

The solution jumped into my mind is adding "fsid" as a new variable. It should be unique. Of course, shortcoming is also obvious: it will be harder to tell which is which, but, at least, it should've worked.

What do you guys think about it?

[Vaibhav Bhembre]

To note in our production clusters we are not using single ceph-exporter anymore. Each of our clusters have their own ceph-exporters. We have found this model much more simpler to manage and should resolve the issue you are facing as well.

[Deleted user]

@neurodrone
Thanks for your information, but I am still little confused. Do you mean each cluster has its own Prometheus?

By far, I also use separate ceph-exporter for different clusters too, and these three ceph-exporter deliver data to the same Prometheus. Although they are separated into jobs, in the Grafana dashboard, these data still cannot be differentiated.

I doodled a picture to show what the architecture of my environment. So I suppose you guys have different Prometheus server for different ceph cluster, right?

[Vaibhav Bhembre]

Thanks for the elaborate diagram, @blackpiglet! We have the exact setup - so multiple ceph-exporters, each per cluster and they are all scraped by a single prometheus (typically a unified global set of prometheus servers for fault-tolerance).

Since the instances that run ceph-exporter are distinct we can relabel the targets within (as seen here) the prometheus server config to persist the time-series separately for each cluster by adding a unique label for every ceph-exporter. That should give you the per-cluster breakdown you need to query. Let me know if this approach makes sense otherwise I can further elaborate.

[Deleted user]

@neurodrone
Thanks! I hadn't heard re-label before. This is really a useful feature. I came out a working configuration.

scrape_configs:
  - job_name: 'ceph_rgw_exporter'
    relabel_configs:
    - source_labels: ["cluster"]
      replacement: "rgw"
      action: replace
      target_label: "clusters"
    static_configs:
    - targets: ['localhost:9128']
      labels:
        alias: ceph_rgw_exporter

  - job_name: 'cephfs_exporter'
    relabel_configs:
    - source_labels: ["cluster"]
      replacement: "cephfs"
      action: replace
      target_label: "clusters"
    static_configs:
    - targets: ['10.19.250.136:9128']
      labels:
        alias: cephfs_exporter

  - job_name: 'ceph_rbd_exporter'
    relabel_configs:
    - source_labels: ["cluster"]
      replacement: "rbd"
      action: replace
      target_label: "clusters"
    static_configs:
    - targets: ['10.19.250.132:9128']
      labels:
        alias: ceph_rbd_exporter