monitoring – Prometheus auto scrape metrics from multiple kube-state-metrics in kubernetes?


I want to use a kubernetes(cluster-0)with multiple kube-state-metrics to monitor multiple other kubernetes cluster(cluster-1,2,3,4)

In the (cluster-0), I split into multiple namespaces like this:

namespace: monitor

(here I will run prometheus to scrape metrics from other kube-state-metrics pod)
namespaces: cluster-0
(here I will run kube-state-metrics pod to monitor cluster-0
....
namespaces: cluster-4
(here I will run kube-state-metrics pod to monitor cluster-4)
...

And so on…

The problems here is how would I setup prometheus to auto scrape new kube-state-metrics since I will expand the kubernetes cluster very fast, or shrink it. And how to tell the different metrics between them, like the having a header …

I have try using config but just locate single - targets: ('serviceIP:8080') is not very good in my situation.