Disk collection issues
dxygit1 opened this issue · comments
df: /host/root/var/lib/kubelet/pods/4acbef13-d0c4-4b3f-b612-da33e6d7b7ac/volumes/kubernetes.io~projected/kube-api-access-fgck4: Permission denied
I want to monitor some directories mounted in/var/lib/kubelet/because PVC is mounted in this directory. I want to monitor resource usage, but there are permission denied errors
can you share your permission of the dir and the deploy yaml of node_exporter?
I deployed it using Helm, and the address is as follows: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter
`Source: prometheus-node-exporter/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: acos-prometheus-node-exporter
namespace: default
labels:
helm.sh/chart: prometheus-node-exporter-4.22.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
app.kubernetes.io/version: "1.6.0"
Source: prometheus-node-exporter/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: acos-prometheus-node-exporter
namespace: default
labels:
helm.sh/chart: prometheus-node-exporter-4.22.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
app.kubernetes.io/version: "1.6.0"
annotations:
prometheus.io/scrape: "true"
spec:
type: ClusterIP
ports:
- port: 9100
targetPort: 9100
protocol: TCP
name: metrics
selector:
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
Source: prometheus-node-exporter/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: acos-prometheus-node-exporter
namespace: default
labels:
helm.sh/chart: prometheus-node-exporter-4.22.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
app.kubernetes.io/version: "1.6.0"
spec:
selector:
matchLabels:
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
revisionHistoryLimit: 10
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
labels:
helm.sh/chart: prometheus-node-exporter-4.22.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
app.kubernetes.io/version: "1.6.0"
spec:
automountServiceAccountToken: false
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: acos-prometheus-node-exporter
containers:
- name: node-exporter
image: quay.io/prometheus/node-exporter:v1.6.0
imagePullPolicy: IfNotPresent
args:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/host/root
- --path.udev.data=/host/root/run/udev/data
- --web.listen-address=[$(HOST_IP)]:9100
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|tmpfs|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
securityContext:
readOnlyRootFilesystem: true
env:
- name: HOST_IP
value: 0.0.0.0
ports:
- name: metrics
containerPort: 9100
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
httpHeaders:
path: /
port: 9100
scheme: HTTP
initialDelaySeconds: 0
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
httpHeaders:
path: /
port: 9100
scheme: HTTP
initialDelaySeconds: 0
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
- name: proc
mountPath: /host/proc
readOnly: true
- name: sys
mountPath: /host/sys
readOnly: true
- name: root
mountPath: /host/root
mountPropagation: HostToContainer
readOnly: true
hostNetwork: true
hostPID: true
nodeSelector:
kubernetes.io/os: linux
tolerations:
- effect: NoSchedule
operator: Exists
volumes:
- name: proc
hostPath:
path: /proc
- name: sys
hostPath:
path: /sys
- name: root
hostPath:
path: /
Source: prometheus-node-exporter/templates/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: acos-prometheus-node-exporter
namespace: default
labels:
helm.sh/chart: prometheus-node-exporter-4.22.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: metrics
app.kubernetes.io/part-of: prometheus-node-exporter
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
app.kubernetes.io/version: "1.6.0"
acos-prometheus: "true"
spec:
jobLabel: app.kubernetes.io/name
selector:
matchLabels:
app.kubernetes.io/name: prometheus-node-exporter
app.kubernetes.io/instance: acos
attachMetadata:
node: false
endpoints:
- port: metrics
scheme: http
scrapeTimeout: 10s`
@dxygit1 have a look at your host filesystem permission
ls -ldh /var/lib/kubelet
ls -ldh /var/lib/kubelet/pods
/ls -ldh var/lib/kubelet/pods/4acbef13-d0c4-4b3f-b612-da33e6d7b7ac/volumes/kubernetes.io~projected/kube-api-access-fgck4
as for me
ls -ldh /var/lib/kubelet/pods drwxr-x--- 78 root root 24K Sep 8 15:11 /var/lib/kubelet/pods
/var/lib/kubelet/pods is not readable for user id 65534 which defined in pod securityContext, so make sure you have the right
permissions
I'm also encountering unreadable text. However, I've been trying to elevate the permissions to root, but I keep getting errors, stating that root permissions cannot be granted. So, I'm not sure where to make adjustments now.
the easist way to try maybe changing the pod securityContext to :
runAsUser: 0
I've already tried, but it's throwing an error.
container's runAsUser breaks non-root policy (pod: "acos-prometheus-node-exporter-9k7xn_acos(bb6ee109-9573-47f3-b061-b29d60632583)", container: node-exporter)
try to set it in securityContext instead of containerSecurityContext.
and also change runAsNonRoot to false
runAsNonRoot: false
This is acceptable. Thank you very much