'--config.file' is not recognized as an internal or external command on 0.24.0 and 0.25.1
sesoldi opened this issue · comments
I've built 2 images based on 0.24.0, and also tried with 0.25.1 with the same result.
I'm attempting to run this on AKS 1.28, on a windows node, but the pod fails to start.
This is our dockerfile, where we're also installing powershell and a few KBs in order to remediate vulnerabilities.
FROM ghcr.io/prometheus-community/windows-exporter:0.24.0
FROM mcr.microsoft.com/powershell
USER ContainerAdministrator
RUN powershell -Command \
"[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;" \
"Install-PackageProvider -Name NuGet -Force;" \
"Install-Module -Confirm:$false -Force -Name PSWindowsUpdate"
RUN powershell -Command \
"Get-WindowsUpdate -Install -KBArticleID KB5031364 -AcceptAll -IgnoreUserInput;" \
"Get-WindowsUpdate -Install -KBArticleID KB5032198 -AcceptAll -IgnoreUserInput;" \
"Get-WindowsUpdate -Install -KBArticleID KB5033118 -AcceptAll -IgnoreUserInput"
USER ContainerUser
These are our DS and CM manifests.
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: win-prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-windows-exporter
app.kubernetes.io/part-of: prometheus-windows-exporter
app.kubernetes.io/version: 0.24.0
helm.sh/chart: prometheus-windows-exporter-0.2.0
name: win-prometheus
namespace: prometheus
spec:
selector:
matchLabels:
app.kubernetes.io/instance: win-prometheus
app.kubernetes.io/name: prometheus-windows-exporter
template:
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
labels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: win-prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-windows-exporter
app.kubernetes.io/part-of: prometheus-windows-exporter
app.kubernetes.io/version: 0.24.0
helm.sh/chart: prometheus-windows-exporter-0.2.0
spec:
automountServiceAccountToken: false
containers:
- args:
- --config.file=%CONTAINER_SANDBOX_MOUNT_POINT%/config.yml
- --collector.textfile.directories=%CONTAINER_SANDBOX_MOUNT_POINT%
- --web.listen-address=:9182
env: null
image: xxxxx/xxxx-win-prometheus-node-exporter:1.28.latest
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
httpHeaders: null
path: /
port: 9182
scheme: HTTP
initialDelaySeconds: 0
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: windows-exporter
ports:
- containerPort: 9182
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
httpHeaders: null
path: /
port: 9182
scheme: HTTP
initialDelaySeconds: 0
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
- mountPath: /config.yml
name: config
subPath: config.yml
hostNetwork: true
hostPID: true
imagePullSecrets:
- name: xxxx-docker-registry-secret
initContainers:
- args:
- New-NetFirewallRule
- -DisplayName
- '''windows-exporter'''
- -Direction
- inbound
- -Profile
- Any
- -Action
- Allow
- -LocalPort
- "9182"
- -Protocol
- TCP
command:
- powershell
image: xxxxx/xxxxxx-win-prometheus-node-exporter:1.28.latest
name: configure-firewall
nodeSelector:
kubernetes.io/os: windows
securityContext:
windowsOptions:
hostProcess: true
runAsUserName: NT AUTHORITY\system
serviceAccountName: win-prometheus
tolerations:
- effect: NoSchedule
operator: Exists
volumes:
- configMap:
name: win-prometheus
name: config
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
---
apiVersion: v1
data:
config.yml: |
collectors:
enabled: '[defaults],container'
collector:
service:
services-where: "Name='containerd' or Name='kubelet'"
telemetry:
addr: ":9182"
path: /metrics
max-requests: 5
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: win-prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-windows-exporter
app.kubernetes.io/part-of: prometheus-windows-exporter
app.kubernetes.io/version: 0.24.0
helm.sh/chart: prometheus-windows-exporter-0.2.0
name: win-prometheus
namespace: prometheus
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: win-prometheus
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-windows-exporter
app.kubernetes.io/part-of: prometheus-windows-exporter
app.kubernetes.io/version: 0.24.0
helm.sh/chart: prometheus-windows-exporter-0.2.0
name: win-prometheus
namespace: prometheus
---
The pod spins up and immediately goes into CrashLoopBackOff.
Describing the pod initially it indicates that it's not able to reach the probes.
Warning Unhealthy 13m (x8 over 14m) kubelet Readiness probe failed: Get "http://10.0.0.10:9182/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 13m (x2 over 14m) kubelet Liveness probe failed: Get "http://10.0.0.10:9182/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
However, looking into the logs of the container it appears that it's not finding the process.
C:\Users\xxxxx\.kube\Projects YAML\prometheus>kubectl -n prometheus logs win-prometheus
Defaulted container "windows-exporter" out of: windows-exporter, configure-firewall (init)
'--config.file' is not recognized as an internal or external command,
operable program or batch file.
I went over the documentation, and the DS template in the chart, and it appears that configuration is identical as suggested:
containers:
- name: windows-exporter
image: {{ include "prometheus-windows-exporter.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- --config.file=%CONTAINER_SANDBOX_MOUNT_POINT%/config.yml
- --collector.textfile.directory=%CONTAINER_SANDBOX_MOUNT_POINT%
- --web.listen-address=:{{ .Values.service.port }}
{{- with .Values.extraArgs }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.containerSecurityContext }}
Yes, thanks for pointing that out. In later testing I did add it, as well as the cmd and env array upon doing a docker inspect on the base 0.24.0 image I noticed was there.
Partial output of docker inspect from base 0.24.0 Dockerfile.
"Env": [
"PATH=C:\\Windows\\system32;C:\\Windows;"
],
"Cmd": [
"cmd",
"/S",
"/C",
"#(nop) ",
"ENTRYPOINT [\"windows_exporter.exe\"]"
],
So my new Dockerfile looks like this:
FROM ghcr.io/prometheus-community/windows-exporter:0.25.1
FROM mcr.microsoft.com/powershell
USER ContainerAdministrator
RUN powershell -Command \
"[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;" \
"Install-PackageProvider -Name NuGet -Force;" \
"Install-Module -Confirm:$false -Force -Name PSWindowsUpdate"
RUN powershell -Command \
"Get-WindowsUpdate -Install -KBArticleID KB5034129 -AcceptAll -IgnoreUserInput"
USER ContainerUser
ENV PATH="C:\Windows\system32;C:\Windows;"
CMD ["cmd", "/S", "/C", "#(nop) ", "ENTRYPOINT [\"windows_exporter.exe\"]"]
Below the docker inspection output of the final build:
PS E:\Dockerfiles> docker image inspect test/test:v1
[
{
"Id": "sha256:d8ea0987dddcd70872beb83f684a58e78480a8f0dad8cdbd9aaaee2f3fc22ba5",
"RepoTags": [
"test/test:v1"
],
"RepoDigests": [],
"Parent": "sha256:260d4a7578991560de028619a2b14f7cc8e00ff890e7713a1347352c6de23108",
"Comment": "",
"Created": "2024-01-21T19:25:29.1256028Z",
"Container": "371028f1da78fdb255de4edbccff377738fcee76f4498cf19f8ec7c19a572b1e",
"ContainerConfig": {
"Hostname": "371028f1da78",
"Domainname": "",
"User": "ContainerUser",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"ProgramFiles=C:\\Program Files",
"PSModuleAnalysisCachePath=C:\\Users\\Public\\AppData\\Local\\Microsoft\\Windows\\PowerShell\\docker\\ModuleAnalysisCache",
"PSCORE=\\PowerShell\\pwsh.exe",
"POWERSHELL_DISTRIBUTION_CHANNEL=PSDocker-WindowsServerCore-ltsc2022",
"POWERSHELL_TELEMETRY_OPTOUT=0",
"PATH=C:\\Windows\\system32;C:\\Windows;"
],
"Cmd": [
"cmd",
"/S",
"/C",
"#(nop) ",
"CMD [\"cmd\" \"/S\" \"/C\" \"#(nop) \" \"ENTRYPOINT [\\\"windows_exporter.exe\\\"]\"]"
],
"Image": "sha256:260d4a7578991560de028619a2b14f7cc8e00ff890e7713a1347352c6de23108",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"DockerVersion": "24.0.5",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "ContainerUser",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"ProgramFiles=C:\\Program Files",
"PSModuleAnalysisCachePath=C:\\Users\\Public\\AppData\\Local\\Microsoft\\Windows\\PowerShell\\docker\\ModuleAnalysisCache",
"PSCORE=\\PowerShell\\pwsh.exe",
"POWERSHELL_DISTRIBUTION_CHANNEL=PSDocker-WindowsServerCore-ltsc2022",
"POWERSHELL_TELEMETRY_OPTOUT=0",
"PATH=C:\\Windows\\system32;C:\\Windows;"
],
"Cmd": [
"cmd",
"/S",
"/C",
"#(nop) ",
"ENTRYPOINT [\"windows_exporter.exe\"]"
],
"Image": "sha256:260d4a7578991560de028619a2b14f7cc8e00ff890e7713a1347352c6de23108",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "windows",
"OsVersion": "10.0.20348.2227",
"Size": 4651313965,
"VirtualSize": 4651313965,
"GraphDriver": {
"Data": {
"dir": "C:\\ProgramData\\Docker\\windowsfilter\\9e6e9f33626a2a23e60dde802b00f6d9e7d16bb92b3474b2d16b37dd670b9eeb"
},
"Name": "windowsfilter"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:aa27a343ff6a62b6bad81ca57f0f66c9bde947554a41af569eca4ed4822a37f8",
"sha256:167e2656ee7d78bc6314110b4724890ff15a20919ef4ecbdef650617d21360a4",
"sha256:3301cb4218042a78175fb808a07b8c1004b1ffcaee7384b6bec0c725693485a6",
"sha256:1a721a25b1ff58cb68353408e24ea5e6808be5976ae09060ee30f6fce6c6b877",
"sha256:f73dcd7bbff87089ca47b9e90559735b9fceee31450c841b6cb9a381516414c6",
"sha256:2a49bce191027631d242169a8714e357948c0d51f34db59aaf47a89f7643179d",
"sha256:fec37ddb5e246cbed57228235704b910a8480500fb870675bd55c3112dccd943",
"sha256:f6cd9074567a4df9d5c3bae512730894ea9bb8e066e209a9f0a8c008750876d4",
"sha256:dd2d100fd0c3e7bc9132bfeeb648bd6b42099af2dcf96187d1bacdcf9f659312",
"sha256:f88141335ba2c6d35447f5c7f29049516636807cf937adde6e59ce35f654b03f",
"sha256:c08349db43d26a02fc8ac2cfef8bdc79eb68dc50636e39511498339c8adc91e1",
"sha256:4f7e6877ea132ad2f7590d3e876499f4bb50d57afcc3c72c15177bc10b4594f5",
"sha256:f7128387b33995d3a07646ce1e2929d6b2975a06db070455bc2b3ab45cee564a",
"sha256:cef45fceb04610efae9d5f6b851ab21fa9ab7cfefea7396c24e3358f933404be"
]
},
"Metadata": {
"LastTagTime": "2024-01-21T20:25:29.1520475+01:00"
}
}
]
But still returns an event within the container that states:
Defaulted container "windows-exporter" out of: windows-exporter, configure-firewall (init)
'--config.file' is not recognized as an internal or external command,
operable program or batch file.
Please just use
Line 9 in 21a7bea
You got confused by the Cmd output of docker inspect which is not the same as CMD at the Dockerfile. They are 2 distinct things.
The behavior here is expected. Your Dockerfile does not contain a ENTRYPOINT, only a CMD definition.
The CMD line gets overridden by pod.spec.containers.args from Kubernetes. From runtime point of view, nothing has been changed. And the error is the same.
Alternately, prepend windows_exporter.exe to the list of strings at pod.spec.containers.args
Thanks for you reply.
Yes, I noticed that and did a few more tests.
FROM ghcr.io/prometheus-community/windows-exporter:0.24.0
FROM mcr.microsoft.com/powershell
USER ContainerAdministrator
RUN powershell -Command \
"[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;" \
"Install-PackageProvider -Name NuGet -Force;" \
"Install-Module -Confirm:$false -Force -Name PSWindowsUpdate"
RUN powershell -Command \
"Get-WindowsUpdate -Install -KBArticleID KB5031364 -AcceptAll -IgnoreUserInput;" \
"Get-WindowsUpdate -Install -KBArticleID KB5032198 -AcceptAll -IgnoreUserInput;" \
"Get-WindowsUpdate -Install -KBArticleID KB5033118 -AcceptAll -IgnoreUserInput"
USER ContainerUser
ENV PATH="C:\Windows\system32;C:\Windows;"
ENTRYPOINT ["windows_exporter.exe"]
Tried initially only adding the entrypoint, and then tried adding the env path and the entrypoint, while also passing the windows_exporter.exe as an argument in the DS manifest
containers:
- args:
- windows_exporter.exe
- --config.file=%CONTAINER_SANDBOX_MOUNT_POINT%/config.yml
- --collector.textfile.directories=%CONTAINER_SANDBOX_MOUNT_POINT%
- --web.listen-address=:9182
But still comes up with the error in both cases
Defaulted container "windows-exporter" out of: windows-exporter, configure-firewall (init)
'windows_exporter.exe' is not recognized as an internal or external command,
operable program or batch file.
I can see any line that relates to copy the windows_exporter.exe binary into the final docker file, like this:
Line 8 in 21a7bea
Wouldn't the executable file be already present in the source image
FROM ghcr.io/prometheus-community/windows-exporter:0.24.0
But looking at your Dockerfile, FROM mcr.microsoft.com/powershell is used as base.
A Docker images can only have one base and at the Dockerfile, only the lastest FROM config is used as base image for the final image
Thank you very much for your guideance, really appreciate it.
After modifying the docker file it appears to be working.
FROM ghcr.io/prometheus-community/windows-exporter:0.24.0 as prometheus
FROM mcr.microsoft.com/powershell
#### Fixing vulns
## install WindowsUpdates
USER ContainerAdministrator
RUN powershell -Command \
"[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;" \
"Install-PackageProvider -Name NuGet -Force;" \
"Install-Module -Confirm:$false -Force -Name PSWindowsUpdate"
RUN powershell -Command \
"Get-WindowsUpdate -Install -KBArticleID KB5031364 -AcceptAll -IgnoreUserInput;" \
"Get-WindowsUpdate -Install -KBArticleID KB5032198 -AcceptAll -IgnoreUserInput;" \
"Get-WindowsUpdate -Install -KBArticleID KB5033118 -AcceptAll -IgnoreUserInput"
USER ContainerUser
COPY --from=prometheus /windows_exporter.exe .
ENV PATH="C:\Windows\system32;C:\Windows;"
ENTRYPOINT ["windows_exporter.exe"]
I'll leave it here in case anybody else runs into this issue.