couldn't see container_* api metrics

[jamyao]

Hello,
I'm running kubernetes 1.10, and followed instruction here and installed metrics server, custom-metrics-api, and prometheus. I'm able to do autoscaling based on http_requests, but i can not do that based on container_network_receive_bytes_total, i noticed that http_requests metrics exposed at "/apis/custom.metrics.k8s.io/v1beta1", see output below:
[xyz@master k8s-prom-hpa]$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|jq .|grep http_requests
"name": "pods/http_requests_count",
"name": "pods/http_requests",
"name": "namespaces/http_requests",
"name": "jobs.batch/http_requests",
"name": "pods/http_requests_sum",
"name": "jobs.batch/http_requests_bucket",
"name": "jobs.batch/http_requests_count",
"name": "jobs.batch/http_requests_sum",
"name": "namespaces/http_requests_count",
"name": "namespaces/http_requests_sum",
"name": "pods/http_requests_bucket",
"name": "namespaces/http_requests_bucket",

but this is the case for "container_network_receive_bytes_total"

[xyz@master k8s-prom-hpa]$ kubectl get --raw container_network_receive_bytes_total|jq .|grep network
"name": "pods/network_udp_usage",
"name": "pods/network_tcp_usage",

In Prometheus , i can see "container_network_receive_bytes_total" with data, actually all the metrics start with "container_" in Prometheus didn't get exposed at /apis/custom.metrics.k8s.io/v1beta1..., any help is really appreciated

my system information:
[xyz@master k8s-prom-hpa]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[xyz@master k8s-prom-hpa]$ kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
metrics.k8s.io/v1beta1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
stable.example.com/v1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

[yaoky@master k8s-prom-hpa]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default php-apache-8699449574-8hlqf 1/1 Running 0 3h
default podinfo-65fc48f955-9xwdd 1/1 Running 0 22h
default podinfo-65fc48f955-fnzx4 1/1 Running 0 22h
default vtc1-deployment-6f864dbf54-lpq2d 1/1 Running 0 22h
default vtc2-deployment-7f65859fc9-nbt2x 1/1 Running 0 22h
kube-system etcd-master 1/1 Running 1 23h
kube-system heapster-cbfb87c9c-sqknr 1/1 Running 0 23h
kube-system kube-apiserver-master 1/1 Running 0 23h
kube-system kube-controller-manager-master 1/1 Running 2 23h
kube-system kube-dns-86f4d74b45-mgb44 3/3 Running 3 23h
kube-system kube-proxy-52ctt 1/1 Running 0 23h
kube-system kube-proxy-cgbb4 1/1 Running 1 23h
kube-system kube-proxy-fw46f 1/1 Running 0 23h
kube-system kube-proxy-rb4l5 1/1 Running 0 23h
kube-system kube-scheduler-master 1/1 Running 2 23h
kube-system metrics-server-6fbfb84cdd-cs7hk 1/1 Running 0 23h
kube-system monitoring-grafana-69df66f668-87qkl 1/1 Running 0 23h
kube-system monitoring-influxdb-78d4c6f5b6-jk59b 1/1 Running 0 23h
kube-system weave-net-4qvhg 2/2 Running 0 23h
kube-system weave-net-6mg8p 2/2 Running 0 23h
kube-system weave-net-9bvqg 2/2 Running 0 23h
kube-system weave-net-z9mmz 2/2 Running 3 23h
monitoring custom-metrics-apiserver-7dd968d85-sw7fl 1/1 Running 0 22h
monitoring prometheus-7dff795b9f-6jpd5 1/1 Running 0 22h

[Stefan Prodan]

The Prometheus adapter keeps only some metrics and does renaming on them, please take a look here: https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/format.md

[jamyao]

Thanks @stefanprodan . According to document you mentioned, "container_network_receive_bytes_total " suppose to be kept by Prometheus adapter, am i correct? it's not the case for me, i couldnt see this metrics

[xyz@master ~]$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|jq .|grep network
"name": "pods/network_tcp_usage",
"name": "pods/network_udp_usage",

Thanks