Components failing to get adresses for interface
nesspll opened this issue · comments
Contex: EKS IPv6
values config file
global:
reportingEnabled: false
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::258570154***:role/pyroscope-storage
pyroscope:
replicaCount: 1
cluster_domain: .cluster.local
extraEnvVars:
POD_IP:
valueFrom:
fieldRef:
fieldPath: status.podIP
extraArgs:
memberlist.bind-addr: "$(POD_IP)"
log.level: "debug"
ingester.enable-inet6: "true"
storage:
backend: s3
s3:
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
bucket_name: pyroscope-blocks
memberlist:
join_members:
- pyroscope-gossip-ring.pyroscope-new.svc.cluster.local:7946
bind_addr: "[::]"
bind_port: 7946
image:
repository: hub.docker.com/grafana/pyroscope
pullPolicy: IfNotPresent
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::258570154***:role/pyroscope-storage
server:
http_listen_network: "tcp"
http_listen_address: "[::]"
http_listen_port: 4040
grpc_listen_network: "tcp"
grpc_listen_address: "[::]"
grpc_listen_port: 9095
components:
querier:
kind: Deployment
replicaCount: 3
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 1
storage:
backend: s3
s3:
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
bucket_name: pyroscope-blocks
distributor:
kind: Deployment
replicaCount: 2
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 500m
storage:
backend: s3
s3:
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
bucket_name: pyroscope-blocks
lifecycler:
ring:
kvstore:
store: memberlist
extraEnv:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
tokens_file_path: /var/pyrscope/tokens/distributor-tokens.json
ingester:
kind: StatefulSet
replicaCount: 3
terminationGracePeriodSeconds: 600
storage:
backend: s3
s3:
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
bucket_name: pyroscope-blocks
resources:
limits:
memory: 4Gi
requests:
memory: 2Gi
cpu: 1
extraEnv:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
tokens_file_path: /var/pyrscope/tokens/ingester-tokens.json
lifecycler:
address: "${POD_IP}"
enable_inet6: true
interface_names:
- eth0
- lo
- v4if0
ring:
kvstore:
store: memberlist
compactor:
kind: StatefulSet
replicaCount: 3
terminationGracePeriodSeconds: 1200
storage:
backend: s3
s3:
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
bucket_name: pyroscope-blocks
persistence:
enabled: false
resources:
limits:
memory: 4Gi
requests:
memory: 2Gi
cpu: 1
lifecycler:
ring:
kvstore:
store: memberlist
extraEnv:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
tokens_file_path: /var/pyrscope/tokens/compactor-tokens.json
analytics:
reporting_enabled: false
minio:
enabled: true
image:
repository: quay.io/minio/minio
mcImage:
repository: quay.io/minio/mc
alloy:
enabled: false
Error in Ingester component:
caller=netutil.go:85 level=debug msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false
caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr="invalid IP" inf=eth0
caller=netutil.go:104 level=debug msg="ignoring address" ipAddr="invalid IP" inf=eth0
caller=netutil.go:90 level=warn msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"
caller=netutil.go:118 level=debug msg="detected IP address after looking up all configured interface names" ipAddr="invalid IP"
failed running pyroscope: error initialising module: version: no useable address found for interfaces [eth0 en0]
ts=2024-11-28T21:36:48.448324076Z caller=server.go:353 level=info msg="server listening on addresses" http=[::]:4040 grpc=[::]:9095ts=2024-11-28T21:36:48.449615341Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/admin auth=false gzip=truets=2024-11-28T21:36:48.451322992Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/api/swagger.json auth=false gzip=truets=2024-11-28T21:36:48.451370724Z caller=api.go:106 level=debug msg="api: registering route" methods=GET,POST,PUT,DELETE,HEAD,OPTIONS prefix=/api auth=false gzip=truets=2024-11-28T21:36:48.451385186Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/debug/fgprof auth=false gzip=truets=2024-11-28T21:36:48.451404633Z caller=api.go:106 level=debug msg="api: registering route" methods=GET prefix=/static/ auth=false gzip=truets=2024-11-28T21:36:48.45141978Z caller=api.go:106 level=debug msg="api: registering route" methods=GET prefix=/ui auth=false gzip=truets=2024-11-28T21:36:48.451433401Z caller=api.go:106 level=debug msg="api: registering route" methods=GET prefix=/assets/ auth=false gzip=truets=2024-11-28T21:36:48.451777831Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/memberlist auth=false gzip=truets=2024-11-28T21:36:48.452556139Z caller=netutil.go:85 level=debug msg="looking for addresses" inf="[eth0 en0]" inet6enabled=falsets=2024-11-28T21:36:48.452654829Z caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr="invalid IP" inf=eth0ts=2024-11-28T21:36:48.452671394Z caller=netutil.go:104 level=debug msg="ignoring address" ipAddr="invalid IP" inf=eth0ts=2024-11-28T21:36:48.45270517Z caller=netutil.go:90 level=warn msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"ts=2024-11-28T21:36:48.452723232Z caller=netutil.go:118 level=debug msg="detected IP address after looking up all configured interface names" ipAddr="invalid IP"failed running pyroscope: error initialising module: version: no useable address found for interfaces [eth0 en0]
Hi @nesspll!
I'm afraid there isn't a single option to enable IPv6. The following flags or corresponding options in the configuration need to be set:
-ingester.enable-inet6-distributor.ring.instance-enable-ipv6-store-gateway.sharding-ring.instance-enable-ipv6-compactor.ring.instance-enable-ipv6-overrides-exporter.ring.instance-enable-ipv6-query-scheduler.ring.instance-enable-ipv6
I also notices some inconsistencies in the values.yaml file you provided – it includes some settings that should be set in the Pyroscope's configuration (pyroscope.config) or passed as extra flags. Please refer to the chart docs for more details
The flags, should the be set under the section of pyroscope.extraArgs also can you point as an example to those inconsistencies, that should move to pyroscope.config section
Seeing the following error for the distributor component, even after adding in the code below (we tried multiple times/places to add the ipv6 code, seems like the 'interface' doesn't recognize ipv6?"
`ts=2024-11-29T11:16:59.44059492Z caller=netutil.go:118 level=debug component=distributor msg="detected IP address after looking up all configured interface names" ipAddr="invalid IP"
failed running pyroscope: error initialising module: distributor: failed to build distributors' lifecycler config: no useable address found for interfaces [eth0 en0]
ts=2024-11-29T11:16:59.440431689Z caller=netutil.go:85 level=debug component=distributor msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false
s=2024-11-29T11:16:59.440276625Z caller=log.go:245 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
`
###code
distributor:
kind: Deployment
replicaCount: 2
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
cpu: 500m
storage:
backend: s3
s3:
endpoint: s3.us-east-1.amazonaws.com
region: us-east-1
bucket_name: bt-sre-poc-pyroscope-blocks
ring:
kvstore:
store: memberlist
instance_enable_ipv6: true
instance_addr: $(POD_IP)
instance_port: 9095
instance_interface_names:
- eth0
extraEnv:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
tokens_file_path: /var/pyrscope/tokens/distributor-tokens.json
Surprisingly, the memberlist advertisement works. Also, we have managed to successfully deploy the compactors!
Does this mean only IPV4 is supported (and not Ipv6) and it will throw an error regardless of IPv6 assignment?
https://github.com/grafana/pyroscope/blob/main/pkg/util/net.go#L16
// GetFirstAddressOf returns the first IPv4 address of the supplied interface names, omitting any 169.254.x.x automatic private IPs if possible.
The pods themsevles seem to be getting IPv6 addresses.
pyroscope-ingester-0 0/1 CrashLoopBackOff 6 (4m59s ago) 11m 2600:1f18:5562:ae03:33d2::8 ip-10-48-4-6.ec2.internal <none> <none> pyroscope-ingester-1 0/1 CrashLoopBackOff 6 (4m34s ago) 10m 2600:1f18:5562:ae04:7a32::3 ip-10-48-4-98.ec2.internal <none> <none> pyroscope-ingester-2 0/1 CrashLoopBackOff 6 (4m47s ago) 10m 2600:1f18:5562:ae03:93b5::b ip-10-48-4-58.ec2.internal <none>
@cyriltovena , any chance you have an input on this one?
We're trying to demo this to our executive leadership and it would be great to have an answer on this one.
Appreciate it.
@ipversiongent you're right about GetFirstAddressOf. I believe there are several issues in Pyroscope that currently make using IPv6 impossible.
Let's aim to fix this by the next release. This is the first patch: #3731. I suspect other issues may arise once IPv6 is enabled. It would be incredibly helpful if you could test the patch and let us know if it resolves the issue for you.
Since we're running a sandbox deployment, we will try to provide feedback on our IPv6 deployment. Is the expecation to have the changes deployed in the new release v1.11?
Let's stay in touch and maybe address this issue methodically. Thank you for all your support and fast responses. Appreciate all your work.
Awesome! Thanks for your help!
We'll discuss this internally and carry out some integration testing. After that, if everything goes well, the fix will be included in our weekly build next Monday, available publicly (kind of release candidate). I'll reach out to you once everything is ready. Thanks again!
Hello Anton. On another note, unrelated to this, is there any place we can find deployment scenarios alongside instrumentation methods? I looked at Rob Chapman's book but the section on Pyroscope is fairly limited and purely theoretical - without any details on deployment and integration strategies.
Apologies for reaching out in this issue and possibly convoluting the conversation. Thank you again, and best of luck. Regards.
Hello @ipversiongent! We've released the f96 weekly build with the fix included – I hope this will address the problem you encountered.
is there any place we can find deployment scenarios alongside instrumentation methods? I looked at Rob Chapman's book but the section on Pyroscope is fairly limited and purely theoretical - without any details on deployment and integration strategies.
I apologize for the delay; I just noticed the comment.
As for the Pyroscope deployment scenarios: we run it in k8s ourselves and therefore I'd recommend using the helm chart. Depending on the availability and capacity requirements, you may prefer the microservice deployment (designed for HA and horizontal scalability) or start with a single-binary deployment and migrate later. If you have specific questions, I'd be happy to address them.
The client side requires more thorough analysis, though. I recommend starting from this page. In short, there are 3 ways:
- Instrument your applications with language-specific SDKs: gives you the full control over the profiling process and offers more capabilities tailored to the application language/runtime.
- Auto-instrumentation via Alloy:
- Go: profiles are pulled from the discovered targets via the http endpoint by Alloy (like Prometheus metrics). A perfect choice if you run Go applications in k8s/docker/VMs/bare-metal.
- Java: Alloy may be configured to discover Java processes and attach the profiler automatically. Good choice if you run Java applications in k8s/docker/VMs/bare-metal.
- eBPF: offers the quickest way to collect CPU profiles for compiled languages. Might not be ideal for interpreted languages like Python or Ruby.
- Auto-instrumentation via eBPF OpenTelemetry collector. Same as Alloy eBPF but integrated with OTel ecosystem. Experimental: many functions may be incomplete or not work as intended.