Failed to append profile to pyroscope, failing with pyroscope write error: status 422
sukumardandu opened this issue · comments
We are working on pushing profiles to pyroscope with alloy agent and we are seeing errors when alloy agent is trying to push those profiles to pyroscope (grafana cloud)
We are only seeing this errors when using Alloy Agent versions from 1.7.0 to 1.7.4, this issue is not being reproduced when using Alloy agent version v1.6.1
Additional Context
Error in the Alloy Agent pod logs:
ts=2025-03-18T18:08:45.086648898Z level=error msg="Failed to append profile" component_path=/ component_id=pyroscope.receive_http.http_receiver appendable=0 err="pyroscope write error: status 422"
ts=2025-03-18T18:08:45.0867745Z level=debug msg="POST /ingest?name=pyroscope-fast-slow-dotnet%7B%7D&from=1742321310000&until=1742321325000&spyName=dotnetspy&spyVersion=0.9.4 (422) 81.277644ms" component_path=/ component_id=pyroscope.receive_http.http_receiver
ts=2025-03-18T18:08:50.144997282Z level=error msg="Failed to append profile" component_path=/ component_id=pyroscope.receive_http.http_receiver appendable=0 err="pyroscope write error: status 422"
ts=2025-03-18T18:08:50.145090044Z level=debug msg="POST /ingest?from=1742321270&name=pyroscope-nodejs-pprof-%7Bapp%3Dpyroscope-nodejs-pprof%2Cregion%3D%2Ccluster%3D%7D&spyName=nodespy&until=1742321330&sampleRate=17 (422) 60.098382ms" component_path=/ component_id=pyroscope.receive_http.http_receiver
To Reproduce
Steps to reproduce the behavior:
Deploy Grafana/Alloy Helm release on K8 cluster with versions v1.7.0 and repeat the process for version until v1.7.4
Deploy a sample application to kubernetes
dotnet: https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/dotnet/fast-slow
nodejs: https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/nodejs/express
Expected behavior
All Profiles show up in pyroscope
Actual: No profiles being showed up and being failed with 422 error
Environment
Infrastructure: Kubernetes
Deployment tool: helm
This issue is also reported here for reference, with some additional context: grafana/pyroscope-go#144
This should have been fixed in grafana/alloy#2918
Any chance you can double check with a devel image in your test environment?
https://hub.docker.com/layers/grafana/alloy-dev/v1.8.0-devel-8625747/images/sha256-671fc64299db64926af8f9b621ad5ae3936b402c4220a6c77859979b427ac2fa
@korniltsev I tried the linked image and it still fails with the duplicate label error.
Ok I try to explain what is happening: Since Alloy 1.7, pyroscope.receive_http when using the ingest endpoint (like most SDKs do, extracts the application name (also called __name__) and copies it into the service_name label.
This behaviour has not changed, as this is wanted to we can rely on a service_name when relabelling. grafana/alloy#2918 does just additionally validate and handle duplicate labels sent (but as far as I know it never send duplicates).
Where in fact the duplication happens is on the backend (Pyroscope). All versions before #3951 did blindly copy the application name into the service_name even if it already existed. So it creates the duplication in the backend.
For Grafana Cloud, we underestimated the impact that had, so we did not trigger a manual release. That said we are rolling this out for Grafana Cloud in the next couple of hours.
For OSS users with hat problem, we will release a v1.13.0 very soon. You can also use grafana/pyroscope:weekly-f110-32faf96b1 in the meantime.
Sorry for the trouble