Cause of `failed to send, dropping X traces to intake` error log

[Vedran Karačić]

Summary of problem

We keep seeing these errors in our DD logs every few days:

failed to send, dropping 1 traces to intake at http://<local_IP>:8126/v0.4/traces after 3 retries

I cannot find any other errors or anything useful on it in the logs or any DD metrics. I've looked at the source code and this line seems to be the error line but I cannot figure out why this could be happening. Is there any way I can figure out the cause of this?
From the sound of it it does not seem like a big issue but would like to confirm with an expert here.

Additionally we periodically see this error line in the logs:

According to the source code this should not be a higher than warning log level but it is an error in our case:
https://github.com/DataDog/dd-trace-py/blob/2.1/ddtrace/internal/writer/writer.py#L281

I'm sorry if these are stupid questions but I'm running out of options to find the cause here.

Which version of dd-trace-py are you using?

2.1

Which version of pip are you using?

22.3.1

[Munir Abdinur]

Hey @vkaracic,

Thanks for opening this issue.

From your log message I can see that you're sending traces to the /v0.4/traces endpoint. This is a legacy endpoint. Using /v0.5/ traces should improve the reliability of sending traces to the agent. To use the v05 endpoint set the following environment variable: DD_TRACE_API_VERSION=v0.5.

Your assessment is correct. This error is logged when the tracer fails to send traces to the agent. This is likely a networking issue but if you detect a bug in the trace-agent or tracer (ex: 400, 500, 501 status codes) feel free to reopen this issue.