Bug: found a segment with no parent error using get_trace_tree
eoinsha opened this issue · comments
Expected Behaviour
Expected to get a trace by a valid ID using iatk.get_trace_tree
Current Behaviour
Received the error:
def _invoke_iatk(self, payload: Payload, caller: str=None) -> dict:
input_data = payload.dump_bytes(caller)
LOG.debug("payload: %s", input_data)
stdout_data = self._popen_iatk(input_data)
jsonrpc_data = stdout_data.decode("utf-8")
data_dict = json.loads(jsonrpc_data.strip())
# Error is only returned in the response if one happened. Otherwise it is omitted
if data_dict.get("error", None):
message = data_dict.get("error", {}).get("message", "")
error_code = data_dict.get("error", {}).get("Code", 0)
> raise IatkException(message=message, error_code=error_code)
E aws_iatk.IatkException: error building trace tree: failed to build trace tree 1-655dc76c-0cb3bd391beaacd86aea4146 with error: found a segment 7577700a7a59d778 with no parent
venv/lib/python3.10/site-packages/aws_iatk/__init__.py:586: IatkException
Code snippet
# Create an order with the Order Service
response = requests.post(api_endpoint)
order_id = response.json()['orderId']
assert order_id is not None
trace_id = response.headers['x-amzn-trace-id']
# Check whether the delivery service received the Order.Created event
def assertion(event: str):
payload = json.loads(event)
assert payload['detail-type'] == "Order.Created"
assert payload['detail']['data']['orderId'] == order_id
assert iatk.wait_until_event_matched(delivery_listener_id, assertion)
# Check the trace to ensure all components participated
trace_tree = iatk.get_trace_tree(
tracing_header=trace_id,
).trace_treeThe full test, along with the underlying application is available in the public cross-account-eventbridge repo: https://github.com/fourTheorem/cross-account-eventbridge/blob/e1d05352871d9a57205f05cf0f103bc9a7e2579a/test/integration/test_events.py
The full raw trace is:
{
"Id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"Duration": 11.902,
"LimitExceeded": false,
"Segments": [
{
"Id": "586db22b30e91e37",
"Document": {
"id": "586db22b30e91e37",
"name": "OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1",
"start_time": 1700644716.489,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644718.73,
"http": {
"response": {
"status": 200
}
},
"aws": {
"request_id": "d5da98b1-9cb3-4376-910e-ab07acd4323c"
},
"origin": "AWS::Lambda",
"resource_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
}
},
{
"Id": "6f2d9e38746beb40",
"Document": {
"id": "6f2d9e38746beb40",
"name": "DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
"start_time": 1700644719.4756236,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644725.8744302,
"parent_id": "69708f66a8ffb766",
"aws": {
"account_id": "123456789012",
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
"cloudwatch_logs": [
{
"log_group": "/aws/lambda/DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
}
],
"resource_names": [
"DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
]
},
"origin": "AWS::Lambda::Function",
"subsegments": [
{
"id": "969fa961524b7008",
"name": "Invocation",
"start_time": 1700644719.4759119,
"end_time": 1700644725.8365042,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
},
"subsegments": [
{
"id": "05ca73ee5a1300ae",
"name": "## index.handleOrderCreated",
"start_time": 1700644719.675,
"end_time": 1700644725.833,
"annotations": {
"ColdStart": true,
"Service": "service_undefined"
},
"subsegments": [
{
"id": "9161d1559a2f1a93",
"name": "Events",
"start_time": 1700644725.274,
"end_time": 1700644725.775,
"http": {
"response": {
"status": 200,
"content_length": 85
}
},
"aws": {
"retries": 1,
"region": "eu-west-1",
"operation": "PutEvents",
"request_id": "5db2bd0a-2848-4dc1-8b80-96ec220682cc"
},
"namespace": "aws"
}
]
}
]
},
{
"id": "cbc333120d1e1f35",
"name": "Initialization",
"start_time": 1700644718.908828,
"end_time": 1700644719.4746127,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
}
},
{
"id": "aa3a5eef54370180",
"name": "Overhead",
"start_time": 1700644725.8365796,
"end_time": 1700644725.8569787,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
}
}
]
}
},
{
"Id": "7577700a7a59d778",
"Document": {
"id": "7577700a7a59d778",
"name": "DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
"start_time": 1700644718.67,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644718.688,
"parent_id": "052266fc17994499",
"http": {
"response": {
"status": 202
}
},
"aws": {
"request_id": "0393ceaf-b102-4a20-a222-aca3994e9ed2"
},
"origin": "AWS::Lambda",
"resource_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
"subsegments": [
{
"id": "69708f66a8ffb766",
"name": "Attempt #1",
"start_time": 1700644718.744,
"end_time": 1700644725.837,
"http": {
"response": {
"status": 200
}
}
},
{
"id": "063e9d55abaa3a6c",
"name": "Dwell Time",
"start_time": 1700644718.67,
"end_time": 1700644718.744
}
]
}
},
{
"Id": "12e382f024e088c4",
"Document": {
"id": "12e382f024e088c4",
"name": "OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
"start_time": 1700644726.9018965,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644728.3905344,
"parent_id": "0d091c65039d1a08",
"aws": {
"account_id": "123456789012",
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
"cloudwatch_logs": [
{
"log_group": "/aws/lambda/OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
}
],
"resource_names": [
"OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
]
},
"origin": "AWS::Lambda::Function",
"subsegments": [
{
"id": "56b9a1dcef42ae06",
"name": "Invocation",
"start_time": 1700644726.9103296,
"end_time": 1700644728.3306756,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
},
"subsegments": [
{
"id": "aec1dd4752f933be",
"name": "## index.handleDeliveryUpdate",
"start_time": 1700644727.031,
"end_time": 1700644728.271,
"annotations": {
"ColdStart": true,
"Service": "service_undefined"
},
"metadata": {
"service_undefined": {
"index.handleDeliveryUpdate response": {
"createdAt": 1700644717444,
"orderId": "4233f354-cace-4814-8505-49401cd88709",
"deliveredAt": 1700644724681,
"updatedAt": 1700644727033
}
}
},
"subsegments": [
{
"id": "34803fa283888039",
"name": "Events",
"start_time": 1700644727.711,
"end_time": 1700644728.232,
"http": {
"response": {
"status": 200,
"content_length": 85
}
},
"aws": {
"retries": 1,
"region": "eu-west-1",
"operation": "PutEvents",
"request_id": "32c502fa-3bb1-4c96-acae-9d05d335c6c7"
},
"namespace": "aws"
}
]
}
]
},
{
"id": "a33f1b345563af1b",
"name": "Overhead",
"start_time": 1700644728.3307269,
"end_time": 1700644728.390205,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
}
},
{
"id": "d1bfbc8caab0216a",
"name": "Initialization",
"start_time": 1700644726.179668,
"end_time": 1700644726.8985534,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
}
}
]
}
},
{
"Id": "02d84a171c2862a8",
"Document": {
"id": "02d84a171c2862a8",
"name": "OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1",
"start_time": 1700644717.2487402,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644718.725041,
"parent_id": "586db22b30e91e37",
"aws": {
"account_id": "123456789012",
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1",
"cloudwatch_logs": [
{
"log_group": "/aws/lambda/OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
}
],
"resource_names": [
"OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
]
},
"origin": "AWS::Lambda::Function",
"subsegments": [
{
"id": "5e807ca74115efe6",
"name": "Invocation",
"start_time": 1700644717.2490065,
"end_time": 1700644718.7037337,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
},
"subsegments": [
{
"id": "d1fcd2dcb9be11a3",
"name": "## index.handleOrderCreate",
"start_time": 1700644717.386,
"end_time": 1700644718.645,
"annotations": {
"ColdStart": true,
"Service": "service_undefined"
},
"metadata": {
"service_undefined": {
"index.handleOrderCreate response": {
"body": "{\"orderId\":\"4233f354-cace-4814-8505-49401cd88709\",\"createdAt\":1700644717444}",
"statusCode": 201
}
}
},
"subsegments": [
{
"id": "17ad530aafabaace",
"name": "Events",
"start_time": 1700644718.084,
"end_time": 1700644718.585,
"http": {
"response": {
"status": 200,
"content_length": 85
}
},
"aws": {
"retries": 1,
"region": "eu-west-1",
"operation": "PutEvents",
"request_id": "55cde20c-ebdf-48e3-a303-00611c9954c4"
},
"namespace": "aws"
}
]
}
]
},
{
"id": "20ae235dd047c173",
"name": "Overhead",
"start_time": 1700644718.7042577,
"end_time": 1700644718.7248337,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
}
},
{
"id": "2f51077ae561e8d7",
"name": "Initialization",
"start_time": 1700644716.6127412,
"end_time": 1700644717.247352,
"aws": {
"function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
}
}
]
}
},
{
"Id": "7da2c8de724ccfd4",
"Document": {
"id": "7da2c8de724ccfd4",
"name": "OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
"start_time": 1700644725.921,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644725.928,
"parent_id": "04a146e43a6f072a",
"http": {
"response": {
"status": 202
}
},
"aws": {
"request_id": "9a0fe80e-3ffe-4de1-ad65-d5c127e34af2"
},
"origin": "AWS::Lambda",
"resource_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
"subsegments": [
{
"id": "0d091c65039d1a08",
"name": "Attempt #1",
"start_time": 1700644725.981,
"end_time": 1700644728.331,
"http": {
"response": {
"status": 200
}
}
},
{
"id": "505646f1d675b542",
"name": "Dwell Time",
"start_time": 1700644725.921,
"end_time": 1700644725.981
}
]
}
},
{
"Id": "04a146e43a6f072a",
"Document": {
"id": "04a146e43a6f072a",
"name": "Events",
"start_time": 1700644725.274,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644725.775,
"parent_id": "9161d1559a2f1a93",
"inferred": true,
"http": {
"response": {
"status": 200,
"content_length": 85
}
},
"aws": {
"retries": 1,
"region": "eu-west-1",
"operation": "PutEvents",
"request_id": "5db2bd0a-2848-4dc1-8b80-96ec220682cc"
},
"origin": "AWS::Events"
}
},
{
"Id": "3fa030f40a45d7f0",
"Document": {
"id": "3fa030f40a45d7f0",
"name": "Events",
"start_time": 1700644727.711,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644728.232,
"parent_id": "34803fa283888039",
"inferred": true,
"http": {
"response": {
"status": 200,
"content_length": 85
}
},
"aws": {
"retries": 1,
"region": "eu-west-1",
"operation": "PutEvents",
"request_id": "32c502fa-3bb1-4c96-acae-9d05d335c6c7"
},
"origin": "AWS::Events"
}
},
{
"Id": "052266fc17994499",
"Document": {
"id": "052266fc17994499",
"name": "Events",
"start_time": 1700644718.084,
"trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
"end_time": 1700644718.585,
"parent_id": "17ad530aafabaace",
"inferred": true,
"http": {
"response": {
"status": 200,
"content_length": 85
}
},
"aws": {
"retries": 1,
"region": "eu-west-1",
"operation": "PutEvents",
"request_id": "55cde20c-ebdf-48e3-a303-00611c9954c4"
},
"origin": "AWS::Events"
}
}
]
}
Possible Solution
No response
Steps to Reproduce
- Deploy the cross-account-eventbridge application using CDK (a single account ID can be used for all services for simplicity)
- Run the CodePipeline to deploy the full application
- Run the integration test:
pytest ./test/integration/test_events.pyVersion used
0.1.0
Hey @eoinsha, thanks for reporting. It seems that at the time of fetching the traces, some segments were still missing. Looking at the raw data, it seems that the segment mentioned does have a parent and this is likely due to latency while getting the trace data.
Looks like you've already tried switching to retry_get_trace_tree_until. Has this worked for you?
@mildaniel the same issue occurred with retry_get_trace_tree_until but we were able to make the issue disappear with a sleep. This is surprising since the documentation doesn't seem to require a sleep for this call.
Hi @eoinsha, thanks for raising the issue. Using retry_get_trace_tree_until is the intended way to tackle latency issue like your case. However, it seems we missed to handle the error "found a segment ... with no parent" as a "retry-able" error that retry_get_trace_tree_until would retry. We will fix it.