Change Request: Distributed Tracing Support for FSPIOP API
mdebarros opened this issue · comments
Open API for FSP Interoperability - Change Request
Table of Contents
1. Preface
___Traceability of API requests within a deployed Mojaloop implementation is paramount to the Hub Operators ability to successfully and effectively provide operational support in issue resolution. This change-request is to
1.1 Change Request Information
| Requested By | Miguel de Barros, ModusBox |
| Change Request Status | Draft ☒ / In review ☐ / Approved ☐ / Rejected ☐ |
| Approved/Rejected Date | |
1.2 Document Version Information
| Version | Date | Author | Change Description |
|---|---|---|---|
| 1.0 | 2020-12-10 | Miguel de Barros | Initial version. |
2. Problem Description
___2.1 Background
The Mojaloop product is implemented using several distributed services, this implementation pattern being known as a micro-service architecture. A result is that a single transaction may be processed or handled by multiple services within the Mojaloop stack. This introduces a challenge for operational monitoring as it is difficult to trace or monitor a transaction as it flows through each of these distributed services.
2.2 Current Behaviour
At present, there is a tracing solution implemented in the current Mojaloop stack as part of the Event Framework.
This requires several non-standard FSPIOP specification headers which contain the trace-context (ref: Event Framework - Tracing Design - 4.1.1 W3C HTTP Headers) to be:
- processed by the Hub, and
- forwarded by participants on call-back requests.
The result being that FSPs are not currently forwarding tracing-context headers FSPIOP call-backs. This means that traces are currently broken into each of the FSPIOP legs of the PayerFSP request and PayeeFSP call-back.
Let's take Transfers as an example:
- Mojaloop receives a Transfer Prepare Request, assuming there is no trace-context header from the PayerFSP, a trace-context is created by Mojaloop ML-API-Adapter. This trace-context is propagated through each of the micro-services and is utilised for monitoring purposes until the notification has been sent to the PayeeFSP.
- When receiving the Transfer Fulfil Callback from the PayeeFSP, which does not contain the forwarded trace-context headers, Mojaloop will not have the necessary context to continue the trace from the prepare-leg (i.e. point 1 above). As such a new context is created from the point of receiving the fulfilment and until the notification is sent to the PayerFSP.
Thus we end up with two distinct trace-context for each leg of the Transfer.
2.3 Requested Behaviour
The W3C Tracing headers as describes in following design document: Event Framework - Tracing Design - 4.1.1 W3C HTTP Headers, are included as part of the FSPIOP specification with functional information on how the Mojaloop Switch and Participants should handle or process them appropriately.
Note: Participants are not required to do any additional processing of these tracing headers, but at a minimum, they should forward the tracing headers as part of FSPIOP call-back. This will ensure that there is an unbroken transactional trace from the switch to the FSP.
Let's take Transfers as an example:
- Mojaloop receives a Transfer Prepare Request, assuming there is no trace-context header from the PayerFSP, a trace-context is created by Mojaloop ML-API-Adapter. This trace-context is propagated through each of the micro-services and is utilised for monitoring purposes until the notification has been sent to the PayeeFSP.
- When receiving the Transfer Fulfil Callback from the PayeeFSP, the trace-context is included as a header, Mojaloop will then process this trace-context and continue the trace from the prepare-leg (i.e. point 1 above). As such the same context is now used from the point of receiving the prepare and until the fulfilment notification is sent to the PayerFSP.
Thus we end-up with single trace-context for both legs of the Transfer.
Example demo using simulated FSPs which support the tracing standard: https://youtu.be/JGL5HfYi0wE.
3. Proposed Solution Options
___Refer to the W3C's Trace Context Level 2 Document for more detail on the standard that is currently supported by the Event Framework.
However, there are other tracing standards that can be considered such as Zipkin's B3 Propagation Design.
We propose to add these as optional headers so that the changes are backwards compatible so that Schemes that implement/deploy Mojaloop can decide on their usage (making them mandatory or optional for their Scheme).
Thank you @mdebarros.
I'm OK with adding the W3C trace headers to the FSPIOP API as optional headers, making these the recommended way of adding traces. I don't want to make them mandatory to implement as would be the implication of the following part:
Participants are not required to do any additional processing of these tracing headers, but at a minimum, they should forward the tracing headers as part of FSPIOP call-back. This will ensure that there is an unbroken transactional trace from the switch to the FSP.
The FSPIOP API is a generic standard which could be used in a scheme where another Switch than the Mojaloop switch is used, or in a bilateral setup without a Switch, which is why I do not want these headers to be mandatory in the API. It should rather be the scheme rules that decides that these trace headers must be used within the scheme. Some schemes might choose to use other tracing mechanisms, or none at all. There should not be a requirement on implementers of the API to support them if they will not be used within a scheme as they are in no way required for the API to work.
Kind regards,
Henrik
Thanks @ehenrka , @jmssebunnya - agree with you; the goal is to add these to the Spec as optional and let Schemes decide on whether to make them required for that Scheme or not
@mdebarros and @elnyry-sam-k, as there have been no more comments regarding the change request, can you please go ahead and make a solution proposal containing the actual changes to the specification?
Thanks @ehenrka , sure
I have proposed in the last SIG meetings to add the optional HTTP headers for tracing support to the existing version 1.1 of the API. This suggestion is related to my approved suggestion back in 2019 (email subject Open API for FSPIOP: Versioning of document and resources), where we decided to separate the document version and versions for different resources in the API.
As the tracing HTTP headers (traceparent, tracestate) are optional and does not need to be handled if they are received by an FSP, we should be able to safely add them to a new patch document version (1.1.1) of the API Definition document without any impact for existing implementations. There will be no changes to any of the resources of the API.
The suggestion then is to:
- Create a new table in Section 3.2.1.1 (HTTP Request Header Fields) to describe optional HTTP header fields, as the current Table 1 only describes HTTP headers that must be implemented.
- Add tracing HTTP headers (
traceparent,tracestate) to the new table for optional HTTP headers and add reference to the Distributed Tracing Support for OpenAPI Interoperability document for detailed information. - Update document version to new patch version 1.1.1, describe changes in Section 1.2
Thanks Henrik (@ehenrka ) for the recommendations. We should be able to implement these and conclude this issue
Please review #95, which contains the changes described in #71 (comment).
Closing as supported since version 1.1.1