[META] Performance Benchmark for TGE
fredo opened this issue · comments
In order to fulfill the criteria we need to provide test results about the performance of the payments in speed.
The criteria are stated as follows:
For baseline performance, we expect to have 2 mediators and the payment will be A-B-C-D payment where A is the initiator, D is the target and B, C are mediators that are well connected hubs. This setup is chosen because we expect the initiator of transfer to optimize for lower fees. Hubs will act as service providers in the network by providing liquidity and mediating transfers for fees. In the steady state of the network, we expect hubs to be localized to facilitate transfers within their respective regions. Edge nodes are seen as users who want to use the network for transfers by leveraging localized hubs as their first hop. And furthermore this localized hub connects to another localized hub connected to the intended target edge node.
Example geographical locations (same economic region such as EU):
A is the initiator who is in France and D is the target who is in Germany where B and C are well connected hubs / mediators located in France and/or Germany respectively.
How is the performance measured?
Time between the initiator sending the locked transfer and the target receiving the unlocked balance proof. Time to get the route from the pathfinding service has been excluded from this measurement.
We will produce two distinct reports for two distinct setups, where we present the performance benchmarks over several transfers.
Setup-1 with Light Client: Initiator A and Target D are mobile clients using the Light Client DApp. Mediators B and C are server based nodes running in the cloud.
Setup-2 with Python Client: Initiator A and Target D are python clients using the WebUI or Client CLI. Mediators B and C are server based nodes running in the cloud.
We need to set up this topology and monitor the payment speed. A separation what would be of interest is the time of the PFS route request and the actual payment.
testnet or mainnet?
Things that came up during discussion of this issue:
- For the overall measurement of the payment performance it should be enough to time the total duration of the REST-API calls
- If we find that we need to do more in-depth performance analysis we probably should add instrumentation to the client. I'd suggest to use OpenTracing
testnet or mainnet?
Do we have an answer for this yet?
testnet or mainnet?
Do we have an answer for this yet?
In scope of TGE benchmarking, this should not matter: performance is measured for happy case mediated transfers from initiation of the payment (post route retrieval) to reception of the unlocked balance by the target. There are no on-chain operations involved.
In order to simplify the setup I would therefore stay on testnet.