Failed price fetching of tokens - Change the way we handle fetching token prices
divine-comedian opened this issue · comments
Currently we fetch the prices of tokens via RPC calls @geleeroyale correct me if I'm wrong...
This can cause heavy loads and when an RPC fails users get errors, we should find a better way to fetch prices.. Particularly on the donation page of the dapp when a user is selecting the token to donate with.
@mhmdksh and @aminlatifi please deep dive and delegate subtasks here as needed. This seems like a P1
as mentioned before I had a look at options together with MoeShehab and we found covalents unified api to be exactly what we need.
Hey guys, I want to push for what myself and @geleeroyale were preaching for some time now and introduce fetching token prices through Covalent using a single API Endpoint
This approach saves us tones of headaches and makes Blockchain querying one step closer to normal development querying.
We now have a Covalent Account with an API Key. I think @brichis is working with them on a grant to support non-profit orgs
To share something tangible. You can check the below example for fetching token prices by contract addresses using Covalent.
The below example fetches the current GIV (1) price using it's (2) contract address and our covalent (3) API Key in (4) USD on the (5) ETH Mainnet Chain
Through Curl:
curl -X GET https://api.covalenthq.com/v1/pricing/historical_by_addresses_v2/eth-mainnet/USD/0x900db999074d9277c5da2a43f252d74366230da0/ -H 'Content-Type: application/json' -u {YOUR_API_KEY}: | jq
Result:
{
"data": [
{
"contract_decimals": 18,
"contract_name": "Giveth",
"contract_ticker_symbol": "GIV",
"contract_address": "0x900db999074d9277c5da2a43f252d74366230da0",
"supports_erc": [
"erc20"
],
"logo_url": "https://logos.covalenthq.com/tokens/1/0x900db999074d9277c5da2a43f252d74366230da0.png",
"update_at": "2024-01-12T12:01:19.003010206Z",
"quote_currency": "USD",
"prices": [
{
"contract_metadata": {
"contract_decimals": 18,
"contract_name": "Giveth",
"contract_ticker_symbol": "GIV",
"contract_address": "0x900db999074d9277c5da2a43f252d74366230da0",
"supports_erc": [
"erc20"
],
"logo_url": "https://logos.covalenthq.com/tokens/1/0x900db999074d9277c5da2a43f252d74366230da0.png"
},
"date": "2024-01-12",
"price": 0.008857863,
"pretty_price": "$0.01"
}
],
"items": [
{
"contract_metadata": {
"contract_decimals": 18,
"contract_name": "Giveth",
"contract_ticker_symbol": "GIV",
"contract_address": "0x900db999074d9277c5da2a43f252d74366230da0",
"supports_erc": [
"erc20"
],
"logo_url": "https://logos.covalenthq.com/tokens/1/0x900db999074d9277c5da2a43f252d74366230da0.png"
},
"date": "2024-01-12",
"price": 0.008857863,
"pretty_price": "$0.01"
}
]
}
],
"error": false,
"error_message": null,
"error_code": null
}
This is a simple method using curl, But there is a lot of other methods that can give us really advanced results, more info here: https://www.covalenthq.com/docs/api/base/get-logs/
It also supports to many different languages: Like javascript, typescript, ruby, python
@aminlatifi We can do more complex stuff with this, I think the next logical step is to do a POC with a PR in our frontend. As this request was blazing fast, and I'm curious how this will affect our dapp performance overall. Would appreciate your feedback
Currently we fetch the prices of tokens via RPC calls @geleeroyale correct me if I'm wrong...
This can cause heavy loads and when an RPC fails users get errors, we should find a better way to fetch prices.. Particularly on the donation page of the dapp when a user is selecting the token to donate with.
We have different approaches for fetching prices based on the token. @MohammadPCh
- GIV: fetch from subgraph
- Solana: fetch from coingecko (by symbol)
- Other EVM Tokens: Fetch from conigecko by contract address and network ID
In this case, it's better to investigate what was the bug by looking at the error details. We may be able to fix it with less effort. @MohammadPCh Can you look at this or nominate one FE person to do it?
@mhmdksh As the queries are sent directly from clients, putting the secret key in the frontend app is not a good idea. So, we have to hide it behind an API. I cannot estimate the number of times it will be called, but overall that service is great. I am pretty sure it can be a promising alternative for price fetching on our backend (impact-graph)
@aminlatifi pls work on this next as part of Tech Debt for Token Price Fetching Tech Debt. Given we are integrating with more chains everyday, our api for token fetching needs to support more chains, should use caching on BE and polling between several free accounts on BE.
We are going to rely on coingecko for more tokens (most of the tokens after) as it supports most of the tokens and is almost a reliable source.
@alireza-sharifpour has prepared a list of coingecko IDs for those tokens missing this field data.
I just tested the CovalentHQ service. It has a handy API that supports token history by network ID and contract address as parameters. But it couldn't return the price value for a couple of tokens I tested.
Following up on @aminlatifi 's request for an analysis of CovalentHQ vs Coingecko vs RPC calls. I did not compare subgraphs as they should be avoided at all cost in my opinion because they are expensive, we have massive problems with their uptime and its frankly just overkill to fetch GIV price with a subgraph.
My TLDR recommendation is:
- stop using subgraphs in as many places as possible (token price fetching needs to go ASAP)
- Use UnifiedAPI for any token that is represented on mainnet
- Keep a list of unsupported tokens we would like to see with Covalent and open a communicatioon channel
- Use CoingeckoAPI for everything that is on Solana, ETC or other obscure chains. Covalent support might come sooner than we think, but because the requests are standardized it should be no problem to hop between assets/chains.
Here is my full analysis:
@aminlatifi - What's the status of this issue? Do we still require research or have we implemented using coingecko already?
@aminlatifi - What's the status of this issue? Do we still require research or have we implemented using coingecko already?
We have solved the issue so far with Coingecko, but It's worth doing research after we handle all p0 issues. Kay have prepared a great report that I haven't looked at it deeply yet!
Please prioritize completing this @aminlatifi @geleeroyale
@jainkrati we need to decide on a course of action. See my report and recommendations above.
Implementation depends on Dev work still - it's only my recommended approach.
Maybe worth mentioning I have good friend who works at CovalentHQ if you decide to go that direction - might be able to negotiate something regarding our sub.
Assuming we move in this direction do we know if this will reduce our running costs in other places? I noticed for example we pay for subs w/ Infura, QuickNode and Alchemy. However I don't have a good grasp on what we pay for with our subgraph services.
@geleeroyale Thank you for the comprehensive report. I concur that Covalent presents an exciting prospect with promising prospects, particularly in terms of its high data availability, reliability, and potential for enhanced precision.
However, considering our current price issue, it seems that fully integrating with Coingecko has effectively addressed this concern. Coingecko's user-friendly interface, and its fair enough stability, and reliability make it a practical choice, requiring minimal effort to add new tokens that can be easily handled by PMs and devs. Therefore, my suggestion is to maintain our current approach with Coingecko unless any issues arise, while also monitoring Covalent for potential integration opportunities in the future.
Thank you @aminlatifi - meanwhile something else came up where we could use covalent, please check out Mohammads inquiry and my responses to an issue with gnossiscan yesterday: https://discord.com/channels/679428761438912522/955413352224555069/1221414128543993926
(I did not want to add an extra issue even tho this is not related to prices, its related to the requests we make. It is probably best we open an epic about examining and optimising our requests if we decide we want to have a sprint that concerns itself with performance and stability of the DApp)
https://discord.com/channels/679428761438912522/955413352224555069/1221414128543993926
I think you are following up on that issue in a separate thread, I think we have reached to a conclusion on this and can put it in done stage.
@aminlatifi - Is there anyway to test the new implementation of coingecko price fetching on the dapp?
@aminlatifi - Is there anyway to test the new implementation of coingecko price fetching on the dapp?
@maryjaf knows the answer better than me. In donate modal and settled donation price that should be visible btw.
@aminlatifi so to understand with this coingecko flow
an admin should be able to add a new token to the front-end using the AdminJS - we add the coingecko id to the appropriate field as well as any other details of the new token..
then the FE price should be filled automatically once this is saved on the adminJS? No other dev work needed?
and for handling BE prices is it the same?
@aminlatifi so to understand with this coingecko flow
an admin should be able to add a new token to the front-end using the AdminJS - we add the coingecko id to the appropriate field as well as any other details of the new token..
then the FE price should be filled automatically once this is saved on the adminJS? No other dev work needed?
and for handling BE prices is it the same?
Yeah, it works as you described. Both FE and BE use the coingeckoId field in the token entity.
@maryjaf - Can you test this one out
On staging AdminJS we should either add a new token or find a token with no coingecko id
if it is an existing token:
- check the price does not show on the front-end when previewing a donation in that token
- add the missing coingecko id for this token, hit save
- test again making a donation - the preview USD amount should match the dollar amount as shown on coingecko, referenced by its id
to find the coingeckoId you can find it in the URL of the token on
https://www.coingecko.com/
in the image above the coingeckoId for example is solana
if it is a new token then test using only steps 2 & 3
@maryjaf - Can you test this one out
On staging AdminJS we should either add a new token or find a token with no coingecko id
if it is an existing token:
- check the price does not show on the front-end when previewing a donation in that token
- add the missing coingecko id for this token, hit save
- test again making a donation - the preview USD amount should match the dollar amount as shown on coingecko, referenced by its id
to find the coingeckoId you can find it in the URL of the token on https://www.coingecko.com/
in the image above the coingeckoId for example is
solanaif it is a new token then test using only steps 2 & 3
I did not find a token that does not have "coingeckoId" value
I removed the "coingeckoId": matic-network for the Matic Token and checked the price in front and the usd value is shown without coingeckoId
after that I add again "coingeckoId": matic-network
Are these steps correct or must it be a token for which "coingeckoId" field was not set from the beginning
@aminlatifi @divine-comedian
@maryjaf Let me describe the logic of price fetching on the FE and BE. What's implemented in staging env at the time of writing:
FE:
- If token
coingeckoIdis available, the token price is fetched from coingecko using this field value - Otherwise, the FE app tries to resolve the token price again with coingecko API this time by a different endpoint. It uses another endpoint accepts
Chain IDandToken Address. This endpoint is less versatile than the former one and may not support some chain ID and token address tuples. If you want this endpoint to fail to find the token price, go for a bridged token, e.g. GIV on optimism?! I am not sure check with the endpoint
https://api.coingecko.com/api/v3/simple/token_price/${chain}?contract_addresses=${tokenAddress}&vs_currencies=usd
BE
- If the token is a stable coin, it returns 1
- Otherwise, if it's
GIVit uses https://givback.develop.giveth.io/api-docs/#/default/get_givPrice - Otherwise, if it has
cryptoCompareId, it uses cryptocompare service - Otherwise, if it has
coingeckoId, it uses coingecko service - If none of the above works, it uses monoswap
@maryjaf I think one way to test BE price fetching is deploying custom tokens yourself. Just deploy a simple ERC20 token yourself and add it to the dashboard with whatever flags and coingecko id property. Let me know if you want to go in that direction and I will help you to deploy yourself, it's not hard.
Thanks @aminlatifi for the detailed explanation
Yeah I couldn't do a exact test by above steps If you have a time we check this together
This issue has been resolved