Discussion: SRA for 0x v2
dekz opened this issue · comments
Firstly thanks to all who implement the current SRA endpoints. It's great to have a standard way to discover these off-chain orders for use on-chain. Shared liquidity is something we strongly believe in. It brings new players into the ecosystem like wallets and automated traders.
I want to open the discussion of what features you believe should be added to the new SRA specification. We cannot sit down and accurately predict what is going to happen and how it will be used. So expect to see this SRA specification grow and change over time. We want this to be collaborative rather than descriptive so it's something Relayers want to add, rather than feel forced to add.
I fully expect Relayers to incorporate the SRA endpoint into custom automated trading SDK's for their Relay. The SRA Endpoint will just be one component in a trading SDK.
Some goals for the SRA Endpoint:
- Allow discovery of orders
- Allow discovery of token pairs being traded
- Be efficient in order discovery
- Satisfy the needs for fee recipients
- Allow 0x orders to be submitted to the Relayer
Both REST and Websocket specifications are part of the SRA endpoint. Websockets are more efficient than polling.
Some discussion initial points:
- [1] New order
- [2] Order Lifecycle updates
- [3] Partial Order default endpoint
- [4] Fee recipient list endpoint
- [5] networkId parameter
New Orders [1]
This is already covered in the current SRA spec in both REST and Websocket. Orders can be fetched via REST and Websocket snapshot. New orders are pushed via the Websocket. The only change here is the new order format schema.
Order Lifecycle updates [2]
This is blockchain state and can be fetched by watching for Contract events. It can also be synced via Order Watcher watching latest block height. This discussion is open for whether the SRA endpoint should support these Lifecycle events (cancel, partial fill, fully filled). These states can only be verified on-chain, if an update is sent through SRA the client will have to trust the relayer in its calculations. We believe this is best solved by watching the block chain or with order watcher, but feel free to voice your opinion.
Partial Order default endpoint [3]
This replaces the fee endpoint, where one would previously pass in the partial order and receive the fees and feeRecipient for this order. There is additional custom parameters a relayer may want to add (such as sender address). This endpoint would accept a partial order and the return result would represent an order with sender, feeRecipientAddress, makerFee and takerFee. This is then verified, signed and submitted to the Relayer.
Fee Recipient List endpoint [4]
When using multiple fee recipients it is useful for reporting and discovery to find out which relayer that belonged to. A relayer can return a list of all fee recipients they use. This is currently unverifiable and only useful after a transaction has occured.
networkId parameter [5]
We've seen usage of both networkId and subdomains to represent kovan rather than mainnet. We believe it would be best to standardise this into a networkId parameter which accepts an id representing the network (1 mainnet, 42 kovan etc). Default is mainnet.
Let me know if there are any other items you would like to discuss as well and I will summarise and add to this discussion.
This is blockchain state and can be fetched by watching for Contract events. It can also be synced via Order Watcher watching latest block height. This discussion is open for whether the SRA endpoint should support these Lifecycle events (cancel, partial fill, fully filled). These states can only be verified on-chain, if an update is sent through SRA the client will have to trust the relayer in its calculations. We believe this is best solved by watching the block chain or with order watcher, but feel free to voice your opinion.
I agree with this. Trusting other relayers to report correct events is probably not advised. If you pull in orders from other relayers, validate that order as you would one of your own orders from that point forward.
This replaces the fee endpoint, where one would previously pass in the partial order and receive the fees and feeRecipient for this order. There is additional custom parameters a relayer may want to add (such as sender address). This endpoint would accept a partial order and the return result would represent an order with sender, feeRecipientAddress, makerFee and takerFee. This is then verified, signed and submitted to the Relayer.
Not sure I understand this. Are we just talking about submitting an object that includes only the relevant fields, rather than requiring an entire order payload? Maybe just sharing the param/result schema for this would be enough of an explanation.
When using multiple fee recipients it is useful for reporting and discovery to find out which relayer that belonged to. A relayer can return a list of all fee recipients they use. This is currently unverifiable and only useful after a transaction has occured.
👍
We've seen usage of both networkId and subdomains to represent kovan rather than mainnet. We believe it would be best to standardise this into a networkId parameter which accepts an id representing the network (1 mainnet, 42 kovan etc). Default is mainnet.
The way we implement this is by using the networkId as a path parameter. I think it's a reasonable approach as it prevents the need to provide a networkId value in query/body params.
I would argue that relayers ought to report partial fills. I agree that this shouldn't be considered a canonical source of information, and clients should verify it before submitting an order to be fliled, but it can help them make a determination on whether an order is worth pursuing at all.
Say I load a UI, for example. That UI runs a query against the SRA and gets 20 orders back. Any decent UI is going to show the users how much is available from that order. If the SRA reported the partial fill amounts, the UI could render that information immediately. If the SRA doesn't report partial fill amounts, then the client application has to run 20 RPC calls to get the fill amounts before it can show the user how much is available from a given order. That adds a lot of computational cost to provide information that is probably already sitting in the same database record that the rest of the order information came from.
Absolutely, when the user clicks to submit an order it should validate that the order is still fillable, going beyond the partial fill amount to whether the maker has the necessary elements to complete the order. But if you're looking at a list of orders and making an initial determination of whether an order is of any interest, having that information available with the first API call instead of having to make 20 more RPC calls could save a lot of effort on orders that could be ruled out on that information alone.
How are things like makerTokenAddress and takerTokenAddress going to translate to the v2 SRA? It looks like the payload lists it as makerAssetData, while the query parameters still refer to makerTokenAddress.
I'm fairly certain people will want to be able to query for a variety of offers of ERC721 tokens from the same contract; for example - give me a list of all offers where the makerAsset is a CryptoStrikers card. At the same time, people may want to search for offers for a specific ERC721 asset; for example - Give me any offers where the takerAsset is CryptoStrikers card with ID 2846.
It looks like the current draft of the SRA doesn't address this.
makerTokenAddress and takerTokenAddress should remain the same in query parameters, but will change in the Signed Order structure (inside makerAssetData). While this will allow searching for orders of a specific type (CryptoStrikers cards) it doesn't fit the use case for individual token identification in the case of NFT.
It is possible to filter through this client side, but depending on the size of orderbooks this could be quite large. The Relayer is definitely in a position to optimise this query via a parameter. Thanks for raising this.
With regards to the SRA Websocket sending updates back, I can see Relayer SDK's wanting these updates without the addition of another channel. These SDK's trust the Relayer as they are created by the Relayer. This reduces the amount of total effort by clients as they don't have to watch and query for balance and allowance changes constantly.
My concern isn't only about a malicious endpoint sending bad orders, but misusing state updates on the channel. What is the finality or block depth for these updates? I think it's hard to prescribe these settings and having Relayers behave differently might be an issue.
I would prefer to see these state updates coming from the Relayer specific SDK which wraps SRA. Perhaps we could allow for non-official piggy back messages on the websocket channel if this reduces overhead.
Sorry, I didn't understand that discussion to be in the context of websockets events, I was thinking more in terms of reporting the current state as the relayer reports the order initially. That's mostly pertinent in terms of partial fills, as most of the other states would mean they wouldn't send the order in the first place.
And regarding searching for specific items, it seems like searching for makerAssetData and takerAssetData would at least let you in down the specific items you're interested in, and the relayer clearly already knows that since that's the value being returned in the payload.
assetData may not be the best query parameter as it has 0x domain specific fields and token specific fields. For example, it contains the ProxyId (0x specific) and the ERC721 Receiver Data. A user looking to buy Kitty A might not know the correct Receiver Data to construct a valid search.
API consumers already need to know a lot about the 0x domain to interact with the API. On the submission side, consumers need to be able to craft the assetData field to submit a field. They also need to know the available search parameters. I feel like this is a problem best addressed by documentation and tooling.
I could be open to alternative ways to search for specific assets, but I don't think it makes sense to make an NFT's token ID searchable independent of the contract address. And as far as the principle of least surprise goes, I think users would expect to be able to filter by the same parameters that the API returns in its result set. I think it would potentially be more confusing to users who aren't already using tooling or documentation if they had to search by parameters that aren't attributes of the result set, or that the field they just searched on isn't obviously in the data they got back.
I agree with @AusIV. We should be using assetData everywhere, queries included. SRA Clients like 0x Connect can abstract things away from the end user, but the assetData becomes a new unique identifier for assets.
There needs to be a websocket endpoint on each exchange that publishes every new order (regardless of the pair) and state-changes of these orders.
If this does not exist, the idea of seeing shared liquidity on 0x will be impossible. Exchanges/Users/Clients should not need to request the pairs they want to see updates for on a particular exchange, there should be an open socket publishing this information at all times.
assetData
@fabioberger @AusIV it's not possible to know assetData in use cases beyond basic ERC20. In ERC721 assetData encoding there is additional user data determined by the maker. The layout looks like so makerAssetData: [address][tokenId][length][receiverData][proxyId]. Where the address, tokenId, and proxyId are always constant with respect to the ERC721 token. The receiverData is arbitrary and can change even if the other parameters remain constant.
This makes it not an ideal query parameter as a user querying would not know the receiverData specified by the maker.
This is not the case in ERC20, but it can also appear for potential future Transfer Proxies like ERC777 as it also supports receiverData.
websocket general channel
There needs to be a websocket endpoint on each exchange that publishes every new order (regardless of the pair)
@lukeautry @AusIV would you like to see this as well?
assetData
So what is your plan for being able to query on individual assets, if not by assetData?
At this point I'm reasonably certain that OpenRelay will allow filtering by makerAssetData and takerAssetData even if that's an extension to the standard, but I'm curious what the official way to query for a specific ERC721 asset will be.
websocket general channel
OpenRelay doesn't currently implement the websocket APIs (it's on our roadmap, but not our highest priority), but I do think it makes sense to have an endpoint where you can get orders even if you haven't explicitly asked about the token pair. OpenRelay will accept orders of any arbitrary token pair, so the pattern of querying for token pairs and then subscribing to those pairs only gets you the token pairs we had at the time you ran the first query, but we could have new token pairs at any time.
Yes, the problem becomes apparent when you start wanting to show users liquidity for a pair of tokens on every exchange that implements 0x. You end up making a new websocket request to every exchange just for this pair.
What do you do as an exchange when the user closes their api connection? You can't be expected to make this number of calls to every exchange at every user request, but you also can't be expected to cache this particular pair request as it might be a rarely traded pair.
What you can do is have a fixed number of websocket connections to every exchange implementing 0x and a publisher/subscriber model for every user of your exchange to see the pairs they want and filter out the pairs they're not currently interested in.
This works if every exchange implementing the 0x protocol has an endpoint that streams all orders and order states, regardless of the pair in the order.
I want to implement a 0x order aggregator for free so the community of 0x exchanges doesn't have to worry about duplicating this same code. I work in fintech as a software engineer and have taken this problem to numerous colleagues and we all agree the only way to easily scale this out and aggregate orders is through an unbiased order websocket connection. This is the exact same functionality that traditional FX exchange websockets offer.
The sooner we can adopt this protocol, the sooner I can start aggregating and make this available!
Hi there, thanks for opening the SRA discussion for 0x v2.
Lifecycle events (cancel, partial fill, fully filled). These states can only be verified on-chain...
Does v2 remove the unfunded state for order validity (case of maker not having enough funds)?
assetData
Just closing the loop on the assetData discussion. @dekz mentioned being able to query by assetData was untenable because of the receiverData field. However, we recently decided to remove this field and therefore are moving forward with having the assetData query param.
Seeing some activity on the new version:
Is it intended that orders are submitted with both the assetData and the corresponding address? It seems like the address should be derivable from the assetData, and having both just creates a chance for them to differ (as they do in the current examples).
I seem to remember a decision with the earlier iteration that only signed data (and the signature) would be included with API submissions and responses. Is this changing?
@AusIV are you referring to the GET /orders endpoint?
We have been talking about this internally and I added a little blurb about why we think this is the way to go, but we are still collecting feedback and would love your opinion.
"While there is some redundancy in supporting maker/takerAssetType, maker/takerAssetAddress, and maker/takerAssetData, they are actually all needed. For example, you cannot query "all ERC712 orders", or "all Cryptokitties orders", or "all ERC20 orders" with just maker/takerAssetData and need the other query params to do so."
@arikan After some discussion we think that adding a remainingFillableAmount to every order object response (new schema pending) is sufficient for communicating state changes. In making a request to orders/ for example, each order in the response will have remainingFillableAmount which should take into account what the maker has. If you want an update on this value at a later time, you can hit the order/ endpoint, or run OrderWatcher locally.
@dekz @fabioberger @AusIV @lukeautry @wildnothing @arikan I just pushed a "complete" version of SRA v2 to #26 (except for schemas etc...). Would like to get it finalized by Friday, so please give your final feedback. I hope I addressed everyones concerns / suggestions. Remember, whatever we come up with now doesn't have to be set in stone.
@fragosti - Ignore my last comment. I had a mental lapse where I confused makerAddress with makerAssetAddress, rather than the account making the order.
@fragosti Not sure if the ship has already sailed, but I'd like to suggest a consistent casing convention.
Query parameters: always snake_case
Body parameters, response json: always camelCase
The divergence comes up most notably with something like per_page. The collection responses have a per_page field. In those cases, the query parameter should be per_page, but the response field should be perPage.
@lukeautry great feedback! Ship has not sailed. I think this convention makes sense. We would like to finalize the spec and address any final feedback during the next dev call, which will be August 6th :)
How do we feel about replacing remainingFillableAmount with an orderInfo object adhering to the OrderInfo type that already exists that is returned by getOrderInfoAsync on ExchangeWrapper
Not sure if that makes sense to me. remainingFillableAmount could be calculated in a lot of ways, but OrderInfo requires a trip to the contract and implies that that will need to be done for each order.
Agree with @lukeautry on this one, let's not force Relayers to make an RPC call for this one. They might have an OrderWatcher running, or some other mechanism that updates remainingFillableAmount in a reactive way in their own DB.
@lukeautry @fabioberger that was my feeling as well, just wanted to get more data. I'll be renaming the variable remainingFillableTakerAmount per @AusIV's comment.