useful but not critical: ability to do "scrolling" GET queries with v3 Monarch API
colleenXu opened this issue · comments
In #774, we migrated to the new v3 Monarch API (using the /association endpoint). This is one of the follow-up tasks that would be useful to do.
The endpoint seems to only allow us to retrieve 500 items at a time (I encountered error 500 (Internal Service Error) when trying to set the limit parameter to > 500 or to -1. -1 worked w/ the old API to return all hits).
Being able to retrieve all the items in the response would be nice, as "scrolling" GET queries. The API responses have a total field that tells us how many total items there are. Ex: this has > 2000 items.
And there is an offset parameter for requests, that we could use to retrieve the next 500 items, and the next, until we've retrieved all the items.
Jackson says there should be code to handle scrolling GET queries already, it just needs to be applied (and customized?) for v3 Monarch API.
They also said that it may be helpful to have an operation-level flag (like supportBatch) that flags operations where we want to do these scrolling GET queries.
On thing I'm unsure of is if all cases can be handled exactly the same way though (ex: same parameter name, same field the response to look for).
@rjawesome Jackson said this may be an issue you could handle. It's not high priority though, if you have other tasks.
I added a special case to the pagination logic to support Monarch.
I also added a feature where pagination can be specified through SmartAPI yamls. For example, for the MonarchAPI the appropriate lines to be added to the yaml for a given operation would be:
parameters:
...
limit: 500
offset: "{{ start }}" # {{ start }} is a special field in templating to denote what numbered entry the pagination is at ( similar to {{ queryInputs }})
...
pagination:
countField: items # number of hits from this response, can point to an array (length is taken) or a numerical field
totalField: total # field in response containing total number of hits from this request
pageSize: 500
...(these don't need to be added for Monarch since I implemented it as a special case similar to biothigns)
@colleenXu I believe the next step for this issue would be for you to review/approve @rjawesome's interface in the SmartAPI yamls. If that interface makes sense, I can start testing and then add to the dev branch/instance.