FEATURE: Streaming endpoint for xref matches
tillprochaska opened this issue · comments
Is your feature request related to a problem? Please describe.
There are cases when it is necessary to process xref matches offline, e.g. in order to use a custom scoring method. Currently, there’s no easy and scalable way to fetch all xref results.
Describe the solution you'd like
There should be a streaming API endpoint similar that allows to stream xref matches similar to the entity streaming endpoint.
Describe alternatives you've considered
-
Aleph has an option to export xref matches. This doesn’t work well for large numbers of matches (100k).
-
The default xref API is limited to a maximum result set of 10k matches, even when using pagination.
Additional context
Based on user feedback