alephdata / aleph

Search and browse documents and data; find the people and companies you look for.

Home Page:http://docs.aleph.occrp.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

FEATURE: Streaming endpoint for xref matches

tillprochaska opened this issue · comments

Is your feature request related to a problem? Please describe.
There are cases when it is necessary to process xref matches offline, e.g. in order to use a custom scoring method. Currently, there’s no easy and scalable way to fetch all xref results.

Describe the solution you'd like
There should be a streaming API endpoint similar that allows to stream xref matches similar to the entity streaming endpoint.

Describe alternatives you've considered

  • Aleph has an option to export xref matches. This doesn’t work well for large numbers of matches (100k).

  • The default xref API is limited to a maximum result set of 10k matches, even when using pagination.

Additional context
Based on user feedback