zalando-nakadi / nakadi-producer-spring-boot-starter

Nakadi event producer as a Spring boot starter

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Allow filtering during snapshot event production

BGehrels opened this issue · comments

Triggering snapshot events is a pretty expensive task if your data set is huge. In this case it may be good to be able to specify filtering conditions (like if-modified-since, for example).

We should find a way so using applications can define their own filtering parameters, instead of predefining some in the library.

My first thought was to just pass the request body of the post request to the actuator endpoint through the SnapshotEventProvider to the host application as an opaque string interpreted solely by the later.

The parameter can be added to the new SnapshotEventGenerator interface (from #49), letting the old SnapshotEventProvider one untouched (and thus compatible).

We can basically pass the parameter through all those:

  • SnapshotEventCreationMvcEndpoint.createSnapshot
  • SnapshotEventCreationEndpoint.invoke
  • SnapshotCreationService.createSnapshotEvents
  • SnapshotEventGenerator.generateSnapshots

This brings the question of where to parse the filter parameter, if we suppose the user can give some JSON in the request body. Do we just take it as a string (which means each invocation of generateSnapshots has to do the parsing again), do we parse it into a generic map (or JSON-object) on the controller level and then pass this through? (It might still be not the best thing to use.)
Or do we have one additional method in the generator which takes the filter as a string and returns an object to be passed later to the generate method?

I would just pass it as an opaque string and leave the decision, which format to use completely to the host application. I think the variety of use cases is so large (by id, everything since day x, everything with State x, everything not in state x) that everything else is more of a burden than a service to the host applications developer.