Refactor Stop Events to a independent component
laaksma opened this issue · comments
Currently Trip update processor ingests messages of type Arrival and Departure, which are Pubtrans data types. We'd want to move the processing logic for these types to a separate service which handles the raw data and produces smaller Stop Time Estimate (name tbd) type messages.
The content would be roughly:
- date
- route
- direction
- trip start time
- stop id
- type (arrival or departure)
- estimated time
- timetabled time
- generation time stamp
This would allow us to use different data sources in the future, and also simplify the analytics related to the estimates.
additional fields needed for TripUpdates
- stopSeq
- scheduledRelationship (scheduled / cancelled)
we also need to define a unique trip key (ID)
What to use as Pulsar key?
all else are required fields except "timetabled-time" which is optional
format in this phase is Jore (direction, etc).
perhaps namespace the fields to be jore-direction, jore-route, etc. ? add comments to protobuf files
reviewing: https://www.digitraffic.fi/rautatieliikenne/#junat
some things need to be mapped but overall should work? if liveEstimateTime is missing we just filter those ones out,?
let's add trip-id to the payload. we'll use this as unique key for this trip. String as data type so it's easily extendable.
this is now fully merged to develop