startreedata / pinot-recipes

This repository contains recipes for Apache Pinot.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

real-time to offline job should also take care of updated records

adudhe-widefield opened this issue · comments

The recipe for real-time to offline job details the steps to be followed to fix the updated records in the offline segment and then upload that segment. However, these are manual steps. Will be good to have this automated. For example, the existing job or a new job takes care of fixing the updated records (i.e. multiple records with same primary key in the offline segment).

Thank you! We'll keep this in mind when we get to this recipe during our ongoing audit/update of the contents of this repo. Your idea is appreciated.

We have an internal task to look into this more. Thanks