apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Home Page:https://dolphinscheduler.apache.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[DSIP-40][APIService] Add LogClient to fetch log

ruanwenjun opened this issue · comments

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

Right now, the log is stored at worker/master and remote storage.
The query logic look like below

image

The LogService is duplicate at worker/master, we need to maintain two service implementation which have the same code.

And once the master/worker is not active, then we can not fetch the log, this problem may occur frequently in k8s.

So it's better to create a LogClient at API side to manage the log fetch logic, and create a LogServer to response for the client request, the LogServer will be imported at master/worker.

Design Detail

image
  1. We will fetch the log from master/worker firstly.
  2. If the master/worker not exist or we cannot find the log file from master/worker then we will try to fetch from remote log storage.

Compatibility, Deprecation, and Migration Plan

Compatibility with previous version.

Test Plan

Test by manual.

Code of Conduct