Giters

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Home Page:https://dolphinscheduler.apache.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[DSIP-38] Add plugin directory in binary package

ruanwenjun opened this issue · comments

commented

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

In the previous binary package, the plugins are put at server/libs.

- dolphinscheduler-bin
   - api-server
     - bin
     - conf
     - libs 
   - master-server
     - bin
     - conf
     - libs 
   - worker-server
     - bin
     - conf
     - libs
   - alert-server
     - bin
     - conf
     - libs

This will bring below problems:

  • Multiple copies of the plugin exist within different services, cause the binary package is very big.
  • Multiple copies of the plugin exist within different services, cause if we want to replace a plugin we need to replace it at different service.
  • It's difficult to support different version plugins, e.g. hive2, hive3

Design Detail

Firstly, we need to add a plugin directory, the binary package will be

- dolphinscheduler-bin
  - api-server
  - master-server
  - alert-server
  - worker-server
  - plugins
    - task-plugin
    - alert-plugin
    - storage-plugin
    - registry-plugin
    - xx

Then we need to shade the plugins and put them in target plugins.

The server should only rely on plugin-api, cannot rely on plugins, and use child first classloader to load plugins, each plugin will have a classloader.

Finally, we need to optimize the dependency tree.

Compatibility, Deprecation, and Migration Plan

Compatibility with the previous version

Test Plan

Test by manual/e2e/UT

Code of Conduct

SubTasks