Giters

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Home Page:https://dolphinscheduler.apache.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Feature][all] We are ready to contribute multiple features

liukunyuan opened this issue · comments

commented

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

We have these feature within our company and hope to contribute them. What feature communities require?

  1. Worker and master grayscale function based on traffic
  2. Optimization of worker and master load protection strategies (optimizing remaining physical memory and CPU algorithms on the machine) to address the issue of worker downtime
  3. Force skip dependency detection during complement
  4. Heartbeat reporting from master and worker to detect disconnection issues
  5. Write the Yarn appid into the task_instance table during task execution to facilitate worker fault tolerance (solving the problem of worker machine downtime and inability to tolerate faults)
  6. Workflow migration project interface
  7. Recommend users to set the cron for the workflow (query the completion time of all upstream workflows in the last 7 days and then recommend the start time of this workflow)
  8. After the task timeout fails, it is necessary to kill the Yarn task
  9. Change workflow instances and task instances to single table queries to solve the problem of slow queries
  10. Provide a batch recovery failure interface to solve the problem of slow recovery of workflow instances caused by cluster or MySQL issues
  11. Add feature control tables to MySQL. All important functions require parameter control to turn on and off

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

commented

What causes the fourth problem?
And I have some suggest:
1.The fifth problem may cause task loss. A data processing task may contain more then one yarn job. If it fails between yarn tasks, this may cause the fault-tolerant task to succeed but miss some yarn task submissions.
2.Can the eighth one be configured? The easy failure of some long tasks may lead to a waste of complement resources.

commented

Please Create DSIP to describe why you want to do this change, it's hard to know what you want to do by a simple description, and some of these should already been supported.

commented

What causes the fourth problem? And I have some suggest: 1.The fifth problem may cause task loss. A data processing task may contain more then one yarn job. If it fails between yarn tasks, this may cause the fault-tolerant task to succeed but miss some yarn task submissions. 2.Can the eighth one be configured? The easy failure of some long tasks may lead to a waste of complement resources.

  1. What causes the fourth problem?
    Mysql network exception for a few minutes
  2. A data processing task may contain more then one yarn job。
    It will write multiple appids to the task_instance table