WORK IN PROGRESS I have so little time so i will try to do a small thing everyday for a while T_T
Changed my plan
- do distributed raft nodes
- store should be sqlite
- should provide tools for dag manipulation
- storage shoul be as rows
- goal: not meant for big graph but smaller ones (it's not a database)
stop thinking too much
Hierarchical Workflow Management System with DAG-Structured Tasks in Go
Develop a robust workflow management system in Go, aimed at structuring and executing units of work in an organized, dependency-resolved manner. The system hierarchy encompasses Topics, Jobs, and Tasks, each level offering a layer of organization and control.
Core Components:
- Topics: High-level categorizations of related work, e.g., "Provisioning" for provisioning bare metal machines.
- Jobs: Individual scopes of work within a Topic, e.g., a job with ID
XXXX
under the "Provisioning" topic. - Tasks: Sets of related, finer-grained units of work associated with a parent Job, organized in a Directed Acyclic Graph (DAG) to dictate the execution order.
- Task Units: The granular work units within a Task's DAG, to be executed in a defined sequence. Each Task Unit represents a specific action required to progress the parent Task towards completion.
- Commands: A set of actions exposed via an API, enabling clients to report Task Unit progress (e.g., progress, success, error, pause, log) and mutate Task Unit states.
- API: The interface for clients to interact with the system, fetch Task Units, and send Commands to report progress.
Workflow:
- Define Topics and create Jobs under them.
- For each Job, define a set of Tasks and organize them in a DAG to specify the execution order.
- Clients fetch available Task Units from the API, based on their assigned Tasks.
- Clients execute Task Units in accordance with the DAG's order and report progress using Commands via the API.
- The system processes Commands, updates Task Unit and Task states, and advances the Job towards completion.
- Upon completion of all Task Units within a Task's DAG, mark the Task as completed.
- Once all Tasks within a Job are completed, mark the parent Job as completed.
That's it
No processing, job of the client We just wait for all units to be completed, to valiate a task, which all tasks of a Job need to be validated to complete it.
Storage Implementations:
- in memory
- sqlite3
The developer should be allowed to provide a storage implementation of it's own, junjo
only provide the mechanism and the rules.