Grouping/Tagging support for 1:n PoST
tgarm opened this issue · comments
Description
The requirement: Sequence control of PoST service
A typical storage server may contain 8 to 12 HDD, each HDD contains several PoST data directories.
To be optimal, all PoST data can be stored sequentially.
When running PoST with this storage server, the optimal way is to request PoST data sequentially in every single disk, and in parallel for all disks.
Like the following code:
disks := map[string][]string{
"disk1": {"dir1", "dir2", "dir3"},
"disk2": {"dir1", "dir2"},
"disk3": {"dir1", "dir2", "dir3", "dir4"},
}
for disk, directories := range disks {
go func(disk string, directories []string) {
for _, directory := range directories {
checkDirectory(directory)
}
}(disk, directories)
}
Currently, I didn't find any way to control the PoST service request sequence in a go-spacemesh node.
If a grouping feature can be added, it would be greatly useful for farmers to optimize/tune their disk reading performance.
How would it look like
Each service can be tagged with a group name or a tag name, this can be specified in the service CLI arguments.
In the node side, we can have sequential control parameters in config file, to specify if the node should request each group of PoST service in sequence or in parallel.
Anyway, the main feature is to control the PoST service request sequence.
Hey,
We do have some basic demo here: https://github.com/spacemeshos/multiple_post_services_demo that demo is using DAGU to achieve it BUT you could use any other DAG mechanism to define them. There are also other no code tools also like https://github.com/open-rust-initiative/dagrs
We deliberately did not put the orchestration into the go-spacemesh as it really depends on the exact use case.
In the demo that I linked above you can see pretty much the same scenario as you described. Please let me know if it's enough for you.
The key to understanding it is that you DO NOT need to run all post services at any given time. It is completely safe to run it only when needed.
Thank you. I'll try it.