Feat | Using KubeVela to manage multiple GameServerSet
Somefive opened this issue · comments
Background
In game area, servers are usually grouped into various partitions. Each partition of servers are set up for serving a specific range of players. In each partition, there could be multiple types of servers that provides different services and communicate with each other server in the same partition, like Battle servers or Scene servers.
Each type of server could contain multiple replicas and be modeled as a GameServerSet. The GameServerSet will create Open Kruise StatefulSet to pull up pods and each pod will be attached with an additional GameServer object to help set the operate state of the pod.
Operating/Managing a large amount of GameServerSets across different partitions can be laborious. To help alleviate the burden of repetitive operations for GameServerSets, we could introduce KubeVela to model the higher level application on top of the GameServerSets.
Architecture
In KubeVela, applications are used to model resources and manage their spec & lifecycles. Besides, there are also delivery pipeline that could describe the operation actions as codes and be reused.
Specifically, we could model the GameServerSets in each partition as a single KubeVela application and let it manage the desired state and delivery process of the GameServerSets, such as updates. On top of that, for the whole game, we could add another application to manage the partition applications.
With architecture, it will be easy to modify the desired state of the GameServerSets along with the partitions or the types (roles).
Implementation
First, to model the partition application, we need a KubeVela ComponentDefinition for the abstraction of GameServerSet. The below CUE templates defines how the GameServerSet is formed, which parameters are exposed and how the health state is evaluated.
"game-server-set": {
alias: ""
annotations: {}
description: "The GameServerSet."
type: "component"
attributes: {
workload: type: "autodetects.core.oam.dev"
status: {
customStatus: #"""
status: {
replicas: *0 | int
} & {
if context.output.status != _|_ {
if context.output.status.readyReplicas != _|_ {
replicas: context.output.status.readyReplicas
}
}
}
message: "\(context.name): \(status.replicas)/\(context.output.spec.replicas)"
"""#
healthPolicy: #"""
status: {
replicas: *0 | int
generation: *-1 | int
} & {
if context.output.status != _|_ {
if context.output.status.readyReplicas != _|_ {
replicas: context.output.status.readyReplicas
}
if context.output.status.observedGeneration != _|_ {
generation: context.output.status.observedGeneration
}
}
}
isHealth: (context.output.spec.replicas == status.replicas) && (context.output.metadata.generation == status.generation)
"""#
}
}
}
template: {
parameter: {
// +usage=The image of the Game Server
image: string
// +usage=The number of replicas
replicas: *1 | int
}
output: {
apiVersion: "game.kruise.io/v1alpha1"
kind: "GameServerSet"
spec: {
updateStrategy: rollingUpdate: podUpdatePolicy: "InPlaceIfPossible"
gameServerTemplate: spec: containers: [{
image: parameter.image
name: "\(context.name)"
}]
}
metadata: name: "\(context.name)"
spec: replicas: parameter.replicas
}
}
On top of that, we could have the abstraction of partition applications, which is indicated as game-server-sets as below. The game-server-sets is a template for generating partition application, and exposes the configuration of different types of underlying GameServerSets in the parameter.
"game-server-sets": {
alias: ""
annotations: {}
description: "The Game Server Sets of one region."
type: "component"
attributes: {
workload: type: "autodetects.core.oam.dev"
status: {
customStatus: #"""
status: {
phase: *"initializing" | string
} & {
if context.output.status != _|_ {
if context.output.status.status != _|_ {
phase: context.output.status.status
}
}
}
message: "\(context.name): \(status.phase)"
"""#
healthPolicy: #"""
status: {
phase: *"initializing" | string
generation: *-1 | int
} & {
if context.output.status != _|_ {
if context.output.status.status != _|_ {
phase: context.output.status.status
}
if context.output.status.observedGeneration != _|_ {
generation: context.output.status.observedGeneration
}
}
}
isHealth: (status.phase == "running") && (status.generation >= context.output.metadata.generation)
"""#
}
}
}
template: {
#GameServerSet: {
// +usage=The image of the Game Server
image: string
// +usage=The number of replicas
replicas: *1 | int
// +usage=The dependencies of the Game Server
dependsOn: *[] | [...string]
}
parameter: [string]: #GameServerSet
output: {
apiVersion: "core.oam.dev/v1beta1"
kind: "Application"
metadata: name: context.name
spec: {
components: [for role, gss in parameter {
name: "\(context.name)-\(role)"
type: "game-server-set"
properties: {
image: gss.image
replicas: gss.replicas
}
_dependsOn: [for d in gss.dependsOn {"\(context.name)-\(d)"}]
if len(_dependsOn) > 0 {
dependsOn: _dependsOn
}
}]
workflow: steps: [{
type: "deploy"
name: "deploy"
properties: policies: []
}]
}
}
}
Finally, we have got the application that manages all the partition application as the user interface, shown as below
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
name: mmo
namespace: game
spec:
components:
- type: game-server-sets
name: partition-1
properties:
battle:
image: nginx:1.17
replicas: 2
scenes:
image: nginx:1.20
replicas: 2
ai:
image: nginx:1.21
replicas: 1
- type: game-server-sets
name: partition-2
dependsOn: ["partition-1"]
properties:
battle:
image: nginx:1.17
replicas: 3
scenes:
image: nginx:1.20
replicas: 3
ai:
image: nginx:1.21
replicas: 2
- type: game-server-sets
name: partition-3
properties:
battle:
image: nginx:1.17
replicas: 1
scenes:
image: nginx:1.20
replicas: 1
ai:
image: nginx:1.21
replicas: 1
policies:
- type: override
name: global-config
properties:
components:
- properties:
scenes:
dependsOn: ["battle"]
- type: apply-once
name: apply-once
properties:
enable: true
workflow:
steps:
- type: deploy
name: deploy
properties:
policies: ["global-config"]The dependsOn lines specify the delivery order between different partitions and different types of GameServerSets. In the above example, we requires the update of partition-2 is after partition-1, and the updates of Scenes GameServerSets are after the updates of Battle GameServerSets.
- To start or stop a partition, we can just add more
game-server-setscomponent to the top layer MMO application. - To update the image of a specific type of GameServerSets, we have two ways to do so. One is to directly update the image field in each
game-server-setscomponent's configuration. This allows users to have fine-grained control for the image of each GameServerSet in each partition. The other way is to set the image field in theglobal-configpolicy, where users only need to config once and it will take effects across all partitions. - To scale the replicas of GameServerSets, the action we need to do is similar to above. We just set the replicas field in the component properties.
Orchestrate
Another action is to set the state of GameServer for the pod, to mark the pod's deletion state, priority and other configurations. This can be achieved with the use of KubeVela WorkflowRun. The reason for using WorkflowRun instead of Application for managing GameServers is because GameServer objects are not directly managed through Applications or GameServerSets. They are post-attached to pods. So using a sideway to manage them can be easier and more simple.
To define the operation behaviour, we use the follow CUE templates. The operate-gs defines the detailed update operation for updating the GameServer object. It first read the GameServer from Kubernetes and re-assemble it with the update fields.
import (
"vela/op"
)
"operate-gs": {
type: "workflow-step"
description: "Operate GameServer."
}
template: {
#Operation: {
deletionPriority?: int
opsState?: "None" | "WaitToBeDeleted"
updatePriority?: int
}
handle: op.#Steps & {
for gsName, o in parameter {
"\(gsName)": op.#Steps & {
read: op.#Read & {
value: {
apiVersion: "game.kruise.io/v1alpha1"
kind: "GameServer"
metadata: {
name: gsName
namespace: context.namespace
}
}
} @step(1)
apply: op.#Apply & {
value: {
for k, v in read.value if k != "spec" {
"\(k)": v
}
if read.value.spec != _|_ {
spec: {
for k, v in read.value.spec {
if k != "deletionPriority" && k != "opsState" && k != "updatePriority" {
"\(k)": v
}
}
if o.deletionPriority != _|_ {
deletionPriority: o.deletionPriority
}
if o.deletionPriority == _|_ && read.value.spec.deletionPriority != _|_ {
deletionPriority: read.value.spec.deletionPriority
}
if o.opsState != _|_ {
opsState: o.opsState
}
if o.opsState == _|_ && read.value.spec.opsState != _|_ {
opsState: read.value.spec.opsState
}
if o.updatePriority != _|_ {
updatePriority: o.updatePriority
}
if o.updatePriority == _|_ && read.value.spec.updatePriority != _|_ {
updatePriority: read.value.spec.updatePriority
}
}
}
}
} @step(2)
}
}
}
parameter: [string]: #Operation
}
The use of the atomic action is as below
apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
name: edit-gs
namespace: game
spec:
workflowSpec:
steps:
- type: operate-gs
name: operate-gs
properties:
partition-1-scenes-0:
opsState: WaitToBeDeleted
partition-2-battle-1:
opsState: WaitToBeDeleted
partition-2-scenes-0:
deletionPriority: 20
This WorkflowRun is a one-time execution. It sets the opsState to WaitToBeDeleted for the first replica of the Scene GameServerSet in partition 1. Similar behaviors are applied to partition 2.
Extra Resource Relationship
In KubeVela, there is resource topology in KubeVela to be used to display the internal architecture of the application. To help visualize the relationships between GameServerSets and StatefulSets, Pods and GameServers, we could apply the following configuration into the KubeVela system. Then we will be able to visualize the full architecture of the KubeVela application.
apiVersion: v1
kind: ConfigMap
metadata:
name: game-server-set-relation
namespace: vela-system
labels:
"rules.oam.dev/resource-format": "yaml"
"rules.oam.dev/resources": "true"
data:
rules: |-
- parentResourceType:
group: game.kruise.io
kind: GameServerSet
childrenResourceType:
- apiVersion: apps.kruise.io/v1beta1
kind: StatefulSet
- apiVersion: game.kruise.io/v1alpha1
kind: GameServer
- parentResourceType:
group: apps.kruise.io
kind: StatefulSet
childrenResourceType:
- apiVersion: v1
kind: Pod