[FEATURE] Provide Policy for ETCD backup and monitor ETCD backup

[Christian Stark]

Is your feature request related to a problem? Please describe.

We demonstrated RHACM to a Customer showing ETCD encryption and ETCD backup.
Customer would like to monitor both. This Enhancement is about how to monitor ETCD backup is working properly.

This policy currently does the backup:
https://github.com/ch-stark/policies-demo/blob/master/policies/backup-etcd-policy-v2.yaml

 Generated yaml-objects              

                kind: ServiceAccount
                kind: ClusterRoleBinding
                kind: PersistentVolumeClaim
                metadata:
                  name: etcd-backup
                kind: ConfigMap
                metadata:
                  name: etcd-backup-script
                kind: CronJob

it works so far

This is an example how to monitor that:
https://github.com/ch-stark/policies-demo/blob/master/policies/policy-checkjob.yaml

it checks if the job that does the backup has succeeded but this is not ideal

Describe the solution you'd like
Support for such a policy

Describe alternatives you've considered

We check manually:

oc logs cronjob-etcd-backup-1603974060-jj7bt
found latest kube-apiserver-pod: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-9
found latest kube-controller-manager-pod: /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-8
found latest kube-scheduler-pod: /etc/kubernetes/static-pod-resources/kube-scheduler-pod-8
found latest etcd-pod: /etc/kubernetes/static-pod-resources/etcd-pod-3
Snapshot saved at /assets/backup/snapshot_2020-10-29_122123.db
snapshot db and kube resources are successfully saved to /assets/backup
Copied backup files to PVC mount point.

We can monitor the timestamp of the backup

Additional context

We discussed that a CustomPolicyController might be needed
Thanks for any help!!!