checking `kubernetes:yaml:ConfigGroup` resources
errordeveloper opened this issue · comments
I am creating a GKE cluser and using kubernetes:yaml:ConfigGroup to deploy some workload to the cluster.
Before I create any resources at all, my policy that looks for certain specific resources doesn't work, albeit it does later on, if cluster was already deployed.
So, in other words, when preview output look like this, it doesn't work:
Type Name Status Info
pulumi:pulumi:Stack ilya-dev (668s) 2 messages
+ ├─ gcp:container:Cluster ilya-dev-1 created (666s)
+ ├─ gcp:container:NodePool ilya-dev-1-np-0 created (111s)
+ ├─ pulumi:providers:kubernetes gke-provider created (0.40s)
└─ kubernetes:yaml:ConfigGroup flux-install (0.00s)
But it works when the kubernetes:yaml:ConfigGroup resource has dependents in the preview output:
pulumi:pulumi:Stack ilya-dev (668s) 2 messages
+ ├─ gcp:container:Cluster ilya-dev-1 created (666s)
+ ├─ gcp:container:NodePool ilya-dev-1-np-0 created (111s)
+ ├─ pulumi:providers:kubernetes gke-provider created (0.40s)
└─ kubernetes:yaml:ConfigGroup flux-install (0.00s)
+ ├─ kubernetes:core/v1:ServiceAccount flux-system/helm-controller created (3s)
+ ├─ kubernetes:core/v1:Namespace flux-system created (0.58s)
+ ├─ kubernetes:core/v1:ServiceAccount flux-system/kustomize-controller created (1s)
+ ├─ kubernetes:core/v1:Service flux-system/notification-controller created (131s)
+ ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy flux-system/allow-scraping created (1s)
+ ├─ kubernetes:core/v1:ServiceAccount flux-system/notification-controller created (1s)
+ ├─ kubernetes:core/v1:ServiceAccount flux-system/source-controller created (2s)
+ ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy flux-system/allow-webhooks created (3s)
+ ├─ kubernetes:apps/v1:Deployment flux-system/notification-controller created (166s)
+ ├─ kubernetes:core/v1:Service flux-system/source-controller created (149s)
+ ├─ kubernetes:apps/v1:Deployment flux-system/helm-controller created (166s)
+ ├─ kubernetes:networking.k8s.io/v1:NetworkPolicy flux-system/allow-egress created (6s)
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole crd-controller-flux-system created (7s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition receivers.notification.toolkit.fluxcd.io created (8s)
+ ├─ kubernetes:apps/v1:Deployment flux-system/kustomize-controller created (165s)
+ ├─ kubernetes:apps/v1:Deployment flux-system/source-controller created (168s)
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding cluster-reconciler-flux-system created (11s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition alerts.notification.toolkit.fluxcd.io created (11s)
+ ├─ kubernetes:core/v1:Service flux-system/webhook-receiver created (136s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition ocirepositories.source.toolkit.fluxcd.io created (12s)
+ ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding crd-controller-flux-system created (12s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition providers.notification.toolkit.fluxcd.io created (12s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition helmrepositories.source.toolkit.fluxcd.io created (12s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition buckets.source.toolkit.fluxcd.io created (13s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition helmcharts.source.toolkit.fluxcd.io created (13s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition gitrepositories.source.toolkit.fluxcd.io created (13s)
+ ├─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition helmreleases.helm.toolkit.fluxcd.io created (13s)
+ └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition kustomizations.kustomize.toolkit.fluxcd.io created (13s)
It will take some effort to make all of the code sharable, but here is what the policy definition looks like at the moment:
import * as gcp from "@pulumi/gcp";
import * as policy from "@pulumi/policy";
import * as kube from "@pulumi/kubernetes";
const stackPolicy: policy.StackValidationPolicy = {
name: "validate-stack-resources",
description: "Validate the stack contains correct resources.",
enforcementLevel: "mandatory",
validateStack: async(args: policy.StackValidationArgs, fail: policy.ReportViolation) => {
let networks = args.resources.filter(r => r.isType(gcp.compute.Network));
if (networks.length != 1) {
fail(`expected one network, found ${networks.length}`)
}
let subnetworks = args.resources.filter(r => r.isType(gcp.compute.Subnetwork));
if (subnetworks.length != 1) {
fail(`expected one subnetwork, found ${networks.length}`)
}
const network = networks[0].asType(gcp.compute.Network);
const subnetwork = subnetworks[0].asType(gcp.compute.Subnetwork);
if (network && subnetwork) {
if (subnetwork.network != network.selfLink) {
fail("subnetwork is linked to an unexpected network")
}
} else {
fail("missing network or subnetwork")
}
let clusters = args.resources.filter(r => r.isType(gcp.container.Cluster));
if (clusters.length !== 1) {
fail(`expected one GKE Cluster but found ${clusters.length}`);
}
const cluster = clusters[0].asType(gcp.container.Cluster);
if (cluster) {
if (cluster.network && network) {
if (!network.selfLink.endsWith(cluster.network)) {
fail("cluster is linked to an unexpected network: "+`${cluster.network} != ${network.selfLink}`)
}
}
let addons = cluster.addonsConfig;
if (!addons.dnsCacheConfig.enabled) {
fail("DNS cache addon should be enabled")
}
if (addons.horizontalPodAutoscaling.disabled) {
fail("HPA addon should be enabled")
}
if (!addons.httpLoadBalancing.disabled) {
fail("HTTP LB should be disabled")
}
if (!addons.configConnectorConfig.enabled) {
fail("Config Connector should be enabled")
}
if (cluster.datapathProvider != "ADVANCED_DATAPATH") {
fail("datapath provider should be 'ADVANCED_DATAPATH'")
}
if (cluster.podSecurityPolicyConfig?.enabled) {
fail("PSP should be disabled")
}
if (cluster.privateClusterConfig.enablePrivateEndpoint) {
fail("private endpoints should be disabled")
}
if (!cluster.privateClusterConfig.enablePrivateNodes) {
fail("private ndos shoudl be enabled")
}
if (!cluster.removeDefaultNodePool) {
fail("default node pool should be disabled")
}
}
let fluxResourcesStats = new Map<string, number>(Object.entries({
"kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition": 10,
"kubernetes:apps/v1:Deployment": 4,
"kubernetes:core/v1:Namespace": 1,
"kubernetes:core/v1:Service": 3,
"kubernetes:core/v1:ServiceAccount": 4,
"kubernetes:networking.k8s.io/v1:NetworkPolicy": 3,
"kubernetes:rbac.authorization.k8s.io/v1:ClusterRole": 1,
"kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding": 2,
}));
let fluxConfigGroup = args.resources.filter(r => r.isType(kube.yaml.ConfigGroup))
if (!fluxConfigGroup) {
let fluxResources = args.resources.filter(r => (r.parent?.name == "flux-install"))
let fluxResourceCount = 0;
for (let [k, v] of fluxResourcesStats) {
let resources = fluxResources.filter(r => (r.type == k))
fluxResourceCount += resources.length
if (resources.length != v) {
fail(`unexpected number of flux resources of type ${k}: ${resources.length} != ${v}`)
}
}
if (fluxResourceCount != 28) {
fail(`unexpected total number of flux resources: ${fluxResourceCount}`)
}
} else {
if (fluxConfigGroup.length != 1) {
fail(`config group length: ${fluxConfigGroup.length}`)
}
}
},
};
const tests = new policy.PolicyPack("infra-policy", {
policies: [stackPolicy],
});
It would make some sense if I could at least get the group resource itself, which I'm attempting to this way:
let fluxConfigGroup = args.resources.filter(r => r.isType(kube.yaml.ConfigGroup))
But it doesn't seem to be present at all.
Hello @errordeveloper thanks for opening this. Could you share the output of pulumi about in the comments?
Odd...I would expect r => r.isType(kube.yaml.ConfigGroup) to have at least one member. I wish I had more to offer at this time.
For posterity:
Thanks for getting back to me @RobbieMcKinstry!
I've made a few changes and use load manifests dirctly from files instead of using a package.
$ pulumi about --stack dev2
CLI
Version 3.43.1
Go Version go1.19.2
Go Compiler gc
Plugins
NAME VERSION
gcp 6.40.0
gcp 6.40.0
kubernetes 3.21.4
kubernetes 3.21.4
nodejs unknown
tls 4.6.1
Host
OS darwin
Version 12.6
Arch arm64
This project is written in nodejs: executable='/Users/ilya/Library/Local/Homebrew/bin/node' version='v18.10.0'
Current Stack: dev2
Found no resources associated with dev2
Found no pending operations associated with dev2
Backend
Name pulumi.com
URL https://app.pulumi.com/errordeveloper
User errordeveloper
Organizations errordeveloper, atomisthq
Dependencies:
NAME VERSION
jest 29.1.2
ts-node 10.9.1
@pulumi/gcp 6.40.0
@pulumi/kubernetes 3.21.4
@pulumi/policy 1.5.0
@pulumi/tls 4.6.1
@types/jest 29.1.2
@babel/preset-env 7.19.4
@babel/preset-typescript 7.18.6
@pulumi/pulumi 3.42.0
yaml 2.1.3
Pulumi locates its logs in /var/folders/gm/dnc405js6nlb275l4v2gh_b40000gn/T/ by default
$ pulumi up --stack dev2 --policy-pack policy
Previewing update (dev2)
View Live: https://app.pulumi.com/errordeveloper/image-build-service/dev2/previews/c15eaa03-60ff-4541-9db9-0195f2c5e148
Type Name Plan Info
+ pulumi:pulumi:Stack image-build-service-dev2 create 1 error
+ ├─ kubernetes:yaml:ConfigGroup flux-install create
+ │ ├─ kubernetes:yaml:ConfigFile ../configs/shared/flux/flux.yaml create 1 message
+ │ └─ kubernetes:yaml:ConfigFile ../configs/shared/flux/sources.yaml create 1 message
+ ├─ gcp:projects:Service logging-service create
+ ├─ gcp:projects:Service iam-service create
+ ├─ gcp:projects:Service monitoring-service create
+ ├─ gcp:projects:Service cloudkms-service create
+ ├─ tls:index:PrivateKey flux-image-build-service create
+ ├─ gcp:projects:Service storage-api-service create
+ ├─ gcp:projects:Service compute-service create
+ ├─ gcp:projects:Service container-service create
+ ├─ gcp:projects:Service iamcredentials-service create
+ ├─ gcp:projects:Service storage-component-service create
+ ├─ gcp:projects:Service cloudresourcemanager-service create
+ ├─ gcp:compute:Network ilya-ibs-dev-2-network create
+ ├─ gcp:serviceAccount:Account ilya-ibs-dev-2-node-sa create
+ ├─ gcp:compute:Router ilya-ibs-dev-2-router create
+ ├─ gcp:compute:Subnetwork ilya-ibs-dev-2-subnetwork create
+ ├─ gcp:projects:IAMMember monitoring-metricwriter-ilya-ibs-dev-2-node-sa-member create
+ ├─ gcp:projects:IAMMember monitoring-viewer-ilya-ibs-dev-2-node-sa-member create
+ ├─ gcp:projects:IAMMember logging-logwriter-ilya-ibs-dev-2-node-sa-member create
+ ├─ gcp:compute:RouterNat ilya-ibs-dev-2-router-nat create
+ ├─ gcp:container:Cluster ilya-ibs-dev-2 create
+ ├─ pulumi:providers:kubernetes gke-provider create
+ └─ gcp:container:NodePool ilya-ibs-dev-2-np-0 create
Diagnostics:
kubernetes:yaml:ConfigFile (../configs/shared/flux/sources.yaml):
Can't decode yaml config when provider is not fully initialized. This can result in empty previews but should resolve correctly during apply.
pulumi:pulumi:Stack (image-build-service-dev2):
error: preview failed
kubernetes:yaml:ConfigFile (../configs/shared/flux/flux.yaml):
Can't decode yaml config when provider is not fully initialized. This can result in empty previews but should resolve correctly during apply.
Policy Violations:
[mandatory] build-service-infra-policy v0.0.1 validate-stack-resources (pulumi:pulumi:Stack: image-build-service-dev2)
Validate the stack contains correct resources.
config group length: 0
[mandatory] build-service-infra-policy v0.0.1 validate-stack-resources (pulumi:pulumi:Stack: image-build-service-dev2)
Validate the stack contains correct resources.
config files length: 0
Outputs:
cluster : "ilya-ibs-dev-2-4bc8a11"
fluxPublicKey : output<string>
nodeServiceAccount: output<string>
project : "sandbox-298914"
region : "europe-west2"
$
So now there also Can't decode yaml config when provider is not fully initialized warnings, which is probably to do with CRDs and CRs.
I understand that some things get a bit complex to do with CRDs and CRs... but I still think the object should be visible to the policy.
Here is relevant policy logic:
let fluxConfigGroup = args.resources.filter(r => r.isType(kube.yaml.ConfigGroup))
if (fluxConfigGroup.length == 0) {
fail(`config group length: ${fluxConfigGroup.length}`)
}
let fluxConfigFiles = args.resources.filter(r => r.isType(kube.yaml.ConfigFile))
if (fluxConfigFiles.length == 0) {
fail(`config files length: ${fluxConfigFiles.length}`)
}
I also added one kube.core.v1.Secret, but this is dependenat on a private key, so I have this in my code:
const fluxPrivateKey = new PrivateKey("flux-image-build-service", { algorithm: "RSA" });
let fluxPrivateKeySecret: kube.core.v1.Secret;
fluxPrivateKey.privateKeyOpenssh.apply(key => {
fluxPrivateKeySecret = new kube.core.v1.Secret("flux-image-build-service", {
metadata: {
// name must be set here to avoid Pulumi auto-naming, as otherwise
// secret name would have to be passed to a manifest that is currently
// static (albeit transform could be used for this, it seems unnecessary)
name: "flux-image-build-service",
namespace: "flux-system",
},
stringData: {
identity: key.trimEnd(),
},
}, kubeOpts);
I'm trying to write policy logic for this also, and it seems like I'm hitting another bump, probably to do with that the resource is defoined in the promise callback... What happens is that args.resources.filter(r => r.isType(kube.core.v1.Secret)) returns array of one element, but it's undefined.
I guess this leave me wondering wheather I'm really hitting the limitations of how policy is implemented at the moment?