Loadbalancer Public IP with Zones enabled
dkanbier opened this issue · comments
Ask your question here
I have configured an AKS cluster using Availability Zones following this document.
The Standard
LoadBalancer called kubernetes
this creates by default has a frontend-ip configuration using a public-ip which has Availability Zones 1, 2, 3 enabled.
When running the azqr
tool it reports the LoadBalancer to not have Availability Zones enabled. I was wondering if this is expected behavior or if I've made an error in the configuration.
Possible cause for azqr
reporting the issue:
When showing the public-ip using az network public-ip show
I can see the Zones
field defined correctly. However when I query the same public-ip using az network lb frontend-ip show
I do not see a Zones
field.
When trying use az network lb frontend-ip show
for an internal-ip I do get a Zones
field. The azqr
tool reports no issues on a LB using this internal-ip.
The test just seems to look at the FrontendIPConfigurations
and the zone info listed there. When using a public ip you need look the zone configuration of the public ip address.
https://github.com/Azure/azqr/blob/main/internal/scanners/lb/rules.go#L36
Looks like a bug to me.
Hi @dkanbier can you please share de JSON for the Load Balancer and the Public IP so I can take a look? (Remember to remove or mask your subscription ID or any other value you want to keep private)
A frontendIPConfigurations
with a public ip:
"frontendIPConfigurations": [
{
"etag": "W/\"7a579181-9309-4228-9c2a-2b9c7d4e276a\"",
"id": "/subscriptions/***/resourceGroups/MC_aks/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/582505be-f971-490b-8b85-7ebad9bfdba0",
"name": "582505be-f971-490b-8b85-7ebad9bfdba0",
"outboundRules": [
{
"id": "/subscriptions/***/resourceGroups/MC_aks/providers/Microsoft.Network/loadBalancers/kubernetes/outboundRules/aksOutboundRule",
"resourceGroup": "MC_aks"
}
],
"privateIPAllocationMethod": "Dynamic",
"provisioningState": "Succeeded",
"publicIPAddress": {
"id": "/subscriptions/***/resourceGroups/MC_aks/providers/Microsoft.Network/publicIPAddresses/582505be-f971-490b-8b85-7ebad9bfdba0",
"resourceGroup": "MC_aks"
},
"resourceGroup": "MC_aks",
"type": "Microsoft.Network/loadBalancers/frontendIPConfigurations"
}
],
versus one with only private ip:
"frontendIPConfigurations": [
{
"etag": "W/\"5f1c68bc-5633-4fd7-85da-1b4948de4b81\"",
"id": "/subscriptions/***/resourceGroups/mc_aks/providers/Microsoft.Network/loadBalancers/kubernetes-internal/frontendIPConfigurations/af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet",
"loadBalancingRules": [
{
"id": "/subscriptions/***/resourceGroups/mc_aks/providers/Microsoft.Network/loadBalancers/kubernetes-internal/loadBalancingRules/af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet-TCP-80",
"resourceGroup": "mc_aks"
},
{
"id": "/subscriptions/***/resourceGroups/mc_aks/providers/Microsoft.Network/loadBalancers/kubernetes-internal/loadBalancingRules/af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet-TCP-8443",
"resourceGroup": "mc_aks"
}
],
"name": "af4b5ab1abce344deb03ae8d21e4a9ad-Aks02IngressSubnet",
"privateIPAddress": "10.0.10.149",
"privateIPAddressVersion": "IPv4",
"privateIPAllocationMethod": "Static",
"provisioningState": "Succeeded",
"resourceGroup": "mc_aks",
"subnet": {
"id": "/subscriptions/***/resourceGroups/spoke/providers/Microsoft.Network/virtualNetworks/vnet-cncr-nonprod-frc/subnets/AksIngressSubnet",
"resourceGroup": "spoke"
},
"type": "Microsoft.Network/loadBalancers/frontendIPConfigurations",
"zones": [
"1",
"2",
"3"
]
}
],
``
Hi @cmendible , thanks for the reply. The root cause in my opinion is what @nlighten is mentioning. The frontendIPConfigurations
object of a Public IP does not contain a zones
field as opposed to a Private IP which does return a zones
field.
When you directly get the Public IP object, you do get a zones
field. This happens both when using the azure-sdk-for-go
or azure-cli
like I tried to explain in my initial post.
azqr
uses the frontendIPConfiguration
object to determine if a LB is zone redundant or not, hence I think it will always report it's not when the LB is using a Public IP.
LoadBalancer JSON:
{
"name": "kubernetes",
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes",
"etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
"type": "Microsoft.Network/loadBalancers",
"location": "westeurope",
"tags": {
"aks-managed-cluster-name": "deleteAKS",
"aks-managed-cluster-rg": "deleteMe"
},
"properties": {
"provisioningState": "Succeeded",
"resourceGuid": "16aeb038-c9c6-4535-973c-d37fd6e8eb09",
"frontendIPConfigurations": [
{
"name": "428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
"type": "Microsoft.Network/loadBalancers/frontendIPConfigurations",
"properties": {
"provisioningState": "Succeeded",
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/publicIPAddresses/428f3d97-c114-4fcf-b58a-cb7b127a7e92"
}
}
}
],
"backendAddressPools": [
{
"name": "aksOutboundBackendPool",
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/aksOutboundBackendPool",
"etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
"properties": {
"provisioningState": "Succeeded",
"backendIPConfigurations": [
{
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/2/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
},
{
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/3/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
},
{
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/4/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
}
]
},
"type": "Microsoft.Network/loadBalancers/backendAddressPools"
},
{
"name": "kubernetes",
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/backendAddressPools/kubernetes",
"etag": "W/\"1fcbefe6-13be-43aa-a09d-f26c7736c332\"",
"properties": {
"provisioningState": "Succeeded",
"backendIPConfigurations": [
{
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/2/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
},
{
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/3/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
},
{
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-nodepool1-38051177-vmss/virtualMachines/4/networkInterfaces/aks-nodepool1-38051177-vmss/ipConfigurations/ipconfig1"
}
]
},
"type": "Microsoft.Network/loadBalancers/backendAddressPools"
}
],
"loadBalancingRules": [],
"probes": [],
"inboundNatRules": [],
"inboundNatPools": []
},
"sku": {
"name": "Standard"
}
}
Public IP JSON:
{
"name": "428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/publicIPAddresses/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"etag": "W/\"62b30655-2450-4daa-91b1-ccd8c8ec1fff\"",
"location": "westeurope",
"tags": {
"aks-managed-cluster-name": "deleteAKS",
"aks-managed-cluster-rg": "deleteMe",
"aks-managed-type": "aks-slb-managed-outbound-ip"
},
"properties": {
"provisioningState": "Succeeded",
"resourceGuid": "4ca22ea6-3e5c-4e7a-9967-fe7bdb138c7c",
"ipAddress": "***",
"publicIPAddressVersion": "IPv4",
"publicIPAllocationMethod": "Static",
"idleTimeoutInMinutes": 4,
"ipTags": [],
"ipConfiguration": {
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/428f3d97-c114-4fcf-b58a-cb7b127a7e92"
}
},
"type": "Microsoft.Network/publicIPAddresses",
"sku": {
"name": "Standard"
}
}
Please note that in both JSONs there is no mention of zones
. However if you query the Public IP directly it will show a zones
field:
az network public-ip show -n 428f3d97-c114-4fcf-b58a-cb7b127a7e92 --resource-group MC_deleteMe_deleteAKS_westeurope
{
"etag": "W/\"62b30655-2450-4daa-91b1-ccd8c8ec1fff\"",
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/publicIPAddresses/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"idleTimeoutInMinutes": 4,
"ipAddress": "***",
"ipConfiguration": {
"id": "/subscriptions/***/resourceGroups/MC_deleteMe_deleteAKS_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes/frontendIPConfigurations/428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"resourceGroup": "MC_deleteMe_deleteAKS_westeurope"
},
"ipTags": [],
"location": "westeurope",
"name": "428f3d97-c114-4fcf-b58a-cb7b127a7e92",
"provisioningState": "Succeeded",
"publicIPAddressVersion": "IPv4",
"publicIPAllocationMethod": "Static",
"resourceGroup": "MC_deleteMe_deleteAKS_westeurope",
"resourceGuid": "4ca22ea6-3e5c-4e7a-9967-fe7bdb138c7c",
"sku": {
"name": "Standard",
"tier": "Regional"
},
"tags": {
"aks-managed-cluster-name": "deleteAKS",
"aks-managed-cluster-rg": "deleteMe",
"aks-managed-type": "aks-slb-managed-outbound-ip"
},
"type": "Microsoft.Network/publicIPAddresses",
"zones": [
"3",
"2",
"1"
]
}
Thank you both! Seems like current validation works only for private IP's and will fail for Public IP's.
I'll have to query Public IP's with zones and add the results to the scan context in order to fix this rule.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
Hey @dkanbier @nlighten can you try the binary from here: https://github.com/Azure/azqr/actions/runs/6768369641 and check if the issue is fixed?