DNS to 1.1.1.1 is required for tunnel start🐛

[Gbh17]

We have an Ubuntu machine running cloudflared which we installed per the cloudflare documentation.
All outbound communication is open within the VM, besides 1.1.1.1 which is blocked on all ports.
cloudflared.service wont start stating that

edge discovery: error looking up Cloudflare edge IPs: the DNS query failed error="lookup argotunnel.com on 127.0.0.53

Everything is open per this documentation:
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/deploy-tunnels/tunnel-with-firewall/

To Reproduce

1. sudo iptables -A OUTPUT -d 1.1.1.1 -j DROP
2. systemctl cloudflared restart

Expected behaviour:
cloudflared should run normally in a working condition since 1.1.1.1 is not in the requirements of this page:
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/deploy-tunnels/tunnel-with-firewall/

Environment and versions

• OS: ubuntu
• Architecture: AMD(azure's D2as v5)
• Version: 18.04.4 LTS

Logs:

Cloudflared service logs:

Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF Starting tunnel tunnelID=6dedf33b-09ee-4c48-8be7-640851a3a45a
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF Version 2024.1.5
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF GOOS: linux, GOVersion: go1.21.5, GoArch: amd64
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF Settings: map[no-autoupdate:true token:*****]
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF Generated Connector ID: a3b3202f-93a6-4b74-bf88-f2572d736b58
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF cloudflared will not automatically update if installed by a package manager.
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF Initial protocol quic
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF ICMP proxy will use 172.17.0.4 as source for IPv4
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z INF ICMP proxy will use fe80::222:48ff:fe85:f11 in zone eth0 as source for IPv6
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add
Jan 30 07:37:22 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:22Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 0 is not between ping group 1 to 0 nor ICM
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed error="lookup argotunnel.com on 127.0.0.53
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR Please try the following things to diagnose this issue: event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR 1. ensure that argotunnel.com is returning "origintunneld" service records. event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR Run your system's equivalent of: dig srv _origintunneld._tcp.argotunnel.com event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR 2. ensure that your DNS resolver is not returning compressed SRV records. event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR See GitHub issue golang/go#27546 event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR For example, you could use Cloudflare's 1.1.1.1 as your resolver: event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/ event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z INF Starting metrics server on 127.0.0.1:35949/metrics
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed error="lookup argotunnel.com on 127.0.0.53
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR Please try the following things to diagnose this issue: event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR 1. ensure that argotunnel.com is returning "origintunneld" service records. event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR Run your system's equivalent of: dig srv _origintunneld._tcp.argotunnel.com event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR 2. ensure that your DNS resolver is not returning compressed SRV records. event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR See GitHub issue golang/go#27546 event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR For example, you could use Cloudflare's 1.1.1.1 as your resolver: event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/ event=0
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z INF Tunnel server stopped
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z ERR Initiating shutdown error="Could not lookup srv records on _v2-origintunneld._tcp.argotunnel.com: lookup argotunnel.c
Jan 30 07:37:52 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:52Z INF Metrics server stopped

tcp dump of host 1.1.1.1(all ports all interfaces) during systemctl restart cloudflared

Additional context:
The VM is running on Azure.
While the failure logs state:

Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR Please try the following things to diagnose this issue: event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR 1. ensure that argotunnel.com is returning "origintunneld" service records. event=0
Jan 30 07:37:37 cfdvm-1 cloudflared[5789]: 2024-01-30T07:37:37Z ERR Run your system's equivalent of: dig srv _origintunneld._tcp.argotunnel.com event=0

We are successfully digging said fqdn(_origintunneld._tcp.argotunnel.com)

[Gbh17]

upgrading to ubuntu 22.04 solved this