First node not accessed via SSH loadbalancer
poikilotherm opened this issue · comments
If I understood the code correctly, the first server node (which is my only node so far to test things with this module, no agents defined) is supposed to be reached via the SSH load balancer.
From what I can see in the OpenStack project, the node is not added as a pool member. The "wait_for_rke" task kicks in and tries to access the host directly (confirmed via tcpdump), as bastion_host is probably empty. Here's my log from terraform apply:
module.controlplane.random_string.rke2_token: Creating...
module.controlplane.random_string.rke2_token: Creation complete after 0s [id=...]
module.controlplane.openstack_compute_keypair_v2.key: Creating...
module.controlplane.openstack_networking_secgroup_v2.server: Creating...
module.controlplane.openstack_networking_secgroup_v2.agent: Creating...
module.controlplane.openstack_identity_application_credential_v3.rke2: Creating...
module.controlplane.openstack_networking_router_v2.router: Creating...
module.controlplane.openstack_networking_network_v2.net: Creating...
module.controlplane.openstack_compute_keypair_v2.key: Creation complete after 2s [id=rdmstack-key]
module.controlplane.openstack_networking_secgroup_v2.agent: Creation complete after 3s [id=8a9c5265-90a8-4be3-bb1d-a4efa461a536]
module.controlplane.openstack_networking_secgroup_rule_v2.agent6: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.agent4: Creating...
module.controlplane.openstack_networking_secgroup_v2.server: Creation complete after 3s [id=a6c0246c-37a0-4341-a19f-0535d151013e]
module.controlplane.openstack_networking_secgroup_rule_v2.server6: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.server4: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.lb_server["192.168.44.0/24-tcp-6443"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.lb_server["192.168.44.0/24-tcp-22"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.lb_server["192.168.44.0/24-tcp-9345"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.agent4: Creation complete after 0s [id=06d43730-22c2-46ca-88b1-6a291b8166a2]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-tcp-10250"]: Creating...
module.controlplane.openstack_identity_application_credential_v3.rke2: Creation complete after 3s [id=e465c614171d4cd0b7db4d88663e9b37]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-icmp-0"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.agent6: Creation complete after 0s [id=c268765c-92f1-4bbd-ae67-f0448900a73e]
module.controlplane.openstack_networking_secgroup_rule_v2.server4: Creation complete after 0s [id=86bb9f36-5716-4fb9-9216-34bfb84b08bf]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-tcp-10250"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-icmp-0"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.server6: Creation complete after 1s [id=c70b2a6a-2ada-42f9-a2d9-50c7966bf8ab]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-tcp-10250"]: Creation complete after 1s [id=db635606-3946-4769-86a9-caef40939930]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-udp-8472"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-2379"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-icmp-0"]: Creation complete after 1s [id=d9db47b0-dcb3-4bbd-9dbe-af069808bf49]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-22"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.lb_server["192.168.44.0/24-tcp-6443"]: Creation complete after 1s [id=a4e57464-bd23-402a-87f1-ee687d1108ce]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-6443"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.lb_server["192.168.44.0/24-tcp-9345"]: Creation complete after 1s [id=0dd3bead-0faa-4a6d-aa29-f08c29ae64cf]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-tcp-10250"]: Creation complete after 1s [id=40356740-2886-483b-b8e4-ddd16533f1cc]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-2380"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-udp-8472"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.lb_server["192.168.44.0/24-tcp-22"]: Creation complete after 2s [id=ed7908ed-d208-497e-91fe-98928f114502]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-10250"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-udp-8472"]: Creation complete after 1s [id=aff34e13-0b1b-4ff7-b00f-f7d6d8b241ca]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-udp-8472"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-icmp-0"]: Creation complete after 2s [id=78c6a3d5-056d-4740-a786-e36229325f74]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-tcp-22"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-2379"]: Creation complete after 1s [id=b1b0005a-8d8e-4138-ac66-83a7f2661d25]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-10250"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-tcp-22"]: Creation complete after 0s [id=0272be99-4c07-4f45-a6ca-f0aab4ae0cab]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-icmp-0"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-udp-8472"]: Creation complete after 1s [id=7572d7d3-3d31-41b8-b88d-e708f04840b4]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-9345"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-22"]: Creation complete after 2s [id=1a6cec88-51af-46e4-8757-00c5bcf30fad]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-tcp-4240"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-6443"]: Creation complete after 2s [id=747a45c0-8d34-48fe-9b93-a23541780a0d]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-tcp-4240"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-tcp-4240"]: Creation complete after 0s [id=c358bb7e-91f3-4ece-a290-f96d90ada166]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-4240"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-2380"]: Creation complete after 2s [id=2da3f0a4-e70f-46d3-ab86-b6916c8cbf24]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-udp-8472"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-agent-tcp-4240"]: Creation complete after 0s [id=2eb24254-f33f-46b6-bbfe-8eabce1b7e61]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-9345"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-10250"]: Creation complete after 2s [id=244b8198-5abf-448e-967d-b1089e80881a]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-6443"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-udp-8472"]: Creation complete after 1s [id=909344cf-3219-4ed2-b5fe-0a835aee02e8]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-4240"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-udp-8472"]: Creation complete after 2s [id=f2a7dfcc-e389-432d-9fb9-0a20d637215e]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-icmp-0"]: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-agent-icmp-0"]: Creation complete after 0s [id=e276a970-bacf-4c3c-8d17-f9ef9642e166]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-10250"]: Creation complete after 2s [id=7280b2b5-7233-4c41-aba0-5b94d4dad9f0]
module.controlplane.openstack_networking_network_v2.net: Creation complete after 8s [id=84353e67-0a43-419a-a7b1-7cd49214e12a]
module.controlplane.openstack_networking_subnet_v2.servers: Creating...
module.controlplane.openstack_networking_subnet_v2.agents: Creating...
module.controlplane.openstack_networking_subnet_v2.lb: Creating...
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-icmp-0"]: Creation complete after 3s [id=094740e1-6d65-48ea-bffd-364d6a74681e]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-9345"]: Creation complete after 3s [id=46f6b80a-566a-4f09-9315-d05c26de85d5]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-4240"]: Creation complete after 3s [id=5e08cc9a-6374-4dab-b7b8-3d80ee002488]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-9345"]: Creation complete after 3s [id=f4cea762-6e49-4639-979f-09a4cd6aafa4]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-agent->rdmstack-server-tcp-6443"]: Creation complete after 2s [id=3ad9a96b-f7e7-422c-93ec-01d5a6b9b156]
module.controlplane.openstack_networking_secgroup_rule_v2.default["rdmstack-server->rdmstack-server-tcp-4240"]: Creation complete after 3s [id=f44b0da8-294a-4928-88b9-5e629cbd901a]
module.controlplane.openstack_networking_router_v2.router: Still creating... [10s elapsed]
module.controlplane.openstack_networking_subnet_v2.agents: Creation complete after 6s [id=5274c016-d67a-4692-847b-aa20a302a117]
module.controlplane.openstack_networking_subnet_v2.lb: Creation complete after 6s [id=7f5d3306-ac15-4f04-975d-6dbac52c1c78]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Creating...
module.controlplane.openstack_networking_subnet_v2.servers: Creation complete after 7s [id=617e5195-5485-4855-afbf-8bf52bdbed8d]
module.controlplane.openstack_networking_router_v2.router: Creation complete after 15s [id=7000c1b5-5b6a-4f03-9347-8ee8c5db5659]
module.controlplane.openstack_networking_router_interface_v2.lb: Creating...
module.controlplane.openstack_networking_router_interface_v2.servers: Creating...
module.controlplane.openstack_networking_router_interface_v2.agents: Creating...
module.controlplane.openstack_networking_router_interface_v2.lb: Creation complete after 8s [id=392150af-25de-430b-8b8f-26c7acd94976]
module.controlplane.openstack_networking_router_interface_v2.agents: Creation complete after 9s [id=e1a230fa-274c-4873-976a-b24aba1f2e7d]
module.controlplane.openstack_networking_router_interface_v2.servers: Creation complete after 9s [id=40066596-caa8-4c83-82ca-fc9dd9553cd4]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [10s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [20s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [30s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [40s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [50s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [1m1s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Still creating... [1m11s elapsed]
module.controlplane.openstack_lb_loadbalancer_v2.lb: Creation complete after 1m20s [id=a9c5863d-d7b1-41bf-a56a-067fedbe3c1e]
module.controlplane.openstack_networking_floatingip_v2.external: Creating...
module.controlplane.openstack_lb_listener_v2.ssh[0]: Creating...
module.controlplane.openstack_lb_listener_v2.rke2: Creating...
module.controlplane.openstack_lb_listener_v2.k8s: Creating...
module.controlplane.openstack_networking_floatingip_v2.external: Creation complete after 9s [id=cb16a7ab-84cf-4a31-b025-d882d4d3956a]
module.controlplane.openstack_lb_listener_v2.rke2: Creation complete after 9s [id=4378b54c-5a1f-4240-b8b1-3d9f592c3724]
module.controlplane.openstack_lb_pool_v2.rke2: Creating...
module.controlplane.openstack_lb_listener_v2.ssh[0]: Still creating... [10s elapsed]
module.controlplane.openstack_lb_listener_v2.k8s: Still creating... [10s elapsed]
module.controlplane.openstack_lb_listener_v2.ssh[0]: Creation complete after 14s [id=ebafae50-219a-47ac-af14-1bf91f6f25f0]
module.controlplane.openstack_lb_pool_v2.ssh[0]: Creating...
module.controlplane.openstack_lb_pool_v2.rke2: Creation complete after 8s [id=2a53d03c-8fb4-4c80-8555-72ad3871cc69]
module.controlplane.openstack_lb_monitor_v2.rke2: Creating...
module.controlplane.openstack_lb_listener_v2.k8s: Still creating... [20s elapsed]
module.controlplane.openstack_lb_listener_v2.k8s: Creation complete after 22s [id=e6d7fe13-a51c-4b7a-92a0-14e7202bb3bd]
module.controlplane.openstack_lb_pool_v2.k8s: Creating...
module.controlplane.module.servers["controlplane"].data.openstack_images_image_v2.image: Reading...
module.controlplane.module.servers["controlplane"].openstack_compute_servergroup_v2.servergroup: Creating...
module.controlplane.module.servers["controlplane"].openstack_blockstorage_volume_v3.volume[0]: Creating...
module.controlplane.module.servers["controlplane"].openstack_networking_port_v2.port[0]: Creating...
module.controlplane.module.servers["controlplane"].data.openstack_images_image_v2.image: Read complete after 0s [id=faf2b13b-1dc8-4e0a-9797-937878e6cb7b]
module.controlplane.module.servers["controlplane"].openstack_compute_servergroup_v2.servergroup: Creation complete after 0s [id=43e06129-c84a-4838-978d-2959ba9ec4f7]
module.controlplane.openstack_lb_pool_v2.ssh[0]: Still creating... [10s elapsed]
module.controlplane.openstack_lb_monitor_v2.rke2: Still creating... [10s elapsed]
module.controlplane.module.servers["controlplane"].openstack_networking_port_v2.port[0]: Creation complete after 6s [id=40688b9c-974f-469b-9d52-be749d013210]
module.controlplane.openstack_lb_pool_v2.ssh[0]: Creation complete after 16s [id=537d4f47-c304-4ec9-aa04-f0ec755cf8cb]
module.controlplane.openstack_lb_monitor_v2.ssh[0]: Creating...
module.controlplane.openstack_lb_pool_v2.k8s: Still creating... [10s elapsed]
module.controlplane.module.servers["controlplane"].openstack_blockstorage_volume_v3.volume[0]: Still creating... [10s elapsed]
module.controlplane.openstack_lb_monitor_v2.rke2: Creation complete after 16s [id=3ce03729-c503-4af3-b404-364ff6d16f1c]
module.controlplane.module.servers["controlplane"].openstack_blockstorage_volume_v3.volume[0]: Creation complete after 11s [id=2fc79b92-a5d6-4d4b-a07f-0a30eed8c24e]
module.controlplane.module.servers["controlplane"].openstack_compute_instance_v2.instance[0]: Creating...
module.controlplane.openstack_lb_monitor_v2.ssh[0]: Still creating... [10s elapsed]
module.controlplane.openstack_lb_pool_v2.k8s: Creation complete after 19s [id=63fbe0af-aaf1-4c90-b969-45544daf6efb]
module.controlplane.openstack_lb_monitor_v2.k8s: Creating...
module.controlplane.openstack_lb_monitor_v2.ssh[0]: Creation complete after 12s [id=36374521-10dd-4ebb-b748-47f11792c699]
module.controlplane.module.servers["controlplane"].openstack_compute_instance_v2.instance[0]: Still creating... [10s elapsed]
module.controlplane.openstack_lb_monitor_v2.k8s: Creation complete after 9s [id=466fadfa-9467-4f69-b0a2-da3add181d2f]
module.controlplane.module.servers["controlplane"].openstack_compute_instance_v2.instance[0]: Still creating... [20s elapsed]
module.controlplane.module.servers["controlplane"].openstack_compute_instance_v2.instance[0]: Creation complete after 25s [id=a4d4d52a-c1a8-4b69-a693-a78524a8fc86]
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0]: Creating...
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0]: Provisioning with 'remote-exec'...
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Connecting to remote host via SSH...
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Host: 192.168.42.159
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): User: ubuntu
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Password: false
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Private key: false
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Certificate: false
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): SSH Agent: true
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Checking Host Key: false
module.controlplane.module.servers["controlplane"].null_resource.wait_for_rke2[0] (remote-exec): Target Platform: unix
I did set bootstrap=true.
Any hints what I might do wrong? I can't grant access to the OpenStack cloud, I could post my modules (nothing secret in there).
Thanks!
OK I have to correct myself: the node is never added as a member. It seems that either local.server_nodes or the SSH rules in loadbalancer.tf are empty (although I doubt that it's the rules, as the pool, monitor, etc get created)
OK so in the execution plan, I do see the module.controlplane.openstack_lb_members_v2.ssh being planned.
Looking at this from the other side, starting at wait_for_rke2, I see that the connection only will use a bastion host if this is not a server. It smells like there is a bug here.
@poikilotherm This is the expected behaviour. As OpenStack load balancer might take a few seconds/minutes to be ready, the first apply can last up to 5 minutes. If the process lasts longer, something might indeed have gone wrong. Does that help?
We are in the process of removing this bottleneck and should have a solution ready next week.
closing as dormant and v3 should solve that pain