flatcar-openstack-hostname.service not using config-drive as default for scraping metadata
alejandro-ripoll opened this issue · comments
Description
When booting up a Flatcar instance in OpenStack using Ignition and config-drive, flatcar-openstack-hostname.service is retrieving information from metadata API instead of trying config-drive first.
Impact
Information on config-drive is ignored which might be a problem on environments where metadata API cannot be used.
Environment and steps to reproduce
- Set-up: Create an OpenStack instance using "Flatcar Container Linux by Kinvolk 3815.2.2 for Openstack" image. Enable config-drive for it and provide any ignition config via userdata.
- Task: Boot up the instance.
- Action(s): Boot up the instance.
- Error: Checking the logs of flatcar-openstack-hostname.service, service is connecting to metadata API instead of using existing config-drive.
Expected behavior
Existing values under /media/configdrive/ should be used. As implemented on coreos/afterburn#462.
Additional information
Service logs:
# journalctl -u flatcar-openstack-hostname.service
May 16 11:29:31 localhost systemd[1]: Starting flatcar-openstack-hostname.service - Flatcar OpenStack Metadata Hostname Agent...
May 16 11:29:31 localhost coreos-metadata[849]: May 16 11:29:31.503 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #1
May 16 11:30:01 localhost coreos-metadata[849]: May 16 11:30:01.504 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:30:02 localhost coreos-metadata[849]: May 16 11:30:02.504 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #2
May 16 11:30:32 localhost coreos-metadata[849]: May 16 11:30:32.504 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:30:34 localhost coreos-metadata[849]: May 16 11:30:34.505 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #3
May 16 11:31:04 localhost coreos-metadata[849]: May 16 11:31:04.505 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:31:08 localhost coreos-metadata[849]: May 16 11:31:08.505 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #4
May 16 11:31:38 localhost coreos-metadata[849]: May 16 11:31:38.506 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:31:43 localhost coreos-metadata[849]: May 16 11:31:43.506 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #5
May 16 11:32:13 localhost coreos-metadata[849]: May 16 11:32:13.506 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:32:18 localhost coreos-metadata[849]: May 16 11:32:18.507 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #6
May 16 11:32:48 localhost coreos-metadata[849]: May 16 11:32:48.507 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:32:53 localhost coreos-metadata[849]: May 16 11:32:53.507 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #7
May 16 11:33:23 localhost coreos-metadata[849]: May 16 11:33:23.508 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:33:28 localhost coreos-metadata[849]: May 16 11:33:28.509 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #8
May 16 11:33:58 localhost coreos-metadata[849]: May 16 11:33:58.509 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:34:03 localhost coreos-metadata[849]: May 16 11:34:03.509 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #9
May 16 11:34:33 localhost coreos-metadata[849]: May 16 11:34:33.510 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:34:38 localhost coreos-metadata[849]: May 16 11:34:38.510 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #10
May 16 11:35:08 localhost coreos-metadata[849]: May 16 11:35:08.510 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:35:13 localhost coreos-metadata[849]: May 16 11:35:13.510 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #11
May 16 11:35:43 localhost coreos-metadata[849]: May 16 11:35:43.511 INFO Failed to fetch: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:35:43 localhost coreos-metadata[849]: Error: failed to run
May 16 11:35:43 localhost coreos-metadata[849]: Caused by:
May 16 11:35:43 localhost coreos-metadata[849]: 0: writing hostname
May 16 11:35:43 localhost coreos-metadata[849]: 1: maximum number of retries (10) reached
May 16 11:35:43 localhost coreos-metadata[849]: 2: failed to fetch
May 16 11:35:43 localhost coreos-metadata[849]: 3: error sending request for url (http://169.254.169.254/latest/meta-data/hostname): operation timed out
May 16 11:35:43 localhost coreos-metadata[849]: 4: operation timed out
May 16 11:35:43 localhost systemd[1]: flatcar-openstack-hostname.service: Main process exited, code=exited, status=1/FAILURE
May 16 11:35:43 localhost systemd[1]: flatcar-openstack-hostname.service: Failed with result 'exit-code'.
May 16 11:35:43 localhost systemd[1]: Failed to start flatcar-openstack-hostname.service - Flatcar OpenStack Metadata Hostname Agent.
Config drive exists:
# ls -ltr /media/configdrive/openstack/latest/meta_data.json
-r--r--r--. 1 root root 1981 May 16 11:29 /media/configdrive/openstack/latest/meta_data.json
Ignition used:
# cat /etc/.ignition-result.json
{
"provisioningBootID": "4963079c-b3fb-4ebe-a983-110b783c2c80",
"provisioningDate": "2024-05-16T11:35:53Z",
"userConfigProvided": true
}
Unsure whether this is relevant, but https://github.com/flatcar/bootengine/blob/flatcar-master/dracut/30ignition/flatcar-openstack-hostname.service#L17 is using openstack-metadata instead of openstack.
Hello @alejandro-ripoll and sorry for the delay. After some archeology work, I found that this change was making sense: https://github.com/flatcar/bootengine/blob/8da532c809c89a9c434ada0fa9532a1c1bf49f4c/dracut/30ignition/flatcar-openstack-hostname.service#L16-L17 because at this time, an old Afterburn version was still being used.
Would you be interested to contribute to Flatcar to fix the file you mentioned? It's quite easy: fork the https://github.com/flatcar/bootengine repo, then update the bootengine ebuild to target your fork with your changes:
- https://github.com/flatcar/scripts/blob/7c3a3d384f3d3f75e94fbf44583e246b72b7b718/sdk_container/src/third_party/coreos-overlay/sys-kernel/bootengine/bootengine-9999.ebuild#L5
- https://github.com/flatcar/scripts/blob/7c3a3d384f3d3f75e94fbf44583e246b72b7b718/sdk_container/src/third_party/coreos-overlay/sys-kernel/bootengine/bootengine-9999.ebuild#L13
Once done, I'll build an OpenStack image that you could test to see if it solves the issue. Let me know!
Thanks a lot for your response @tormath1.
I've forked both bootengine and flatcar-scripts and applied my changes on:
Not sure what the next step would be. Should I raise a PR for bootengine?
Thanks again.
Perfect thanks. You can now open a pull request for bootengine and scripts for reviewing the changes, I'll start a Flatcar build to try your changes and assert that it does not bring regression on OpenStack platform.