Packer fails to build Vagrant box for Ubuntu 20.04
fasmat opened this issue · comments
Feature Description
Ubuntu has switched its installer from the debian installer to subiquity. Starting with 20.04 no more alternate images will be provided: https://discourse.ubuntu.com/t/server-installer-plans-for-20-04-lts/13631
This breaks unattended packer builds (vmware-iso, virtualbox-iso, hyperv-iso) for Ubuntu. Starting with 20.04 no alternate images seem to be available that could be used until recently to build vagrant boxes with packer.
Use Case(s)
If unattended installations for ubuntu will be supported in the future, subiquity has to be used instead of d-i.
Right, we need packer template for cloud-init config.
Some part of code:
"boot_command": [
"<enter><wait><enter><wait><f6><esc>",
"autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/",
"<enter>"
],
One problem I have is that grub now automatically boots the live system after a few seconds and before packer starts typing the boot_command over VNC. Maybe this can be adjusted with the boot_wait parameter but I didn't try yet.
"iso_checksum": "36f15879bd9dfd061cd588620a164a82972663fdd148cce1f70d57d314c21b73",
"iso_url": "http://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/ubuntu-20.04-legacy-server-amd64.iso",
@JulyIghor Thanks for the link to the legacy installer. This can be used as a substitute for now, but Ubuntu has been using subiquity since Ubuntu 18.04 as the official installer and with 20.04 dropped support for d-i.
On top of that, the new installer runs its own ssh server during installation which packer picks up on in error.
We actuslly need to let the autoinstall finish AND reboot, which THEN runs the ssh server we want.
No idea how to accomplish that so far. :/
On top of that, the new installer runs its own ssh server during installation which packer picks up on in error.
We actuslly need to let the autoinstall finish AND reboot, which THEN runs the ssh server we want.
No idea how to accomplish that so far. :/
Actually with cloud-init we can do everything without SSH server at all.
So this packer template is deprecated.
I have managed to use this #9115 (comment) and get #cloud-config applied. But it does not work in UEFI mode.
As I understand it requires to add additional parameters to initd command line.
I would fix it if I knew Ubuntu better.
I got grub command line, what to do next to get it booted?
@JulyIghor how did you achieve to get to the grub command line? For me Grub always boots up ubuntu before packer starts typing, so unless I manually intervene the boot command isn't sent to the VM before it is too late.
I'm unfortunately also not sure how to proceed. According to https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart the boot_command you provided should already be sufficient to install the ubuntu system, assuming the cloud-config contains all information necessary for an autoinstall.
@JulyIghor how did you achieve to get to the grub command line? For me Grub always boots up ubuntu before packer starts typing, so unless I manually intervene the boot command isn't sent to the VM before it is too late.
I'm unfortunately also not sure how to proceed. According to https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart the boot_command you provided should already be sufficient to install the ubuntu system, assuming the cloud-config contains all information necessary for an autoinstall.
You actually just need to be quick enough to stop the ISO's grub boot loader from starting the installer and instead type the boot command.
In my experiments, based on the work of @geerlingguy and @nickcharlton, a boot wait of 5s was spot on. The exact value for you may differ depending on the system performance.
@JulyIghor how did you achieve to get to the grub command line? For me Grub always boots up ubuntu before packer starts typing, so unless I manually intervene the boot command isn't sent to the VM before it is too late.
I'm unfortunately also not sure how to proceed. According to https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart the boot_command you provided should already be sufficient to install the ubuntu system, assuming the cloud-config contains all information necessary for an autoinstall.
I was able to do it only when UEFI disabled in VM bios. And this is not we need.
FYI - this is the work from @nickcharlton with a working cloud-init configuration template for 20.04:
In the mean time, if someone wants to take a look at this, I've written up my notes which have a working configuration: https://nickcharlton.net/posts/automating-ubuntu-2004-installs-with-packer.html
On top of that, the new installer runs its own ssh server during installation which packer picks up on in error.
We actuslly need to let the autoinstall finish AND reboot, which THEN runs the ssh server we want.
No idea how to accomplish that so far. :/
On this problem specifically, @rubenst2013, a potential solution is to use pause_before_connecting to stop us connecting to the installer's ssh.
In practice with my experiments, I'd been seeing random build failures because of it.
FYI - this is the work from @nickcharlton with a working cloud-init configuration template for 20.04:
In the mean time, if someone wants to take a look at this, I've written up my notes which have a working configuration: https://nickcharlton.net/posts/automating-ubuntu-2004-installs-with-packer.html
HTTP url is correct? or it should be http://{{.HTTPIP}}:{{.HTTPPort}} ?
FYI - this is the work from @nickcharlton with a working cloud-init configuration template for 20.04:
In the mean time, if someone wants to take a look at this, I've written up my notes which have a working configuration: https://nickcharlton.net/posts/automating-ubuntu-2004-installs-with-packer.html
on Parallels Desktop it does not work, ALT+F2 doing nothing, but 'E' does open editor where I can add command.
Ah, nice catch @JulyIghor. It should indeed be: http://{{ .HTTPIP }}:{{ .HTTPPort }}/. Jekyll/the markdown parser is breaking it.
This boot command should work, but it doesn't
"boot_command": [
"<tab><tab><tab><tab><tab><c><wait><bs><bs>",
"set gfxpayload=keep", "<enter>",
"linux /casper/vmlinuz quiet autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/ ---", "<enter>",
"initrd /casper/initrd", "<enter>",
"boot", "<enter>"
]
This boot command should work, but it doesn't
"boot_command": [ "<tab><tab><tab><tab><tab><c><wait><bs><bs>", "set gfxpayload=keep", "<enter>", "linux /casper/vmlinuz quiet autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/ ---", "<enter>", "initrd /casper/initrd", "<enter>", "boot", "<enter>" ]
It did no requests to http server, and I got Language selection dialog
I'm wondering if the existing value for the boot options is what's causing this (unless you're entering them differently from my one): I'm appending autoinstall ... onto what's already there, so the boot command ends up being:
initrd=/casper/initrd quiet -- autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/
I'm wondering if the existing value for the boot options is what's causing this (unless you're entering them differently from my one): I'm appending
autoinstall ...onto what's already there, so the boot command ends up being:initrd=/casper/initrd quiet -- autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/
@nickcharlton @JulyIghor We could schedule a workshop via discord or some other platform to figure this out and then post the results here. what do you think? 💡
@nickcharlton @JulyIghor We could schedule a workshop via discord or some other platform to figure this out and then post the results here. what do you think? 💡
https://t.me/joinchat/CO-Y3hxWngKWrsmTUsPV7Q - the chat history lost, sorry
I'm wondering if the existing value for the boot options is what's causing this (unless you're entering them differently from my one): I'm appending
autoinstall ...onto what's already there, so the boot command ends up being:initrd=/casper/initrd quiet -- autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/
Here is working grub command line
set gfxpayload=keep
linux /casper/vmlinuz "ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/" quiet autoinstall ---
initrd /casper/initrd
boot
Hi @JulyIghor,
The quotes on the command line didn't work for me when I tried them out of curiosity.
Though I think you are simply missing a dash. As explained here, grub uses a tripple-dash to separate different command line parts:
https://stackoverflow.com/questions/11552950/triple-dash-on-linux-kernel-command-line-switches
Hi @JulyIghor,
The quotes on the command line didn't work for me when I tried them out of curiosity.
Though I think you are simply missing a dash. As explained here, grub uses a tripple-dash to separate different command line parts:
https://stackoverflow.com/questions/11552950/triple-dash-on-linux-kernel-command-line-switches
This example is about grub config. I'm talking about grub command line.
If I remove quotes, it just ignored. As it said in the grub welcome screen, it works bash like. So quotes required to escape some characters I think.
@nickcharlton I'm trying to get your example to work on MacOS with Qemu and I see the SSH handshaking timing out eventually, but also when connecting over VNC I can see the setup screen sitting there and not progressing. Also the password "ubuntu" doesn't work for the SSH server that's started.
{
"builders": [
{
"boot_command": [
"<enter><enter><f6><esc><wait> ",
"autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
"<enter>"
],
"boot_wait": "5s",
"disk_interface": "virtio",
"format": "qcow2",
"http_directory": "http",
"iso_checksum": "sha256:caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f",
"iso_url": "http://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso",
"memory": 1024,
"name": "ubuntu-2004",
"net_device": "virtio-net",
"shutdown_command": "echo 'packer' | sudo -S shutdown -P now",
"ssh_timeout": "20m",
"ssh_password": "ubuntu",
"ssh_username": "ubuntu",
"vm_name": "ubuntu-install",
"type": "qemu",
"headless": true
}
],
"provisioners": [
{
"inline": [
"ls /"
],
"type": "shell"
}
]
}If you can offer up any guidance it would be appreciated.
This is based upon your example repo with the Qemu provider in place.
Right before that I see:
Then:
Finally the installer starts and sits there, or it just bombs out with this error:
2020/06/24 13:03:41 packer-builder-qemu plugin: failed to unlock port lockfile: close tcp 127.0.0.1:5970: use of closed network connection
2020/06/24 13:03:41 packer-builder-qemu plugin: failed to unlock port lockfile: close tcp 127.0.0.1:3368: use of closed network connection
==> ubuntu-2004: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> ubuntu-2004: Deleting output directory...
2020/06/24 13:03:41 [INFO] (telemetry) ending qemu
2020/06/24 13:03:41 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2020/06/24 13:03:41 machine readable: ubuntu-2004,error []string{"Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain"}
==> Builds finished but no artifacts were created.
2020/06/24 13:03:41 [INFO] (telemetry) Finalizing.
Build 'ubuntu-2004' errored: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> Some builds didn't complete successfully and had errors:
--> ubuntu-2004: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> Builds finished but no artifacts were created.
2020/06/24 13:03:42 waiting for all plugin processes to complete...
2020/06/24 13:03:42 /usr/local/bin/packer: plugin process exited
2020/06/24 13:03:42 /usr/local/bin/packer: plugin process exited
Otherwise it just hangs with Nick's example here:
Having tried another well-documented example with QEMU on MacOS, I get the same result -
https://imagineer.in/blog/packer-build-for-ubuntu-20-04/
@JulyIghor how did you achieve to get to the grub command line? For me Grub always boots up ubuntu before packer starts typing, so unless I manually intervene the boot command isn't sent to the VM before it is too late.
I'm unfortunately also not sure how to proceed. According to https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart the boot_command you provided should already be sufficient to install the ubuntu system, assuming the cloud-config contains all information necessary for an autoinstall.
Did you have any success? I am facing the same issue.
@JulyIghor how did you achieve to get to the grub command line? For me Grub always boots up ubuntu before packer starts typing, so unless I manually intervene the boot command isn't sent to the VM before it is too late.
I'm unfortunately also not sure how to proceed. According to https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart the boot_command you provided should already be sufficient to install the ubuntu system, assuming the cloud-config contains all information necessary for an autoinstall.Did you have any success? I am facing the same issue.
I used it with Parallels Desktop, so maybe it is difference.
This gets a lot further with VirtualBox. I haven't seen if it gets all the rest to the end but it got much further along and is doing the installation. Timing issue with qemu on Mac perhaps?
With VirtualBox, this is how far it gets:
==> ubuntu-20.04-live-server: Connected to SSH!
==> ubuntu-20.04-live-server: Uploading VirtualBox version info (6.1.10)
==> ubuntu-20.04-live-server: Uploading VirtualBox guest additions ISO...
==> ubuntu-20.04-live-server: Provisioning with shell script: /var/folders/p7/ptxtv_pd3n12fd6mc5wjhk1h0000gn/T/packer-shell078496510
ubuntu-20.04-live-server: bin dev lib libx32 mnt root snap sys var
ubuntu-20.04-live-server: boot etc lib32 lost+found opt run srv tmp
ubuntu-20.04-live-server: cdrom home lib64 media proc sbin swap.img usr
==> ubuntu-20.04-live-server: Gracefully halting virtual machine...
ubuntu-20.04-live-server: Failed to set wall message, ignoring: Interactive authentication required.
ubuntu-20.04-live-server: Failed to power off system via logind: Interactive authentication required.
ubuntu-20.04-live-server: Failed to open initctl fifo: Permission denied
ubuntu-20.04-live-server: Failed to talk to init daemon.
==> ubuntu-20.04-live-server: Timeout while waiting for machine to shutdown.
==> ubuntu-20.04-live-server: Provisioning step had errors: Running the cleanup provisioner, if present...
==> ubuntu-20.04-live-server: Cleaning up floppy disk...
==> ubuntu-20.04-live-server: Deregistering and deleting VM...
==> ubuntu-20.04-live-server: Deleting output directory...
Build 'ubuntu-20.04-live-server' errored: Timeout while waiting for machine to shutdown.
==> Some builds didn't complete successfully and had errors:
--> ubuntu-20.04-live-server: Timeout while waiting for machine to shutdown.
==> Builds finished but no artifacts were created.
Config:
{
"builders": [
{
"boot_command": [
"<enter><enter><f6><esc><wait> ",
"autoinstall ds=nocloud-net;seedfrom=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
"<enter><wait>"
],
"boot_wait": "5s",
"format": "ovf",
"headless": true,
"http_directory": "http",
"iso_checksum": "sha256:caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f",
"iso_urls": [
"iso/ubuntu-20.04-live-server-amd64.iso",
"https://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso"
],
"memory": 1024,
"name": "ubuntu-20.04-live-server",
"output_directory": "output/live-server",
"shutdown_command": "shutdown -P now",
"ssh_handshake_attempts": "20",
"ssh_password": "ubuntu",
"ssh_pty": true,
"ssh_timeout": "20m",
"ssh_username": "ubuntu",
"type": "virtualbox-iso",
"guest_os_type": "Ubuntu_64"
}
],
"provisioners": [
{
"inline": [
"ls /"
],
"type": "shell"
}
]
}
With http/user-data:
#cloud-config
autoinstall:
version: 1
locale: en_US
keyboard:
layout: en
variant: us
network:
network:
version: 2
ethernets:
ens33:
dhcp4: true
storage:
layout:
name: lvm
identity:
hostname: ubuntu
username: ubuntu
password: $6$rounds=4096$8dkK1P/oE$2DGKKt0wLlTVJ7USY.0jN9du8FetmEr51yjPyeiR.zKE3DGFcitNL/nF1l62BLJNR87lQZixObuXYny.Mf17K1
ssh:
install-server: yes
user-data:
disable_root: false
late-commands:
- 'sed -i "s/dhcp4: true/&\n dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
- echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/ubuntu
@alexellis in my repository: https://github.com/fasmat/ubuntu you find a working configuration for both Virtualbox and VMWare.
The end game is to produce a small image that can be dd'd to a machine. I just tried that and got a bit further, but the raw image file is 39GB!
space-mini:live-server2 alex$ # VBoxManage clonehd ./packer-ubuntu-20.04-live-server-1593009318-disk001.vmdk image.img --format raw
space-mini:live-server2 alex$ du -h image.img
39G image.img
space-mini:live-server2 alex$
@fasmat I think that brings us back to the original issue - I don't need a box image and don't want to use Vagrant, this is to produce an image that can be flashed to the hard disk of a real server. Would your example also work for Ubuntu Server (subject to changing the ISO image URL and SHA)?
It uses the ubuntu live server iso to create the VM, the final image (after installing the ubuntu-desktop package and some other tools) is ~ 1 GB. I'm building the images on macOS.
@alexellis concerning the size of the image: it's a raw image containing the whole hard disk. If you zip this file you will see that the file easily compresses to < 1 GB. If you want the file to be smaller uncompressed then you have to setup the partition size during the installation to be less than 40 (which is I believe the default).
Yes of course: 960M image.img.tgz <- through tar zcf
@alexellis in my repository: https://github.com/fasmat/ubuntu you find a working configuration for both Virtualbox and VMWare.
Maybe your configuration works due to the headless parameter. I am using the vsphere-iso. The boot_command works right after the first installation interface (GUI) only. Then, I have to esc myself and wait the boot_command to write the commands. The headless is not supported for the vsphere-iso.
So far as I understand it, the issue you're getting @alexellis is that the installer's SSH session is being connected to. The password is set for the user that's (ubuntu) in the user-data, but the installer one is randomised (you can get this by watching the installer manually, but in our case here we don't want that).
Increasing the SSH handshake just allows Packer to fail a lot (i.e.: during the install) until the machine is booted after the install is completed. This is what's causing everyone to have such a non-determinate install experience — 20 is just what seemed to work reliably for me. I'd try bumping it up and seeing what happens with Packer in debug mode.
The other option is to change the port the installer SSH is opened on (which I think is in this issue or one linked from it somewhere).
Otherwise it just hangs with Nick's example here:
This is because you are missing the window to escape out of the live boot. Try changing the "boot_wait" to 2 seconds.
Just wanted to add I'm experiencing the same issue with the live cd (though not using Vagrant, just creating a vSphere template). I've tried incrementing the boot_wait value 1s at a time and it's just not consistent with how long it takes at startup.
I found the following Medium post (https://medium.com/@tlhakhan/ubuntu-server-20-04-autoinstall-2e5f772b655a) that touches on this issue a bit and suggests using a iPXE boot environment over packer but this obviously isn't an option for everyone (including me).
@stevenmiller I have it working on vsphere-iso with a 2s wait.
@stevenmiller I have it working on vsphere-iso with a 2s wait.
I just tried with 2s again and it worked, however the build ended up erroring out with an separate ssh communicator problem. Went to re-run the packer build and it hasn't worked since, it always misses and ends up at the language selection screen of the live installer.
@stevenmiller here is what I use:
{
"CPUs": "{{ user `cpus` }}",
"RAM": "{{ user `memory` }}",
"RAM_reserve_all": true,
"boot_command": [
"<esc><esc><esc>",
"<enter><wait>",
"/casper/vmlinuz ",
"root=/dev/sr0 ",
"initrd=/casper/initrd ",
"autoinstall ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/",
"<enter>"
],
"boot_wait": "2s",
"convert_to_template": true,
"disk_controller_type": "pvscsi",
"guest_os_type": "ubuntu64Guest",
"host": "{{user `esxi_host`}}",
"insecure_connection": "true",
"iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
"iso_urls": [
"{{user `iso_path`}}/{{user `iso_name`}}"
],
"network_adapters": [
{
"network": "VM Network",
"network_card": "vmxnet3"
}
],
"password": "{{user `esxi_password`}}",
"ssh_password": "jacaladmin",
"ip_settle_timeout": "5m",
"ssh_port": 22,
"ssh_username": "jacal",
"storage": [
{
"disk_size": "{{ user `disk_size` }}",
"disk_thin_provisioned": true
}
],
"type": "vsphere-iso",
"username": "{{user `esxi_username`}}",
"vcenter_server": "{{user `vcenter_server`}}",
"vm_name": "{{ user `template` }}",
"http_directory": "{{template_dir}}/http",
"ip_wait_address": "{{user `wait_address`}}",
"tools_sync_time": true
}
Packer version:
goffinf@DESKTOP-LH5LG1V:~$ packer version
Packer v1.6.0
Builder: proxmox
Looks like plenty of us are seeing a similar problem. I too just get stuck .. Waiting for SSH to become available
The boot_command does appear to get me through to the right place (initially the boot_command wasn't quick enough and I ended up at the language selection for the manual install as @alexellis described above, but reducing boot_wait to 2s fixed that). I added a wait20 after the first enter so I could check ..
"builders": [
{
"boot_command": [
"<enter><wait20><enter><f6><esc><wait>",
"autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
"<enter>"
],
...
I see this during the wait:
Then this after:
and then this ....
until SSH times out and Packer cleans up ...
I think I have looked at all the links within this thread and many others .. and whilst there have been a number of helpful suggestions I haven't been able to find anything that allows SSH to connect
I did note that if you manually install, and auto-install user-data config is stored at this location
/var/log/installer/autoinstall-user-data.
So I tried using that, but, not unexpectedly, it didn't get around the SSH connection issue.
I have attached the Packer json and user-data that I am using below (meta-data is an empty file per the instructions).
If anyone has any further suggestions I would be most grateful.
Regards
Fraser.
host.json
{
"builders": [
{
"boot_command": [
"<enter><wait20><enter><f6><esc><wait>",
"autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
"<enter>"
],
"boot_wait": "{{user `boot_wait`}}",
"disks": [
{
"disk_size": "{{user `home_volume_size`}}",
"storage_pool": "local-lvm",
"storage_pool_type": "lvm-thin",
"type": "scsi",
"format": "raw"
}
],
"http_directory": "{{user `http_directory`}}",
"insecure_skip_tls_verify": true,
"iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
"iso_file": "{{user `iso_file`}}",
"memory": 2048,
"name": "ubuntu-20-04-base",
"network_adapters": [
{
"bridge": "vmbr0",
"model": "virtio"
}
],
"node": "{{user `proxmox_target_node`}}",
"password": "{{user `proxmox_server_pwd`}}",
"proxmox_url": "https://{{user `proxmox_server_hostname`}}:{{user `proxmox_server_port`}}/api2/json",
"ssh_handshake_attempts": "50",
"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",
"ssh_pty": true,
"ssh_timeout": "{{user `ssh_timeout`}}",
"type": "proxmox",
"unmount_iso": true,
"username": "{{user `proxmox_server_user`}}"
}
],
"provisioners": [
{
"execute_command": "{{ .Vars }} sudo -E -S sh '{{ .Path }}'",
"inline": [
"ls /"
],
"type": "shell"
}
],
"variables": {
"boot_wait": "2s",
"http_directory": "http",
"iso_checksum": "caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f",
"iso_checksum_type": "sha256",
"iso_file": "local:iso/ubuntu-20.04-live-server-amd64.iso",
"proxmox_server_hostname": "proxmox-002",
"proxmox_server_port": "8006",
"proxmox_server_pwd": "xxxxxxxxxx",
"proxmox_server_user": "root@pam",
"proxmox_target_node": "home",
"ssh_handshake_attempts": "20",
"ssh_password": "ubuntu",
"ssh_username": "ubuntu",
"ssh_timeout": "10m"
}
}
user-data:
#cloud-config
autoinstall:
identity:
hostname: ubuntu-20-04-base
password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
#password: '$6$wdAcoXrU039hKYPd$508Qvbe7ObUnxoj15DRCkzC3qO7edjH0VV7BPNRDYK4QR8ofJaEEF2heacn0QgD.f8pO8SNp83XNdWG6tocBM1'
username: ubuntu
keyboard:
layout: en
variant: 'gb'
late-commands:
- sed -i 's/^#*\(send dhcp-client-identifier\).*$/\1 = hardware;/' /target/etc/dhcp/dhclient.conf
- 'sed -i "s/dhcp4: true/&\n dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
locale: en_GB
network:
network:
version: 2
ethernets:
ens18:
dhcp4: true
dhcp-identifier: mac
ssh:
allow-pw: true
authorized-keys:
- "ssh-rsa AAAAB3N......"
install-server: true
version: 1
Also experiencing this bailing out on Waiting for SSH to become available... in the vsphere-iso builder. Tried to tweak with "boot_wait": "2s" as suggested above by @jhawk28, varying it between 2 and 5s with fingers crossed. Also adjusted my packer json to match that shared in #9115 (comment) but no luck so far.
Just to Share with everyone what has been working for me.
ubuntu2004_x64.json:
{
"builders": [
{
"CPUS": "{{ user `CPUS` }}",
"RAM": "{{ user `RAM` }}",
"boot_command": [
"<esc><esc><esc>",
"<enter><wait>",
"/casper/vmlinuz ",
"root=/dev/sr0 ",
"initrd=/casper/initrd ",
"autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ubuntu/",
"<enter>"
],
"boot_order": "disk,cdrom",
"boot_wait": "5s",
"cluster": "{{ user `cluster` }}",
"communicator": "ssh",
"convert_to_template": true,
"datacenter": "{{ user `datacenter` }}",
"datastore": "{{ user `datastore` }}",
"disk_controller_type": "pvscsi",
"folder": "{{ user `folder` }}",
"guest_os_type": "{{ user `guest_os_type` }}",
"http_directory": "{{ template_dir }}/../http",
"insecure_connection": "{{ user `insecure_connection` }}",
"ip_settle_timeout": "5m",
"iso_checksum": "{{ user `iso_checksum` }}",
"iso_urls": ["{{ user `iso_url` }}"],
"network_adapters": [
{
"network": "{{ user `network` }}",
"network_card": "vmxnet3"
}
],
"password": "{{ user `password` }}",
"resource_pool": "",
"shutdown_command": "{{ user `shutdown_command` }}",
"ssh_handshake_attempts": "20",
"ssh_password": "{{ user `ssh_password` }}",
"ssh_username": "{{ user `ssh_username` }}",
"storage": [
{
"disk_size": "{{ user `disk_size` }}",
"disk_thin_provisioned": true
}
],
"type": "vsphere-iso",
"username": "{{ user `username` }}",
"vcenter_server": "{{ user `vcenter_server` }}",
"vm_name": "{{ user `vm_name` }}-{{ timestamp }}",
"vm_version": "{{ user `vm_version` }}"
}
],
"post-processors": [
{
"output": "{{ template_dir }}/packer-manifest.json",
"strip_path": true,
"type": "manifest"
}
],
"provisioners": [
{
"scripts": [
"{{ template_dir }}/../scripts/base.sh",
"{{ template_dir }}/../scripts/vmware.sh",
"{{ template_dir }}/../scripts/cleanup.sh",
"{{ template_dir }}/../scripts/zerodisk.sh"
],
"type": "shell"
}
],
"variables": {
"CPUS": "1",
"RAM": "1024",
"cluster": "",
"datacenter": "",
"datastore": "",
"disk_size": "8192",
"folder": "",
"guest_os_type": "ubuntu64Guest",
"insecure_connection": "true",
"iso_checksum": "caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f",
"iso_url": "http://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso",
"network": "",
"password": "",
"shutdown_command": "sudo /sbin/halt -p",
"ssh_password": "",
"ssh_username": "ubuntu",
"username": "",
"vcenter_server": "",
"vm_name": "ubuntu2004_x64",
"vm_version": "10"
}
}user-data:
#cloud-config
autoinstall:
version: 1
early-commands:
# Block inbound SSH to stop Packer trying to connect during initial install
- iptables -A INPUT -p tcp --dport 22 -j DROP
identity:
hostname: ubuntu-server
password: "$6$3yklPgGbsS$yqLzE7Oag1Bk97a/tpAnr5BpgysH.6lpSoROGhyrlbGkHKmZ/hwWZytPXhClUCXFH2w61zC0Poot48bMXjDJF1" # generate with mkpasswd -m sha-512
username: ubuntu
late-commands:
- sed -i 's/^%sudo.*/%sudo ALL=(ALL:ALL) NOPASSWD:ALL/g' /etc/sudoers
- sed -i 's/^#*\(send dhcp-client-identifier\).*$/\1 = hardware;/' /target/etc/dhcp/dhclient.conf
- "echo 'Defaults:ubuntu !requiretty' > /target/etc/sudoers.d/ubuntu"
- "echo 'ubuntu ALL=(ALL) NOPASSWD: ALL' >> /target/etc/sudoers.d/ubuntu"
- "chmod 440 /target/etc/sudoers.d/ubuntu"
- 'sed -i "s/dhcp4: true/&\n dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
packages:
- bc
- curl
- lsb-release
- ntp
- open-vm-tools
- openssh-server
- wget
ssh:
# For now we install openssh-server during package installs
allow-pw: true
install-server: false
storage:
layout:
name: direct
config:
- type: disk
id: disk0
match:
size: largest
- type: partition
id: boot-partition
device: disk0
size: 1024M
- type: partition
id: root-partition
device: disk0
size: -1I've had the most success with the 2s boot_wait time, but it's still a crapshoot if it makes the window or not. I had it work one time last week and it hasn't worked for me since. It misses its window to enter the boot_command and ends up at the language selection screen of the installer.
I'm running into the same issue that @alexellis was running into where it's giving me the language selection screen. I can, however, see packer enter the boot_commands so I don't think I'm missing the window.
Here are my boot commands:
"boot_command": [
"<esc><wait><esc><wait><f6><wait><esc><wait>",
"autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ ",
"<enter>"
],
I've manually tried to reproduce these steps but I keep getting the Welcome/Language selection screen.
My boot_wait is set to 2s.
I'm using the proxmox builder.
Any suggestions?
@DarrellTang Have you tried the following?
"boot_command": [
"<esc><esc><esc>",
"<enter><wait>",
"/casper/vmlinuz ",
"root=/dev/sr0 ",
"initrd=/casper/initrd ",
"autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ubuntu/",
"<enter>"
],
"boot_wait": "2s",Also, to everyone else:
I have gone through qemu, virtualbox-iso, vmware-iso, and vsphere-iso builders and I get consistent builds. The other important thing that I found worked for me was to specifically set the following in user-data:
early-commands:
# Block inbound SSH to stop Packer trying to connect during initial install
- iptables -A INPUT -p tcp --dport 22 -j DROP
packages:
- bc
- curl
- lsb-release
- ntp
- open-vm-tools
- openssh-server
- wget
ssh:
# For now we install openssh-server during package installs
allow-pw: true
install-server: falseIf I set install-server: true for SSH almost every time I had failures. Adding the iptables in early-commands ensures that SSH is blocked until the first reboot after installing the OS.
Not sure any of this helps others, but wanted to share.
NOTE: Reference Packer templates https://github.com/mrlesmithjr/packer-templates
@mrlesmithjr What vSphere/vCenter release are you running?
@stevenmiller 6.7 currently.
@mrlesmithjr thanks for the suggestions! Unfortunately, no luck with your settings either. I can see it enter the command line settings and succeed but it still takes me to the language selection screen:
Here's the json & user-data files I'm trying now: https://gitlab.com/darrelltang/k8s-homelab/-/tree/master/packer
@DarrellTang Well damn
I think that it takes you to the language screen if the cloud init cannot be read.
@jhawk28 Nailed it. The firewall on my laptop was still on preventing the user-data from being read! Sorry for the firedrill, and many thanks for the help!
@DarrellTang the new ubuntu installer is not very forgiving. If something doesn't go just right, it just "recovers" in a way that doesn't tell you what went wrong.
I'm good now as well, there must have been some issue with my user-data file as I recreated it from scratch and it is reading consistently now.
The limits/durations that worked with vsphere-iso
"boot_wait": "2s",
"ip_settle_timeout": "15m", # Set this to > max time for the VM after reboot to have the latest IP address natively
"ssh_timeout": "17m", # > ip_settle_timeout + buffer to have the SSH server up and ready
For the boot command:
"boot_command": [
"<esc><wait><esc><wait><esc><wait><enter>",
"/casper/vmlinuz root=/dev/sr0 initrd=/casper/initrd autoinstall ",
"ds=nocloud-net;s=http://<HTTP Server>:8090/<enter>"
]
For the HTTP server I used, python simple http server
python3.6 -m http.server 8090 --bind
You can also use the Packer built-in:
"http_directory": "{{user http-directory}}",
Same here...
Just getting stuck on ==> vsphere-iso: Waiting for SSH to become available...
I have issues with this as well. I've spent the last 3 days just trying to make a basic fully unattended Ubuntu Server VM, to no avail (I removed custom stuff to eliminate it as a source of the problem). This shouldn't be a difficult task,
Otherwise it just hangs with Nick's example here:
This is because you are missing the window to escape out of the live boot. Try changing the "boot_wait" to 2 seconds.
My VM ends up on the same screen as in this comment, after having managed to configure and run with the correct build commands.
It seems as if the SSH handshake is never made and thus the user-data is never read. If I drop down to the shell at the above screen and navigate to the cloud-config and user-data files on the guest, they are both empty, and the logs say they are invalid (because of course they are empty).
I've looked everywhere and there's plenty of reference to installing an SSH server but:
- The whole point is that Packer sets up its own SSH server
- I can't install an additional SSH server at the stage in the install process where it's required (i.e. to configure the server in the first place)
This whole process has been one issue after another and not smooth at all.
I'm running on Windows 10, using virtualbox-iso
Here are my files:
user-data
version: 1
identity:
hostname: ubuntu-server
username: ubuntu
password: ubuntu
ubuntu-2004.json
{
"builders": [
{
"boot_command": [
"<enter><wait><enter><wait2><enter><wait><f6><esc> ",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"/casper/vmlinuz ",
"initrd=/casper/initrd ",
"autoinstall<wait2> ds=nocloud-net; <wait2>",
"s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ubuntu/",
"<wait2><enter>"
],
"boot_wait": "3s",
"cpus": 2,
"guest_os_type": "Ubuntu_64",
"headless": false,
"http_directory": "http",
"iso_checksum": "sha256:443511f6bf12402c12503733059269a2e10dec602916c0a75263e5d990f6bb93",
"iso_url": "https://releases.ubuntu.com/20.04/ubuntu-20.04.1-live-server-amd64.iso",
"memory": 1024,
"name": "ubuntu-2004",
"ssh_timeout": "60m",
"ssh_handshake_attempts": "30",
"ssh_port": 22,
"ssh_password": "ubuntu",
"ssh_username": "ubuntu",
"type": "virtualbox-iso"
}
],
"provisioners": [
{
"inline": [
"ls /"
],
"type": "shell"
}
]
}
I've based all of this off of this example: https://nickcharlton.net/posts/automating-ubuntu-2004-installs-with-packer.html
I've made minimal modifications to reduce complexity and rule out certain configs causing the issue here, as well as modifying the boot command for my own system performance etc.
It seems as if the SSH handshake is never made and thus the user-data is never read. If I drop down to the shell at the above screen and navigate to the cloud-config and user-data files on the guest, they are both empty, and the logs say they are invalid (because of course they are empty).
You may want to increase ssh_handshake_attempts from 30 to a higher number. I have a working config and use 100 because anything less than 50 did not work.
This HCL2 config works (for me) for building a Vagrant box for Ubuntu 20.04 LTS via virtualbox-iso:
source "virtualbox-iso" "ubuntu2004" {
iso_url = "https://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso"
iso_checksum = "caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f"
guest_os_type = "Ubuntu_64"
boot_wait = "5s"
boot_command = [
"<enter><enter><f6><esc><wait> ",
"autoinstall ds=nocloud-net;seedfrom=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
"<enter><wait>",
]
memory = 4096
cpus = 4
disk_size = 102400
http_directory = "./http" # serves the user-data config
communicator = "ssh"
ssh_username = "vagrant"
ssh_password = "vagrant"
ssh_wait_timeout = "10000s"
ssh_handshake_attempts = "100"
virtualbox_version_file = ".vbox_version"
guest_additions_path = "VBoxGuestAdditions_{{.Version}}.iso"
shutdown_command = "echo 'vagrant' | sudo -S shutdown -P now"
}
I can share the user-data too if that would be helpful.
@tdyas I'm still getting the same problem even with 100+ handshakes and 10000s timeout. Your user-data maybe useful but I'm confused as to what is happening in the following screenshots.
After the boot command is entered, I get the following screen:
Which seems to be correctly setting up cloud-init from what I can see in the logs.
After this screen completes, I get the following screen:
I believe this is because the user-data didn't get read, but then I do not understand why there's cloud-init in the logs. At this stage of the install process my terminal still says Waiting for SSH to become available...
Any ideas what the problem may be?
That means your boot command was passed correctly, but the user-data was not able to be read. Your logs should tell what the http ip/port that is used for the embedded http server. In your VM, change to a different TTY (Ctrl+Shift+F2 I think) so that you can see if you can ping/wget/curl that IP address from your vm.
Thank you @jhawk28 I'm investigating now. I'm still confused on one point though, if it is indeed something to do with the ip and port. I've read a lot of guides and they mostly imply that I don't need to setup an SSH server on the host, but some suggest setting up a python simplehttpserver, for example. I thought the whole point of the http_directory setting in the packer config was to set this up automatically? In this case, the IP and Port are automatically supplied via {{ .HTTPIP }}:{{ .HTTPPort }} so it shouldn't be the root of the issue.
@DVAlexHiggs yes, packer should be providing the http ip/port and an embedded http server for the http_directory. Just because it is provided, it doesn't mean the ip address is accessible by the VM.
The other possibility is that your user-data has an error. You may be able to check the install logs for clues (use the other TTY).
Ok I seem to have found the cause of the issue. I've discovered that the ssh connection is successful, but authentication is failing because the ubuntu user doesn't seem to exist. Only a ubuntu-server user exists, and an installer user which seems to be created by cloud-init, and I can't seem to find a password for either. Essentially, I have no user to authenticate with or to provide the credentials for via ssh_username and ssh_password. Not sure where to go from here
The username/password configuration should all be in your user-data file. See https://ubuntu.com/server/docs/install/autoinstall-reference for the documentation.
But the user-data file is hosted over ssh in the first place, which I need to provide the details for in the packer config. Is that not circular? Sorry, maybe I am missing something.
The user-data provides the answers for the subiquity installer. This is equivalent to the preseed for debian, kickstart for redhat, and answers for windows. The user-data file needs to be provided to the installer by embedding it in the iso, loading a floppy with the correct cloud init tag, or over HTTP. You should be able to connect to your ubuntu vm with the username/password defined in your user-data once subiquity finishes the install.
I am providing the user-data over http. My understanding is that I need user-data to create users, but the guest can't access user-data because I have no users to authenticate over ssh with. Apologies for going off-topic, by the way, this issue is just giving me a really big headache. Is there somewhere more appropriate we can continue this?
I'll send you an email so its not so back and forth.
Ok I've got it working after some discussion with @jhawk28 (Thanks!)
Here's a gist with my working config, using Ubuntu Server 20.04.1: https://gist.github.com/DVAlexHiggs/03cdbef887736f03dcfe6d1749c18669
I had to use floppy disk configurations in the end, as I couldn't get the HTTP server to work (guest couldn't connect to host)
I've been playing around with this all weekend with Ubuntu Server 20.04.01 and the vsphere-iso builder, and can only get the build to complete successfully (ie: not go to the Welcome/Language screen) if I use the nocloud-net datasource and the seedfrom target of https://ubuntu-2004-cloud-init.s3.beryju.org/ from this blog post: https://beryju.org/blog/automating-ubuntu-server-20-04-with-packer. Working backward from a known good config, I've tried copying the contents of his hosted user-data file to my build directory (note: the hosted user-data file has been modified from what's listed in the blog post to include the early-command for disabling SSH) and using the standard seedfrom value of http://{{ .HTTPIP }}:{{ .HTTPPort }}/, as well as setting the http_port_min & http_port_max to 80 so that I could specify a seedfrom value of http://{{ .HTTPIP }}/ without the port specification, and I also setup a quick Apache web server with a self-signed certificate to try https://{{ .HTTPIP }}/. Connectivity has been confirmed as not an issue here. The only variation that I can think of that I haven't been able to duplicate yet is using a web server with a trusted certificate.
I've also tried a ton of variations with the nocloud datasource and floppy with both the cidata and CIDATA labels to no avail.
Very frustrating.
@tlindsay42 try using the nocloud-net with a floppy with the cidata label. I have gotten it working with vsphere-iso so post your template if you want me to look at it.
I saw DVAlexHiggs update before mine and tried that last night, but it didn't work, and using nocloud-net with a floppy doesn't make sense per the documentation. I'm going to try it again though momentarily.
While I'm doing that, I thought to post the exact same files in a GitLab Pages site after my previous comment, and confirmed that works, so I've been able to duplicate a successful build with the nocloud-net datasource and a seedfrom value targeting another website with a trusted certificate.
Here's that config:
{
"builders": [
{
"CPUs": 2,
"RAM": 2048,
"boot_command": [
"<esc><esc><esc>",
"<enter><wait>",
"/casper/vmlinuz ",
"root=/dev/sr0 ",
"initrd=/casper/initrd ",
"autoinstall ",
"ds=nocloud-net;s=https://tlindsay42.gitlab.io/packer-test/",
"<enter>"
],
"boot_wait": "2s",
"cluster": "{{ user `cluster` }}",
"convert_to_template": true,
"datacenter": "{{ user `datacenter` }}",
"datastore": "{{ user `datastore` }}",
"disk_controller_type": "pvscsi",
"folder": "{{user `folder`}}",
"guest_os_type": "ubuntu64Guest",
"host": "{{ user `host` }}",
"http_directory": "http",
"insecure_connection": "{{user `insecure_connection`}}",
"ip_settle_timeout": "5m",
"iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
"iso_url": "{{user `iso_url`}}",
"network_adapters": [
{
"network": "{{ user `network` }}",
"network_card": "vmxnet3"
}
],
"password": "{{ user `password` }}",
"ssh_handshake_attempts": "100",
"ssh_password": "vagrant",
"ssh_pty": true,
"ssh_timeout": "20m",
"ssh_username": "vagrant",
"storage": [
{
"disk_size": 8192,
"disk_thin_provisioned": false
}
],
"type": "vsphere-iso",
"username": "{{ user `username` }}",
"vcenter_server": "{{ user `vcenter_server` }}",
"vm_name": "{{ user `vm_name` }}-{{isotime}}",
"vm_version": "{{user `vm_version`}}"
}
],
"post-processors": []
}The odd nocloud-net datasource with floppy config appears to be working now. Will repeat a few times to confirm consistency.
We are using the s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ in our environment. The main issue is making sure that your vm can contact the IP+port of the machine running packer. If you have multiple NICs on your packer machine, there are some things that can be done to choose the correct IP.
Every build when using s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ winds up at the Welcome/Language screen for me (ie: fails), but I'd rather use the floppy anyway for now to eliminate the connectivity dependency.
To confirm though, I only have one vNIC on my packer instance and I verified connectivity back to the packer instance's IP & port specified in the build a few times by running curl to the user-data file from an existing VM in the same network segment. It's a lab environment with no microsegmentation or other firewall rules to complicate things.
In case it's helpful, I'm using packer v1.6.1.
$ packer --version
1.6.1The odd
nocloud-netdatasource with floppy config appears to be working now. Will repeat a few times to confirm consistency.
Also, this appears to be working consistently now, so I'm guessing that there was another issue in play when I tried it last night. Thanks @jhawk28!
Hi, I've been leaving this open for a while to give the community a place to figure out how to get this working, but I'm wondering if it makes sense to close now that it seems there's some consensus. Is there anything actionable that Packer can do to make this more straightforward for people?
Hi, I've been leaving this open for a while to give the community a place to figure out how to get this working, but I'm wondering if it makes sense to close now that it seems there's some consensus. Is there anything actionable that Packer can do to make this more straightforward for people?
Update the Unattended Installation for Debian guide?
I still have the same behaviour as @tlindsay42 prior to his switch to using a floppy. Unfortunately that’s not an option for me because the builder I am using (Proxmox) does not support it and throws an error, so I’m still stuck. I switched to using a local python HTTP server for the user-data endpoint and I see the GET request from the launched instance for the user-data file, but I still end up at the welcome language screen until SSH eventually times out. Any ideas about how I can debug further much appreciated ?
@goffinf post your config and I can see if anything jumps out. The welcome screen means that subiquity didn't like your file.
Thanks @jhawk28 ... it’s 1:20a.m here in the UK so I’ll do that in the morning (I’ll perhaps try a simplified user-data first since you seem pretty certain that’s where the problem is).
Fraser.
@jhawk28 ... So, on closer inspection, it turns out your earlier comment about ensuring that the HTTP server providing the user-data and meta-data endpoint is reachable from the launched VM turned out to be the problem. I thought this was working but clearly not. Using a local python3 server help me see that and also threw up something else that was unexpected. The reason for the first problem is because I am running Packer on WSL2 which is on a completely different subnet to the VM (172.x vs 192.x). So instead I just ran the python server on the host and all was good (ish).
The unexpected issue was the way in which the HTTP server URL was formed. The first thing I noticed was this 404:
::ffff:192.168.0.168 - - [31/Aug/2020 20:10:37] code 404, message File not found
::ffff:192.168.0.168 - - [31/Aug/2020 20:10:37] "GET /---meta-data HTTP/1.1" 404 -
There appeared to be two problems here. The first was those 3 dashes ! I noted I had changed the boot_command that I was using to one from a blog post that I was reading ... spot the 3 dashes
"boot_command": [
"<esc><wait><esc><wait><f6><esc><wait>",
"<bs><bs><bs><bs><bs>",
"autoinstall ip=dhcp ds=nocloud-net;s=http://<host-ip>:<host-port>/ ",
"--- <enter>"
],
So I switched back to my earlier one which didn't include those:
"boot_command": [
"<enter><enter><f6><esc><wait>",
"autoinstall ds=nocloud-net;s=http://<host-ip>:<host-port>/ ",
"<wait><enter>"
],
That improved things but I was still getting a 404 for meta-data which halts the progress of the build:
::ffff:192.168.0.169 - - [31/Aug/2020 20:17:18] code 404, message File not found
::ffff:192.168.0.169 - - [31/Aug/2020 20:17:18] "GET /meta-data HTTP/1.1" 404 -
I had set the http_directory property to "http" but that seems to have been ignored.
So for now I updated the url to include it and that worked ok. I'm not sure if that is because I am using my own HTTP server rather than the one Packer creates ??
"boot_command": [
"<enter><enter><f6><esc><wait>",
"autoinstall ds=nocloud-net;s=http://<host-ip>:<host-port>/http/ ",
"<wait><enter>"
],
...
::ffff:192.168.0.171 - - [31/Aug/2020 20:24:40] "GET /http/meta-data HTTP/1.1" 200 -
::ffff:192.168.0.171 - - [31/Aug/2020 20:24:40] "GET /http/user-data HTTP/1.1" 200 -
Anyway, that allowed cloud-init to run successfully to completion, although only after I also removed install for the qemu-guest-agent which failed (other packages installed fine). It looks like I still have a problem with DNS resolution since I couldn't install it manually either once the VM rebooted and I was able to SSH to it.
Interestingly the Packer build remained resolutely at 'Waiting for SSH to become available...' and eventually timed out and destroyed the VM. I am using the same username/password that I create in cloud-init in the ssh_username and ssh_password properties, and I can successfully log into the VM using those, so I'm not sure why Packer can't connect. The packer log suggests its because the qemu-guest-agent isn't available, so I'll try and sort that out next ....
packer.log:
2020/08/31 23:01:37 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/08/31 23:01:42 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/08/31 23:01:47 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/08/31 23:01:52 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
...
packer build host_python.json
packer build host_python.json
ubuntu-20-04-base: output will be in this color.
==> ubuntu-20-04-base: Creating VM
==> ubuntu-20-04-base: No VM ID given, getting next free from Proxmox
==> ubuntu-20-04-base: Starting VM
==> ubuntu-20-04-base: Starting HTTP server on port 8437
==> ubuntu-20-04-base: Waiting 3s for boot
==> ubuntu-20-04-base: Typing the boot command
==> ubuntu-20-04-base: Waiting for SSH to become available...
==> ubuntu-20-04-base: Timeout waiting for SSH.
==> ubuntu-20-04-base: Stopping VM
==> ubuntu-20-04-base: Deleting VM
Build 'ubuntu-20-04-base' errored: Timeout waiting for SSH.
==> Some builds didn't complete successfully and had errors:
--> ubuntu-20-04-base: Timeout waiting for SSH.
==> Builds finished but no artifacts were created.
Anyway, the progress is that I no longer get left at the Welcome screen and cloud-init does run per the docs.
Thanks so much for your support.
Fraser.
I think you need to install the qemu-guest-agent package. That is how packer figures out what the IP address of the VM is in order to connect over SSH. We install open-vm-tools for VMWare.
Yes indeed. I did have it on the package list but because of the DNS problem it failed. I need to get to the bottom of that first ... so I’ll be digging around in systemd-resolv tomorrow no doubt.
don't forget to run packer with -on-error=ask: https://www.packer.io/docs/commands/build#on-error-cleanup
Ugh, the same packer build that managed to get thru to load and run cloud-init has returned to dropping me at the Welcome screen. Didn’t change anything apart from adding the qemu-guest-agent in the list of packages to be installed .. different day different ****. I have worked with Packer for a decade and never had this much trouble with it, even doing Windows AMI builds and having to rely on WinRM.
We are using the
s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/in our environment. The main issue is making sure that your vm can contact the IP+port of the machine running packer. If you have multiple NICs on your packer machine, there are some things that can be done to choose the correct IP.
Oh god damn. I didn't know packer builder machine must be accessible to template VM. I was running packer build from my laptop over public network.. 😫
edit: nah I cannot get it working. Anyone got working example for Ubuntu 20.04 on Proxmox builder ? HELP
Can't past this screen.
Using "boot_wait": "5s", stuck on this.
Using "boot_wait": "2s", stuck on this.
Yeah that caught me out too (accessibility to the Packer build machine that is). It doesn’t have to be the machine running Packer since you can provide any HTTP address and port where you want to serve the user-data and meta-data files. Because I am running Packer on WSL2 and that is a different subnet to the Proxmox template Build VM, that failed (although it’s not that obvious that’s the reason), so I just ran a python simple http server on the same laptop but not in WSL2 and put that IP address and port in the boot_command, and all was good.
python3 -m http.server 8081 (I think ... I don’t have my laptop on right now)
Similarly to you my Packer build alternates between the Welcome page and just getting stuck waiting for an SSH connection. Varying the target Proxmox node and the boot wait time and some as yet undiscovered factor takes me down one path or another but I can’t seem to get consistent behaviour. When I do get cloud-init to run, I note that the install of the Qemu guest agent fails because external connectivity is not established for some reason (this doesn’t happen when I create Proxmox VMs in any other way, they just work). Since the Qemu guest agent isn’t available, Packer is never notified of the template VM’s IP address, hence the SSH connection timeout.
I am surprised your second screen shot didn’t show your boot_command, unless it disappeared before you managed to hit the button ? .... can you show what you are using ?
Because this problem was burining so much time, I have moved on to provisioning a Proxmox template VM by creating a linked clone of a base OS template that I created from the latest Ubuntu live ISO in the Proxmox UI (which I installed the Qemu guest agent into and an SSH public key). I launch that using the Proxmox Terraform provider then run an Ansible playbook to install and configure all of the software I want to bake into the template. Then, as part of the same CI pipeline, save that as a template using Proxmox CLI commands. It achieves the same automated and repeatable result, but it still niggles me that I can’t get Packer to work, so, I will probably return to it when I have a bit more time and patience.
HTHs
Fraser.
That's unfortunate to hear. I thought this ubuntu 20 new changes is supposed to solve complex preseed config and improve the workflow of tools like Packer. But it's not helping. I don't know how is this going to be helpful besides packer.
About boot_command I'm seeing some commands being typed slowly on black screen at the beginning, or under second picture sometimes. Nevertheless I still go into Language selection menu eventually. (Even though it looked like boot_command is working)
Yesterday I've tried many many tutorials on the internet, but none of them worked. Today I'm going to try ubuntu 20 legacy version.. Hope that'll work.
The reason that it goes to the welcome screen is because the subiquity installer had an issue.
- It could be that an inability to connect to retrieve the cloud-init file. You can try using a floppy with the label cidata, debug why the vm cannot connect to the packer http server, or serve the user-data/meta-data files in a different way
- It could be that the cloud-init file is invalid. Check things like the line endings and use tabs instead of spaces (its a yaml file). Use a minimal known config that has worked for someone else and add things you need from the documentation: https://ubuntu.com/server/docs/install/autoinstall-quickstart
- there may be an issue with the boot command. vsphere needs the
root=/dev/sr0in order to work. proxmox may need something else
Since you see it typing, that is good (means it caught the grub timing)
NOTE: the ubuntu auto install uses kvm as an example so it should be possible with proxmox
kvm -no-reboot -m 1024 \ -drive file=image.img,format=raw,cache=none,if=virtio \ -cdrom ~/Downloads/ubuntu-20.04-live-server-amd64.iso \ -kernel /mnt/casper/vmlinuz \ -initrd /mnt/casper/initrd \ -append 'autoinstall ds=nocloud-net;s=http://_gateway:3003/'
So ... I can now get a consistent successful build of a Proxmox template. The problem I now have is that cloning from that template there is no network. That will be tomorrow's issue to resolve.
@shinebayar-g the config that I am now using is below:
There was one gotcha.
- Originally I had this network section in my user-data. However, with it present, when the template VM reboots Packer is not able to establish the SSH connection even though the Qemu guest agent is running and returning an IP:
network:
network:
version: 2
ethernets:
eth0:
dhcp4: true
dhcp-identifier: mac
I'm not 100% sure why that is but it may be that Qemu is returning the IP prior to reboot and the rebooted VM gets a new one. Not really sure, but removing the section allows Packer to connect. This may be the cause of my network problem when I clone from the template, so I might return to it again tomorrow... ?
packer.log (install - reboot - SSH connect)
...
2020/09/08 13:23:05 Build debug mode: false
2020/09/08 13:23:05 Force build: false
2020/09/08 13:23:05 On error: ask
2020/09/08 13:23:05 Waiting on builds to complete...
2020/09/08 13:23:05 Starting build run: ubuntu-20-04-base
2020/09/08 13:23:05 Running builder: proxmox
2020/09/08 13:23:05 [INFO] (telemetry) Starting builder proxmox
2020/09/08 13:23:05 packer-builder-proxmox plugin: No URLs were provided to Step Download. Continuing...
2020/09/08 13:23:05 ui: ^[[1;32m==> ubuntu-20-04-base: Creating VM^[[0m
2020/09/08 13:23:05 ui: ^[[1;32m==> ubuntu-20-04-base: No VM ID given, getting next free from Proxmox^[[0m
2020/09/08 13:23:07 ui: ^[[1;32m==> ubuntu-20-04-base: Starting VM^[[0m
2020/09/08 13:23:09 packer-builder-proxmox plugin: Found available port: 8332 on IP: 0.0.0.0
2020/09/08 13:23:09 ui: ^[[1;32m==> ubuntu-20-04-base: Starting HTTP server on port 8332^[[0m
2020/09/08 13:23:09 ui: ^[[1;32m==> ubuntu-20-04-base: Waiting 3s for boot^[[0m
2020/09/08 13:23:12 ui: ^[[1;32m==> ubuntu-20-04-base: Typing the boot command^[[0m
2020/09/08 13:23:12 packer-builder-proxmox plugin: [INFO] Waiting 1s
2020/09/08 13:23:16 packer-builder-proxmox plugin: [INFO] Waiting 1s
2020/09/08 13:23:20 packer-builder-proxmox plugin: [DEBUG] Unable to get address during connection step: 500 QEMU guest agent is not running
2020/09/08 13:23:20 packer-builder-proxmox plugin: [INFO] Waiting for SSH, up to timeout: 20m0s
2020/09/08 13:23:20 ui: ^[[1;32m==> ubuntu-20-04-base: Waiting for SSH to become available...^[[0m
...
2020/09/08 13:24:29 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/09/08 13:24:37 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/09/08 13:24:45 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/09/08 13:24:53 packer-builder-proxmox plugin: [DEBUG] Error getting SSH address: 500 QEMU guest agent is not running
2020/09/08 13:24:59 packer-builder-proxmox plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp 192.168.0.208:22: connect: connection refused
2020/09/08 13:25:04 packer-builder-proxmox plugin: [DEBUG] TCP connection to SSH ip/port failed: dial tcp 192.168.0.208:22: connect: connection refused
2020/09/08 13:25:09 packer-builder-proxmox plugin: [INFO] Attempting SSH connection to 192.168.0.208:22...
2020/09/08 13:25:09 packer-builder-proxmox plugin: [DEBUG] reconnecting to TCP connection for SSH
2020/09/08 13:25:09 packer-builder-proxmox plugin: [DEBUG] handshaking with SSH
2020/09/08 13:25:09 packer-builder-proxmox plugin: [DEBUG] handshake complete!
2020/09/08 13:25:09 packer-builder-proxmox plugin: [INFO] no local agent socket, will not connect agent
2020/09/08 13:25:09 ui: ==> ubuntu-20-04-base: Connected to SSH!
2020/09/08 13:25:09 packer-builder-proxmox plugin: Running the provision hook
2020/09/08 13:25:09 [INFO] (telemetry) Starting provisioner shell
...
host.json
{
"builders": [
{
"boot_command": [
"<enter><enter><f6><esc><wait>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
"<bs><bs><bs>",
"/casper/vmlinuz ",
"root=/dev/sr0 ",
"initrd=/casper/initrd ",
"autoinstall net.ifnames=0 biosdevname=0 ip=dhcp ipv6.disable=1 ds=nocloud-net;s=http://192.168.0.29:8080/http/",
"<wait><enter>"
],
"boot_wait": "{{user `boot_wait`}}",
"cloud_init": true,
"cloud_init_storage_pool": "local-lvm",
"communicator": "ssh",
"cores": 1,
"cpu_type": "host",
"disks": [
{
"disk_size": "{{user `home_volume_size`}}",
"storage_pool": "local-lvm",
"storage_pool_type": "lvm-thin",
"type": "scsi",
"format": "raw"
}
],
"http_directory": "http",
"insecure_skip_tls_verify": true,
"iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
"iso_file": "{{user `iso_file`}}",
"memory": 2048,
"name": "ubuntu-20-04-base",
"network_adapters": [
{
"bridge": "vmbr0",
"model": "virtio"
}
],
"node": "{{user `proxmox_target_node`}}",
"os": "l26",
"password": "{{user `proxmox_server_pwd`}}",
"proxmox_url": "https://{{user `proxmox_server_hostname`}}:{{user `proxmox_server_port`}}/api2/json",
"qemu_agent": true,
"scsi_controller": "virtio-scsi-pci",
"sockets": 1,
"ssh_handshake_attempts": "50",
"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",
"ssh_timeout": "{{user `ssh_timeout`}}",
"type": "proxmox",
"template_name": "packer-test-dhcp",
"template_description": "A packer template build test",
"unmount_iso": true,
"username": "{{user `proxmox_server_user`}}"
}
],
"provisioners": [
{
"execute_command": "{{ .Vars }} sudo -E -S sh '{{ .Path }}'",
"inline": [
"while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
],
"type": "shell"
}
],
"variables": {
"boot_wait": "3s",
"home_volume_size": "20G",
"http_directory": "http",
"iso_checksum": "443511f6bf12402c12503733059269a2e10dec602916c0a75263e5d990f6bb93",
"iso_checksum_type": "sha256",
"iso_file": "local:iso/ubuntu-20.04.1-live-server-amd64.iso",
"proxmox_server_hostname": "proxmox-002",
"proxmox_server_port": "8006",
"proxmox_server_pwd": "REPLACE-WITH-YOUR-PROXMOX-USER",
"proxmox_server_user": "REPLACE-WITH-YOUR-PROXMOX-USER",
"proxmox_target_node": "home",
"ssh_handshake_attempts": "100",
"ssh_password": "REPLACE-WITH-YOUR-SSH-USER",
"ssh_username": "REPLACE-WITH-YOUR-SSH-USER-PWD",
"ssh_timeout": "20m"
}
}
user-data
#cloud-config
autoinstall:
version: 1
keyboard:
layout: en
variant: uk
locale: en_GB
late-commands:
- 'sed -i "s/^#*\(send dhcp-client-identifier\).*$/\1 = hardware;/" /target/etc/dhcp/dhclient.conf'
- 'sed -i "s/dhcp4: true/&\n dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
packages:
- bc
- curl
- wget
- openssh-server
- qemu-guest-agent
ssh:
allow-pw: true
authorized-keys:
- "ssh-rsa REPLACE-WITH-YOUR-SSH-PUBLIC-KEY"
install-server: true
storage:
layout:
name: direct
user-data:
disable_root: false
timezone: Europe/London
users:
- name: REPLACE-WITH-THE-USER-YOU-WANT-TO-CREATE
passwd: REPLACE-WITH-THE-HASHED-PWD
groups: [adm, cdrom, dip, plugdev, lxd, sudo]
lock-passwd: false
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized-keys:
- "ssh-rsa REPLACE-WITH-YOUR-SSH-PUBLIC-KEY"
write_files:
- path: /usr/local/bin/hello-world.sh
permissions: "0755"
content: |
#!/bin/bash
FORENAME=${1:-goffinf};
echo "Hello $FORENAME" >> /usr/local/bin/greeting;
runcmd:
- /usr/local/bin/hello-world.sh 'World'
UEFI works for me if i use quotes at ds , took me a while to find, but with these bootcmmands my efi works on vmware with packer vsphere-iso
"boot_command": [
"<esc><wait>",
"<esc><wait>",
"linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://{{.HTTPIP}}:{{.HTTPPort}}/\"<enter><wait>",
"initrd /casper/initrd<enter><wait>",
"boot<enter>"
],
In case anyone else is in my situation (hours of searching, none of the well-documented examples work), this thread is what cracked the problem for me. Specifically this comment from JulyIghor .
When you drop down to the grub> command prompt, quotes are required for the ds parameter. My boot command, therefore:
"boot_command": [
"<esc><esc><esc>",
"set gfxpayload=keep<enter>",
"linux /casper/vmlinuz ",
"\"ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/\" ",
"quiet autoinstall ---<enter>",
"initrd /casper/initrd<enter>",
"boot<enter>"
],
This is when autoinstalling Ubuntu 20.04.1 using the Live ISO. I used a boot_wait of 3s. Provided you use a few commands initially, you should drop to the grub> prompt. 🙂 <esc> is safer than <enter> since it will never trigger the normal installation process.
All the references to <f6> I've seen everywhere have been particularly foxing. F6 doesn't do anything at any part of the installation as far as I can see. Not sure why it is present in others' boot commands. Maybe this has some function when using a different installation ISO?
My installation environment is Hyper-V on Windows 10, building a generation 2 (UEFI) image.
I suspect my boot command can be simplified. (Is gfxpayload really needed, for example?) I was just delighted to get it working at all.
Thank you @JulyIghor !