Deployment issue of 20/11/2018
samoht opened this issue · comments
https://twitter.com/h4nnes/status/1064776312970256384
@hannesm mirage.io is still down :/
@avsm hm, (Failure "device 51728 not found (available = [ 51808 ])") is some change in the recent release. xenstore ordering seems different
@hannesm this wasn't changed in 3.2.0 nor 3.3.0, maybe @eriangazag knows when (iirc he did recent re-deployments) -- sounds to me that passing block devices changed its semantics (but i thought that chamge was backed off)
The last known working version of the website was apparently built with a pinned version of mirage to 3.0.7 (???). This is possible that we were carrying a patch for the block device renumbering for a very long time...
A proper fix should probably be to change the name of the disk used to deploy https://mirage.io here: https://github.com/mirage/mirage-www-deployment/blob/master/scripts/deploy-tls.sh#L40
/cc @djs55 @jonludlam Any opinion on the right fix to make the device name match the reality?
I guess the core problem is that the kernel is really a function
val kernel: disk_path -> disk_path -> result
and we need to find a nice way to supply the disk_path arguments at deployment-time, since the deployment system will be different to the build one.
The .xl.in file + kernel binary is a bit of a hacky way to represent the function, since we have to apply the arguments using sed. It would work if the xl.in file and the kernel had matching bus/slot numbers but it's still ugly.
Perhaps another option is to generate a kernel + a metadata file which lists the required arguments and then write a tool (mirage run mirage-www-deploy?) which knows how to apply the arguments and hides the internal details.
Another option would be to use some other way to find the right disk, instead of the bus/slot number. When we discussed this here https://lists.xenproject.org/archives/html/mirageos-devel/2015-01/msg00100.html I wondered whether we could add an id to every disk (in an MBR?) and then use this at runtime inside the kernel to find the correct disk. It wouldn't matter what device name /slot/number they had.
Closing as https://mirage.io is back online :-) Many thanks to everyone involved!
yes, thank you to everyone involved! I'm glad we're now using a current version of mirage and can redeploy our website
@djs55 did you commit your changes to https://github.com/mirage/mirage-www-deployment or did you just update the configuration on the server?
Can you push that change upstream so that we won't have the same issue next time? :-) Thanks!
I've pushed my change. Note however the git diff on the deployment machine is large so it looks like there are more unchecked-in changes:
diff --git a/scripts/create-vm.sh b/scripts/create-vm.sh
index 566b864..57947f9 100755
--- a/scripts/create-vm.sh
+++ b/scripts/create-vm.sh
@@ -26,7 +26,7 @@ NAME=$1
ROOT=$(git rev-parse --show-toplevel)
KERNEL=$ROOT/xen/$(cat "$ROOT/xen/latest")
-VM="mir-$NAME"
+VM="$NAME"
cd "$KERNEL"
diff --git a/scripts/post-merge.hook b/scripts/post-merge.hook
index fec56b5..a8455e6 100755
--- a/scripts/post-merge.hook
+++ b/scripts/post-merge.hook
@@ -34,14 +34,14 @@
cd $(dirname "$(readlink -f "$0")") # canonicalise directory
unset GIT_DIR
-HTTP=openmirage.org
-HTTPS=mirage.io
+HTTP=openmirageorg
+HTTPS=mirageio
ROOT=$(git rev-parse --show-toplevel)
KERNEL=$ROOT/xen/$(cat "$ROOT/xen/latest")
-if [ -f "$KERNEL/mir-${HTTP}.xen.bz2" -a \
- -f "$KERNEL/mir-${HTTPS}.xen.bz2" ];
+if [ -f "$KERNEL/${HTTP}.xen.bz2" -a \
+ -f "$KERNEL/${HTTPS}.xen.bz2" ];
then
"$ROOT/scripts/deploy.sh" $HTTP
"$ROOT/scripts/deploy-tls.sh" $HTTPS "$ROOT/tls"
diff --git a/scripts/prepare-config.sh b/scripts/prepare-config.sh
index 0a2b956..2700831 100755
--- a/scripts/prepare-config.sh
+++ b/scripts/prepare-config.sh
@@ -31,6 +31,6 @@ KERNEL=$ROOT/xen/$(cat "$ROOT/xen/latest")
cd "$ROOT"
-sed -e "s,@NAME@,$NAME,g; s,@KERNEL@,$KERNEL/mir-$NAME.xen,g; s:@DISK@:$DISK:g" \
+sed -e "s,@NAME@,$NAME,g; s,@KERNEL@,$KERNEL/$NAME.xen,g; s:@DISK@:$DISK:g" \
< xl.conf.in \
>| "$KERNEL/$NAME.xl"
deleted file mode 100644
index aed972a..0000000
--- a/xen/1ea46520ce5c5102e1c52bf010b4ad0ac77d7dab/mirage.io.xl.in
+++ /dev/null
@@ -1,14 +0,0 @@
-# Generated by mirage.io (Tue, 3 May 2016 21:29:41 GMT).
-
-name = '@NAME@'
-kernel = '@KERNEL@'
-builder = 'linux'
-memory = @MEMORY@
-on_crash = 'preserve'
-
-disk = [ 'format=raw, vdev=xvdb, access=rw, target=@BLOCK:tar_block1.img@', 'format=raw, vdev=xvdd, access=rw, target=@BLOCK:tar_block2.img@', 'format=raw, vdev=xvde, access=rw, target=@BLOCK:fat_block2.img@', 'format=raw, vdev=xvdf, access=rw, target=@BLOCK:tar_block3.img@', 'format=raw, vdev=xvdc, access=rw, target=@BLOCK:fat_block1.img@', 'format=raw, vdev=xvdg, access=rw, target=@BLOCK:fat_block3.img@' ]
-
-# if your system uses openvswitch then either edit /etc/xen/xl.conf and set
-# vif.default.script="vif-openvswitch"
-# or add "script=vif-openvswitch," before the "bridge=" below:
-vif = [ 'bridge=@NETWORK:tap0@' ]
diff --git a/xen/1ea46520ce5c5102e1c52bf010b4ad0ac77d7dab/openmirage.org.xl.in b/xen/1ea46520ce5c5102e1c52bf010b4ad0ac77d7dab/openmirage.org.xl.in
deleted file mode 100644
index 9b4e722..0000000
--- a/xen/1ea46520ce5c5102e1c52bf010b4ad0ac77d7dab/openmirage.org.xl.in
+++ /dev/null
@@ -1,14 +0,0 @@
-# Generated by openmirage.org (Tue, 3 May 2016 21:26:00 GMT).
-
-name = '@NAME@'
-kernel = '@KERNEL@'
-builder = 'linux'
-memory = @MEMORY@
-on_crash = 'preserve'
-
-disk = [ 'format=raw, vdev=xvdb, access=rw, target=@BLOCK:tar_block1.img@', 'format=raw, vdev=xvdd, access=rw, target=@BLOCK:tar_block2.img@', 'format=raw, vdev=xvde, access=rw, target=@BLOCK:fat_block2.img@', 'format=raw, vdev=xvdf, access=rw, target=@BLOCK:tar_block3.img@', 'format=raw, vdev=xvdc, access=rw, target=@BLOCK:fat_block1.img@', 'format=raw, vdev=xvdg, access=rw, target=@BLOCK:fat_block3.img@' ]
-
-# if your system uses openvswitch then either edit /etc/xen/xl.conf and set
-# vif.default.script="vif-openvswitch"
-# or add "script=vif-openvswitch," before the "bridge=" below:
-vif = [ 'bridge=@NETWORK:tap0@' ]
diff --git a/xl.conf.in b/xl.conf.in
index 863fb59..28838da 100644
--- a/xl.conf.in
+++ b/xl.conf.in
@@ -1,5 +1,5 @@
name='@NAME@'
-memory='1024'
+memory='512'
kernel='@KERNEL@'
vif=['bridge=xenbr0']
on_crash='restart'