dgiagio / warp

Do you think this would work to package Python applications?

Using conda-pack I have a standalone relocatable self-contained directory (containing Python core, all 3rd party dependencies and the application) and a launch script, say, on Linux. It just needs to be packed/unpacked, so it seems warp might work for this use case also.

I've just looked at conda-pack and it's an interesting project. I really expect it to work flawlessly with warp. Could you give a try?

I would love to have some feedback on that so I can add instructions for Python as well.

@dgiagio perfect! I'll give it a shot soon, and report back.

@dgiagio ok, so I finally got to it. I can compile the Node JS application from the README, creating myapp.bin, thus showing that warp itself works on my machine (Ubuntu 18.04). But when I pack the directory with the Python tree, I get:

$ ./warp-packer --arch linux-x64 --input_dir myapp --exec launch --output myapp.bin
Compressing input directory "myapp"...
Error: Os { code: 21, kind: Other, message: "Is a directory" }

Any idea what might be causing it?

I used strace, and it's caused by a symbolic link to a directory:

$ ll lib/icu/
total 48
drwxr-xr-x  3 certik certik  4096 Oct 23 15:43 ./
drwxr-xr-x 21 certik certik 36864 Oct 23 15:43 ../
drwxr-xr-x  2 certik certik  4096 Oct 23 15:43 58.2/
lrwxrwxrwx  1 certik certik     4 Oct 23 14:48 current -> 58.2/
lrwxrwxrwx  1 certik certik    17 Oct 23 14:48 Makefile.inc -> 58.2/Makefile.inc
lrwxrwxrwx  1 certik certik    16 Oct 23 14:48 pkgdata.inc -> 58.2/pkgdata.inc

The current is a link to the 58.2 directory, but strace says that's where it fails:

openat(AT_FDCWD, "XXX/lib/icu/current", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
read(4, 0x7fff247f51b8, 8192)           = -1 EISDIR (Is a directory)

I am guessing it's failing at this line:

warp/warp-packer/src/main.rs

Line 94 in 4cf163f

tar.append_dir_all(".", dir)?;

, but I have no idea how to fix it.

This been fixed by #11 but I haven't made a release out of that yet.

Alternatively you can checkout the latest sources and cargo build --release. Then copy the target/release/warp-runner to your $HOME/.local/share/warp/runners/<version>/linux-x64 and use the target/release/warp-packer as you did before.

@dgiagio thanks. In the meantime I fixed it by removing the offending symlinks:

rm icu/current
rm icu/Makefile.inc
rm icu/pkgdata.inc
rm lib/terminfo

Now it works.

The binary seems to be working.

I found another unrelated problem. I launch a jupyter notebook, and it gets stopped by Ctrl-C, and one then has to press "y". Hower, the Ctrl-C kills the app itself, and so there is no way to press "y" and the notebook resumes, now detached from the terminal:

[I 16:38:11.115 NotebookApp] Adapting to protocol v5.1 for kernel e1f61094-c0f6-476c-88c0-4ee6968048fa
^C
[I 16:38:13.967 NotebookApp] interrupted
Serving notebooks from local directory: /home/certik/.local/share/warp/packages/myapp.bin/myapp/share/myapp/nb
1 active kernel
The Jupyter Notebook is running at:
http://localhost:8889/?token=f00516ace91803a460aa4d25abcd8b5305d7d8a46308d665
Shutdown this notebook server (y/[n])? certik@redhawk:~/repos/myapp$ No answer for 5s: resuming operation...

As you can see, the rust app finished, but the notebook is still running in the background.

What would be the best way to fix this?

Could you share the directory or explain how you created your app? I would like to reproduce.

Thanks.

Sure. Create a myapp directory and put the following launch script in it (make it executable):

#!/bin/bash

set -e

DIR="$(cd "$(dirname "$0")" ; pwd -P)"
ROOT=$DIR/myapp_dist
. $ROOT/bin/activate
conda-unpack
jupyter notebook

Then execute the following build.sh script that downloads Conda, creates a sample environment with jupyter, uses conda-pack to create a POSIX tree, then packs it using warp:

#!/bin/bash

set -e
set -x

C=`pwd`

wget --no-check-certificate https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p $C/conda_root
export PATH="$C/conda_root/bin:$PATH"
conda config --set always_yes yes --set changeps1 no

conda install -c conda-forge conda-pack
conda create -n myapp jupyter
conda pack -n myapp -o myapp_dist.tar.gz

mkdir myapp/myapp_dist
tar xzf myapp_dist.tar.gz -C myapp/myapp_dist

curl -Lo warp-packer https://github.com/dgiagio/warp/releases/download/v0.2.1/linux-x64.warp-packer
chmod +x warp-packer

./warp-packer --arch linux-x64 --input_dir myapp --exec launch --output myapp.bin
chmod +x myapp.bin

Then launch the myapp.bin, it will run a jupyter notebook. Then press Ctrl-C and you will reproduce the behavior above.

P.S. If you run jupyter notebook from a terminal, then you stop it by pressing Ctrl-C twice (that's how I do it). If you only press it once, it will ask you if you want to quit.

Could you please try adding exec before jupyter notebook in your launch script?

E.g.

exec jupyter notebook

Ok, so that helped a lot, now the notebook gets killed:

[I 17:08:21.628 NotebookApp] Adapting to protocol v5.1 for kernel 9b3db73d-feba-4289-91b0-bed187ca7b67
^C
[I 17:16:06.713 NotebookApp] interrupted
Serving notebooks from local directory: /home/certik/.local/share/warp/packages/myapp.bin/myapp_dist/share/myapp/nb
1 active kernel
The Jupyter Notebook is running at:
http://localhost:8890/?token=6c05e1ebeadc0e54efa84325a640a5a82e6d36a1713ba89e
Shutdown this notebook server (y/[n])? certik@redhawk:~/repos/myapp-warp(fix)$ 
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/certik/.local/share/warp/packages/myapp.bin/myapp_dist/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/certik/.local/share/warp/packages/myapp.bin/myapp_dist/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/certik/.local/share/warp/packages/myapp.bin/myapp_dist/lib/python3.6/site-packages/notebook/notebookapp.py", line 1507, in _confirm_exit
    line = sys.stdin.readline()
OSError: [Errno 5] Input/output error

Still not ideal, as users will be confused why it's giving some errors, but at least it gets killed when the rust app exits.

Thanks for the feedback. I will reproduce and hopefully come up with a fix soon.

@dgiagio figured it out. Instead of exec jupyter notebook, one has to use exec jupyter notebook < /dev/null. This causes sys.stdin.isatty() in Python to return False, which in turn makes jupyter not to install the extra confirmation handler per this line:

https://github.com/jupyter/notebook/blob/d60445e92666a0c5651a33e180c92051d94a7d44/notebook/notebookapp.py#L1474

and then it simply terminates on a single press of Ctrl-C. This then plays well with warp and things terminate properly on a single Ctrl-C press.

So I think everything is fixed at this point and things work with Python.

Would this work with Python?