HaskellR dependency question (nix-related)
skeydan opened this issue · comments
Hi,
probably this goes too far, sorry, but I'm asking just in case you'd see a way how to do this.
I'm using nix on Fedora, and I'm trying to use either kerasR (from CRAN) or keras (from github) with HaskellR.
In any case, I need keras (R) to find the TensorFlow installation, which in my native installation is a conda environment.
So in my native R, I'd do
reticulate::use_condaenv("tf3.5", required = TRUE)
I guess there is no way using nix to use conda environments, right? Also a native TensorFlow installation would not work I guess... would it?
I'm asking just in case you have an idea how this could work, otherwise I'll think of other interesting applications :-)
Many thanks in advance
Sigrid
Hi! It could possibly work! I know nothing about conda/tf/keras but there's no fundamental reason why it should not. I'll look into that and see if I can't help to get something going.
A question: is there a reason why you want to use your native installation? Usually nix handles everything that it needs which would include using own TensorFlow &c.
Hi! That would be totally great! No, if everything could run in nix that would be perfect (and cool!) - there's no reason to use the native installation then.
If you have a look at
... after installing keras
devtools::install_github("rstudio/keras")
they do
library(keras)
install_tensorflow()
"this will provide you with a default installation of TensorFlow suitable for use with the keras R package. "
... so I don't know how this could work with nix?
The other way would be kerasR, which installs from CRAN, but that normally needs a pre-existing installation of tensorflow. When I use that in R, I do reticulate::use_condaenv to point it to the correct python env.
I don't know if any of these - keras from rstudio or kerasR - look possible to you?
install_tensorflow() can be patched out if needed :)
kerasR sounds like a better way forward I think. We can just add tensor flow as dependency of that package and it might just work. I'll play around in the evening.
Great! Thank you!
kerasR sounds like a better way forward I think.
I don't know which are the most important aspects here, but
- keras has all features (from the Python version), while kerasR doesn't have everything implemented (e.g., stateful LSTM, TimeDistributed layer...)
- also, rstudio's behind keras, which probably means more manpower for future maintenance (Keras releases...
But of course it's up to you :-)
Aha. I didn't know the difference (I'm just an oblivious third party). I'll look into keras-by-rstudio then.
cool, thanks!
Hi,
Please look at the keras-thing branch, notably at
Basically what happens there is that I'm packaging keras from GitHub. As that package requires newer versions of tensorflow and reticulate packages, I include those. Then I add all of them to rPackages and I depend on rPackages.keras.
I do not know how to test it, if you give it a go and it doesn't work for you, let me know how to try it out. I was only able to verify that tensorflow works through R:
> library(tensorflow)
> sess = tf$Session()
2017-06-07 19:36:28.158006: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-07 19:36:28.158031: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-07 19:36:28.158039: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-07 19:36:28.158044: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-07 19:36:28.158050: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
> hello <- tf$constant('Hello, TensorFlow!')
> sess$run(hello)
b'Hello, TensorFlow!'
>
Usually such overrides of packages would go elsewhere (such as in ~/.nixpkgs/config.nix) through a configuration option but I put them directly in shell.nix as that's where you've been putting your thing so far, at least as far as I understand.
I hope this moves you forward. I have not investigated an "automated" way to achieve installing from GitHub: it would take significantly more effort to make work in general. However if you have many, many packages that aren't on CRAN/MRAN, maybe something can be done. For CRAN packages it's simple as there already exist scripts to generate rPackages from those.
Hi, thank you very much! It nearly works, - just not completely yet.
The way I test is:
`[key@key HaskellR]$ stack exec H
Prelude H.Prelude> p [r| library(keras) |]
[1] "keras" "stats" "graphics" "grDevices" "utils" "datasets"
[7] "methods" "base"
Prelude H.Prelude> p [r| reticulate::py_config() |]
python: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/bin/python
libpython: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/lib/libpython3.5m.so
pythonhome: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3:/nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3
version: 3.5.3 (default, Jan 17 2017, 07:57:56) [GCC 5.4.0]
numpy: /nix/store/8r9ihbvq3r226yra5qfp0fkcwaxjzj67-python3.5-numpy-1.11.3/lib/python3.5/site-packages/numpy
numpy_version: 1.11.3
tensorflow: /nix/store/ixq9kpy5cjq1px8f17wni3jl5l3vh6w6-python3.5-tensorflow-1.0.0/lib/python3.5/site-packages/tensorflow
python versions found:
/nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/bin/python
/usr/bin/python
/usr/bin/python3
Prelude H.Prelude> p [r| model <- keras_model_sequential() |]
*** Exception: R Runtime Error: Error:
Prelude H.Prelude> p [r| reticulate::py_config() |]
python: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/bin/python
libpython: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/lib/libpython3.5m.so
pythonhome: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3:/nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3
version: 3.5.3 (default, Jan 17 2017, 07:57:56) [GCC 5.4.0]
numpy: /nix/store/8r9ihbvq3r226yra5qfp0fkcwaxjzj67-python3.5-numpy-1.11.3/lib/python3.5/site-packages/numpy
numpy_version: 1.11.3
tensorflow: /nix/store/ixq9kpy5cjq1px8f17wni3jl5l3vh6w6-python3.5-tensorflow-1.0.0/lib/python3.5/site-packages/tensorflow
python versions found:
/nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/bin/python
/usr/bin/python
/usr/bin/python3
Prelude H.Prelude> p [r| model <- keras_model_sequential() |]
*** Exception: R Runtime Error: Error: Python module tensorflow.contrib.keras.python.keras was not found.
Detected Python configuration:
python: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/bin/python
libpython: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/lib/libpython3.5m.so
pythonhome: /nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3:/nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3
version: 3.5.3 (default, Jan 17 2017, 07:57:56) [GCC 5.4.0]
numpy: /nix/store/8r9ihbvq3r226yra5qfp0fkcwaxjzj67-python3.5-numpy-1.11.3/lib/python3.5/site-packages/numpy
numpy_version: 1.11.3
tensorflow: /nix/store/ixq9kpy5cjq1px8f17wni3jl5l3vh6w6-python3.5-tensorflow-1.0.0/lib/python3.5/site-packages/tensorflow
python versions found:
/nix/store/zvm545rqc4d97caqq9h7344bnd06jhzb-python3-3.5.3/bin/python
/usr/bin/python
/usr/bin/python3
Prelude H.Prelude>
`
The problem seems to be that the tensorflow installation does not contain the tensorflow.contrib.keras.python.keras module:
[key@key HaskellR]$ ll /nix/store/ixq9kpy5cjq1px8f17wni3jl5l3vh6w6-python3.5-tensorflow-1.0.0/lib/python3.5/site-packages/tensorflow/contrib/ | grep keras
For comparison, when I compare my 2 conda envs containing tensorflow (the one from rstudio install_tensorflow and another one) I see that just the one from rstudio has that contrib.keras module.
Do you think you could set it up in a way that "tensorflow as rstudio wants it" would be installed?
(I mean, "as rstudio-keras wants it", because rstudio-tensorflow of course works fine without...)
Thanks a lot,
Sigrid
I can't replicate your failure there. I have added a commit onto keras-thing branch which pins the exact version of Nix packages I'm using. On commit c07c2fd of this repository, I get:
[shana@lenalee:~/programming/HaskellR]$ nix-shell
building path(s) ‘/nix/store/0j5n4qf984saxprqhsp004836b2zjxad-nixpkgs-5b5f3f542a8d62f16ba360dfde42a98c7ee02e1a-src’
trying https://github.com/NixOS/nixpkgs/archive/5b5f3f542a8d62f16ba360dfde42a98c7ee02e1a.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 155 0 155 0 0 336 0 --:--:-- --:--:-- --:--:-- 364
100 11.3M 100 11.3M 0 0 4475k 0 0:00:02 0:00:02 --:--:-- 7334k
unpacking source archive /tmp/nix-build-nixpkgs-5b5f3f542a8d62f16ba360dfde42a98c7ee02e1a-src.drv-0/5b5f3f542a8d62f16ba360dfde42a98c7ee02e1a.tar.gz
[nix-shell:~/programming/HaskellR]$ stack --nix build && stack --nix exec -- H
Prelude H.Prelude> p [r| library(keras) |]
[1] "keras" "stats" "graphics" "grDevices" "utils" "datasets"
[7] "methods" "base"
Prelude H.Prelude> p [r| reticulate::py_config() |]
python: /nix/store/5kfn0xxh3ipjdyjly2d5wrmh4cidsm8k-python3-3.5.3/bin/python
libpython: /nix/store/5kfn0xxh3ipjdyjly2d5wrmh4cidsm8k-python3-3.5.3/lib/libpython3.5m.so
pythonhome: /nix/store/5kfn0xxh3ipjdyjly2d5wrmh4cidsm8k-python3-3.5.3:/nix/store/5kfn0xxh3ipjdyjly2d5wrmh4cidsm8k-python3-3.5.3
version: 3.5.3 (default, Jan 17 2017, 07:57:56) [GCC 5.4.0]
numpy: /nix/store/7xk7ylqagd3w72y1p39rlcsamx3hcxl3-python3.5-numpy-1.12.1/lib/python3.5/site-packages/numpy
numpy_version: 1.12.1
tensorflow: /nix/store/qmr3h3f1s1x1dr64sb61igig8469imn6-python3.5-tensorflow-1.1.0/lib/python3.5/site-packages/tensorflow
Prelude H.Prelude> p [r| model <- keras_model_sequential() |]
Model
________________________________________________________________________________
Layer (type) Output Shape Param #
================================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0
________________________________________________________________________________
Prelude H.Prelude>
[nix-shell:~/programming/HaskellR]$ find /nix/store/qmr3h3f1s1x1dr64sb61igig8469imn6-python3.5-tensorflow-1.1.0/lib/python3.5/site-packages/tensorflow -name keras -type d | head
/nix/store/qmr3h3f1s1x1dr64sb61igig8469imn6-python3.5-tensorflow-1.1.0/lib/python3.5/site-packages/tensorflow/contrib/keras
/nix/store/qmr3h3f1s1x1dr64sb61igig8469imn6-python3.5-tensorflow-1.1.0/lib/python3.5/site-packages/tensorflow/contrib/keras/api/keras
/nix/store/qmr3h3f1s1x1dr64sb61igig8469imn6-python3.5-tensorflow-1.1.0/lib/python3.5/site-packages/tensorflow/contrib/keras/python/keras
Please let me know if that exact commit works for you.
Hi,
this works great, thank you so much!!!
Now I can do deep learning with HaskellR :-)
Thx again,
Sigrid
Great! Don't hesitate to open new issues if you stumble upon more problems. I'd also advise to make a copy of the keras-thing branch in local git checkout or otherwise as I can't guarantee we'll hold onto it forever: more likely than not, we'll just delete it in near future.
Thanks, I just checked out the branch! Thanks again for the help!