pyo3 and windows 11:
xnerhu opened this issue · comments
PS C:\projects\nersent\prompusie> make common.nn
makefile:155: warning: overriding recipe for target 'proto.gen'
makefile:25: warning: ignoring old recipe for target 'proto.gen'
cross-env LIBTORCH_USE_PYTORCH=1 maturin build --manifest-path common/nn/Cargo.toml -o build/common/nn --interpreter python3.10 --release
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: C:\projects\nersent\prompusie\common\rust\Cargo.toml
workspace: C:\projects\nersent\prompusie\Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: C:\projects\nersent\prompusie\common\nn\Cargo.toml
workspace: C:\projects\nersent\prompusie\Cargo.toml
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
⚠️ Warning: Please use maturin in pyproject.toml with a version constraint, e.g. `requires = ["maturin>=1.0,<2.0"]`. This will become an error.
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: C:\projects\nersent\prompusie\common\rust\Cargo.toml
workspace: C:\projects\nersent\prompusie\Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: C:\projects\nersent\prompusie\common\nn\Cargo.toml
workspace: C:\projects\nersent\prompusie\Cargo.toml
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at C:\Python310\python.exe
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: C:\projects\nersent\prompusie\common\rust\Cargo.toml
workspace: C:\projects\nersent\prompusie\Cargo.toml
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package: C:\projects\nersent\prompusie\common\nn\Cargo.toml
workspace: C:\projects\nersent\prompusie\Cargo.toml
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
warning: C:\projects\nersent\prompusie\common\nn\Cargo.toml: unused manifest key: lib.version
warning: C:\projects\nersent\prompusie\common\rust\Cargo.toml: unused manifest key: lib.version
warning: unused import: `std::sync::Arc`
--> common\rust\pyo3_utils.rs:1:5
|
1 | use std::sync::Arc;
| ^^^^^^^^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused imports: `PyRef`, `types::PyDict`
--> common\rust\pyo3_utils.rs:3:12
|
3 | use pyo3::{types::PyDict, PyAny, PyRef};
| ^^^^^^^^^^^^^ ^^^^^
warning: unused imports: `HashMap`, `PathBuf`, `VecDeque`, `fs::create_dir_all`
--> common\rust\utils.rs:2:19
|
2 | collections::{HashMap, VecDeque},
| ^^^^^^^ ^^^^^^^^
3 | ffi::OsStr,
4 | fs::create_dir_all,
| ^^^^^^^^^^^^^^^^^^
5 | path::{Path, PathBuf},
| ^^^^^^^
warning: unused import: `lazy::*`
--> common\rust\utils.rs:8:14
|
8 | use polars::{lazy::*, prelude::*};
| ^^^^^^^
warning: unused imports: `prelude::*`, `types::PyDict`
--> common\rust\utils.rs:9:12
|
9 | use pyo3::{prelude::*, types::PyDict};
| ^^^^^^^^^^ ^^^^^^^^^^^^^
warning: `rust_lib` (lib) generated 5 warnings (run `cargo fix --lib -p rust_lib` to apply 5 suggestions)
warning: unused import: `lazy::*`
--> common\nn\dataset\dataset_processing.rs:4:14
|
4 | use polars::{lazy::*, prelude::*};
| ^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused imports: `Rng`, `Uniform`, `thread_rng`
--> common\nn\dataset\dataset_processing.rs:7:35
|
7 | distributions::{Distribution, Uniform},
| ^^^^^^^
8 | thread_rng, Rng,
| ^^^^^^^^^^ ^^^
warning: unused import: `core::panic`
--> common\nn\tokenizers\char_tokenizer.rs:1:5
|
1 | use core::panic;
| ^^^^^^^^^^^
warning: unused imports: `PathBuf`, `VecDeque`, `ffi::OsStr`
--> common\nn\tokenizers\char_tokenizer.rs:3:28
|
3 | collections::{HashMap, VecDeque},
| ^^^^^^^^
4 | ffi::OsStr,
| ^^^^^^^^^^
5 | fs::create_dir_all,
6 | path::{Path, PathBuf},
| ^^^^^^^
warning: unused import: `lazy::*`
--> common\nn\tokenizers\char_tokenizer.rs:9:14
|
9 | use polars::{lazy::*, prelude::*};
| ^^^^^^^
warning: unused import: `types::PyDict`
--> common\nn\tokenizers\char_tokenizer.rs:10:24
|
10 | use pyo3::{prelude::*, types::PyDict};
| ^^^^^^^^^^^^^
warning: unused imports: `PyTensor`, `wrap_tch_err`
--> common\nn\lib.rs:8:16
|
8 | use pyo3_tch::{wrap_tch_err, PyTensor};
| ^^^^^^^^^^^^ ^^^^^^^^
warning: unused import: `rayon::prelude`
--> common\nn\tokenizers\char_tokenizer.rs:11:5
|
11 | use rayon::prelude::*;
| ^^^^^^^^^^^^^^
warning: unused import: `Distribution`
--> common\nn\dataset\dataset_processing.rs:7:21
|
7 | distributions::{Distribution, Uniform},
| ^^^^^^^^^^^^
warning: unused import: `prelude`
--> common\nn\tokenizers\char_tokenizer.rs:9:23
|
9 | use polars::{lazy::*, prelude::*};
| ^^^^^^^
warning: unused variable: `timestep_prompt_idx`
--> common\nn\dataset\dataset_processing.rs:181:17
|
181 | let timestep_prompt_idx = add_prompt(timestep_prompt);
| ^^^^^^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_timestep_prompt_idx`
|
= note: `#[warn(unused_variables)]` on by default
warning: unused variable: `label_prompt_idx`
--> common\nn\dataset\dataset_processing.rs:187:17
|
187 | let label_prompt_idx = add_prompt(label_prompt);
| ^^^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_label_prompt_idx`
warning: unused variable: `py`
--> common\nn\dataset\dataset_processing.rs:101:25
|
101 | pub fn generate_prompts(py: Python<'_>, config: &PyDict) {
| ^^ help: if this is intentional, prefix it with an underscore: `_py`
warning: variable does not need to be mutable
--> common\nn\dataset\dataset_processing.rs:190:13
|
190 | let mut metadata_file = TimeSeriesDatasetFile {
| ----^^^^^^^^^^^^^
| |
| help: remove this `mut`
|
= note: `#[warn(unused_mut)]` on by default
warning: unused variable: `py`
--> common\nn\dataset\dataset_processing.rs:242:25
|
242 | pub fn generate_windows(py: Python<'_>, config: &PyDict) {
| ^^ help: if this is intentional, prefix it with an underscore: `_py`
warning: unused variable: `py`
--> common\nn\dataset\dataset_processing.rs:356:25
|
356 | pub fn generate_batches(py: Python<'_>, config: &PyDict) {
| ^^ help: if this is intentional, prefix it with an underscore: `_py`
warning: unused variable: `py`
--> common\nn\lib.rs:14:23
|
14 | pub fn py_levenshtein(py: Python<'_>, a: &str, b: &str) -> usize {
| ^^ help: if this is intentional, prefix it with an underscore: `_py`
warning: unused variable: `py`
--> common\nn\lib.rs:25:11
|
25 | fn nn_lib(py: Python<'_>, m: &PyModule) -> PyResult<()> {
| ^^ help: if this is intentional, prefix it with an underscore: `_py`
warning: `nn_lib` (lib) generated 18 warnings (run `cargo fix --lib -p nn_lib` to apply 15 suggestions)
Finished release [optimized] target(s) in 0.63s
📦 Built wheel for CPython 3.10 to build/common/nn\nn_lib-0.0.1-cp310-none-win_amd64.whl
pip install build/common/nn/nn_lib-*.whl --force-reinstall
WARNING: Ignoring invalid distribution -ip (c:\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -lip (c:\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ransformers (c:\python310\lib\site-packages)
Processing c:\projects\nersent\prompusie\build\common\nn\nn_lib-0.0.1-cp310-none-win_amd64.whl
WARNING: Ignoring invalid distribution -ip (c:\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -lip (c:\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ransformers (c:\python310\lib\site-packages)
Installing collected packages: nn-lib
Attempting uninstall: nn-lib
Found existing installation: nn_lib 0.0.1
Uninstalling nn_lib-0.0.1:
Successfully uninstalled nn_lib-0.0.1
Successfully installed nn-lib-0.0.1
PS C:\projects\nersent\prompusie> cross-env PYTHONPATH="." python ai/xd.py
Traceback (most recent call last):
File "C:\projects\nersent\prompusie\ai\xd.py", line 1, in <module>
import nn_lib
File "C:\Python310\lib\site-packages\nn_lib\__init__.py", line 1, in <module>
from .nn_lib import *
ImportError: DLL load failed while importing nn_lib: The specified module could not be found.
pyo3 = { version = "0.18", features = ["extension-module", "macros"] }
pyo3-ffi = "0.18"
serde-pickle = "1.0"
rayon = "1.8.0"
rust_lib = { path = "../rust" }
levenshtein = "1.0.5"
torch-sys = "0.14.0"
pyo3-tch = "0.14.0"
PS C:\WINDOWS\system32> pip3 show torch
WARNING: Ignoring invalid distribution -ip (c:\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -lip (c:\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ransformers (c:\python310\lib\site-packages)
Name: torch
Version: 2.1.0+cu118
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: c:\python310\lib\site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: accelerate, clip, clip-interrogator, compel, fairscale, invisible-watermark, neuralforecast, openai-whisper, pytorch-lightning, sentence-transformers, timm, torchaudio, torchmetrics, torchvision
PS C:\WINDOWS\system32> pip --version
pip 23.3.2 from C:\Python310\lib\site-packages\pip (python 3.10
use pyo3::prelude::*;
use pyo3_tch::{wrap_tch_err, PyTensor};
// #[pyfunction]
// fn add_one(tensor: PyTensor) -> PyResult<PyTensor> {
// let tensor = tensor.f_add_scalar(1.0).map_err(wrap_tch_err)?;
// Ok(PyTensor(tensor))
// }
#[pymodule]
fn nn_lib(py: Python<'_>, m: &PyModule) -> PyResult<()> {
// m.add_function(wrap_pyfunction!(add_one, m)?)?;
Ok(())
}If I remove
use pyo3_tch::{wrap_tch_err, PyTensor};
python file runs without errors, so it's pyo3 tch related issue.
I don't have set up any LIBTORCH env in windows. I only compile with LIBTORCH_USE_PYTORCH=1
I reordered imports in python, to import torch first, before the library and it works.