Windows native support

Question

Windows native support

jleni opened this issue 7 years ago · comments

The library is currently not supported in Windows. The xmss reference code requires access to /dev/urandom plus there are some incompatibilities with Microsoft compiler.

For the moment, only the webassembly is supported. We are currently working on improving the code base to support a native Windows version.

Juan Leni · Answer 1 · Thu Mar 22 2018 23:18:11 GMT+0800 (China Standard Time)

A lot of progress has been made by @kstuart
Unfortunately, changes in cmake seemed to have affected the deployment of pyqrllib. After merging this PR, it was not possible to import pyqrllib in some cases. This will required further investigation.
The code is now located in https://github.com/theQRL/qrllib/tree/feature/windows until we can determine the issue

kstuart · Answer 2 · Thu Mar 22 2018 23:25:39 GMT+0800 (China Standard Time)

That's interesting, sorry about that. Are there any details or steps to reproduce you can provide?

Juan Leni · Answer 3 · Thu Mar 22 2018 23:27:15 GMT+0800 (China Standard Time)

so we detected that when releasing a new package to pypi (0.8.14)
I have not checked yet, but I assume that it could be reproduced by running

python setup.py sdist

kstuart · Answer 4 · Fri Mar 23 2018 00:08:18 GMT+0800 (China Standard Time)

Ok, I see it. for some reason setup.py in copying the modules (.so files) into /usr/local/lib/python3.5/dist-packages instead of /usr/local/lib/python3.5/dist-packages/pyqrllib

must have been something I did last minute as I've older builds in the right place, will investigate.

Have checked pyqryptonight and that's putting everything in the right place.

donnerparty · Answer 5 · Wed Jun 27 2018 15:56:05 GMT+0800 (China Standard Time)

Some additional context for the errors I have seen.

`pip3 install pyqryptonight
Collecting pyqryptonight
Using cached https://files.pythonhosted.org/packages/80/e3/86f695447bb68e9a0d148740795a66c8b209f38634f581574e62aa952dba/pyqryptonight-0.99.3.tar.gz
Building wheels for collected packages: pyqryptonight
Running setup.py bdist_wheel for pyqryptonight ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" bdist_wheel -d /tmp/tmpofduhv_4pip-wheel- --python-tag cp35:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/pyqryptonight
copying pyqryptonight/init.py -> build/lib.linux-x86_64-3.5/pyqryptonight
copying pyqryptonight/pyqryptonight.py -> build/lib.linux-x86_64-3.5/pyqryptonight
running build_ext
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- BUILD_TESTS OFF
-- PYTHON WRAPPER ON
-- WEBASSEMBLY OFF
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find SWIG (missing: SWIG_EXECUTABLE SWIG_DIR)
Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.5/Modules/FindSWIG.cmake:75 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:32 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/pip-build-wvr1mqxx/pyqryptonight/build/temp.linux-x86_64-3.5/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py", line 71, in
setup_package()
File "/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py", line 67, in setup_package
use_pyscaffold=True)
File "/home/donner/.local/lib/python3.5/site-packages/setuptools/init.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 179, in run
self.run_command('build')
File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py", line 43, in run
subprocess.check_call(cmake_call, cwd=self.build_temp, env=env)
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-build-wvr1mqxx/pyqryptonight', '-DBUILD_PYTHON=ON', '-DBUILD_TESTS=OFF', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-build-wvr1mqxx/pyqryptonight/build/lib.linux-x86_64-3.5', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1

Failed building wheel for pyqryptonight
Running setup.py clean for pyqryptonight
Failed to build pyqryptonight
Installing collected packages: pyqryptonight
Running setup.py install for pyqryptonight ... error
Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-ni__y8ug-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/pyqryptonight
copying pyqryptonight/init.py -> build/lib.linux-x86_64-3.5/pyqryptonight
copying pyqryptonight/pyqryptonight.py -> build/lib.linux-x86_64-3.5/pyqryptonight
running build_ext
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- BUILD_TESTS OFF
-- PYTHON WRAPPER ON
-- WEBASSEMBLY OFF
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find SWIG (missing: SWIG_EXECUTABLE SWIG_DIR)
Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.5/Modules/FindSWIG.cmake:75 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:32 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/pip-build-wvr1mqxx/pyqryptonight/build/temp.linux-x86_64-3.5/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py", line 71, in <module>
    setup_package()
  File "/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py", line 67, in setup_package
    use_pyscaffold=True)
  File "/home/donner/.local/lib/python3.5/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/donner/.local/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
    self.run_command('build')
  File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/pip-build-wvr1mqxx/pyqryptonight/setup.py", line 43, in run
    subprocess.check_call(cmake_call, cwd=self.build_temp, env=env)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-build-wvr1mqxx/pyqryptonight', '-DBUILD_PYTHON=ON', '-DBUILD_TESTS=OFF', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-build-wvr1mqxx/pyqryptonight/build/lib.linux-x86_64-3.5', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1`

kstuart · Answer 6 · Wed Jun 27 2018 16:16:24 GMT+0800 (China Standard Time)

You appear not to be performing a Windows native build?

If you want to try building on Windows use the Windows build instructions: https://github.com/kstuart/qrllib#windows-1

converghub · Answer 7 · Mon Aug 06 2018 07:46:42 GMT+0800 (China Standard Time)

When parsing string, I got CMake error: invalid escape sequence \brabra.

Perhaps you'll need to do this by character replacement feature of string method in CMakeLists.txt.

string(REPLACE "\\" "/" CMAKE_CURRENT_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
string(REPLACE "\\" "/" CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_LIBRARY_OUTPUT_DIRECTORY})

Juan Leni · Answer 8 · Thu Aug 09 2018 15:01:06 GMT+0800 (China Standard Time)

@converghub CMAKE has a native/correct way of doing this. Please have a look at:

file(TO_CMAKE_PATH path result)
file(TO_NATIVE_PATH path result)

https://cmake.org/cmake/help/v3.0/command/file.html

@donnerparty From what I see in the error message. SWIG is missing and you seem to be compiling in Linux. How is that related to this issue?

kstuart · Answer 9 · Sat Aug 11 2018 02:33:14 GMT+0800 (China Standard Time)

@converghub Your comment is not helpful.

No steps to follow? Are you doing your own thing or following the instructions from the README?
You mention cmake, but not what version you're using?
I've no idea what string you're referring to? No context or detail provided.

Are you building qrllib or did you accidentally post to the wrong project?

...but seriously, when seeing an issue please provide enough detail.

Native builds on windows are working.

If you're using cmake 3.12 there is a compatibility issue (nothing to do with escape sequences) that's fixed in the latest PR #155.

If you're looking to run a native windows QRL node you'll also need py-leveldb for windows, I built a python 3.6 bdist wheel a while back that you can grab here.

0xFF0 · Answer 10 · Fri Dec 31 2021 12:30:17 GMT+0800 (China Standard Time)

Based on kstuart's work (thanks for your work), I think I have found a good solution to this problem.

To simplify the solution, I am using mingw and not Visual Studio.

All the modifications were added with #ifdef _WIN32 except for the swig interface (in /src/api/*.i) where uint was replaced by unsigned int.

I have not submitted a PR because I did not test the code on other OS, but I think it should work.

Here's the PoC: https://github.com/0xFF0/Qrllib_Windows