Windows Support: Hadoop fails to load Loaded Libraries
kadaan opened this issue · comments
Windows tries to load the native hadoop binaries which do not exist. It tries to load them from a wide range of folder because Windows returns Windows-<Version> from os.name. After changing this to use 'Windows' as the os.name for all Windows versions and the addition of the native libraries to /nativelib/Windows_amd64, HadoopNative correctly loads the libraries from the temp files that are created. Unfortunately, when loadAllCodecs runs it subsequently fails with the following error:
java.lang.RuntimeException: failed to load Hadoop native library
at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:60)
at com.facebook.presto.hadoop.TestHadoopNative.testCodecRoundTrip(TestHadoopNative.java:51)
Caused by: java.lang.RuntimeException: native snappy library not available: SnappyCompressor has not been loaded.
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:68)
at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:189)
at com.facebook.presto.hadoop.HadoopNative.loadAllCodecs(HadoopNative.java:73)
at com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.java:55)
Changing HadoopNative.requireHadoopNative to create a temporary directory and making loadLibrary copy the embedded files with the original names into this directory allows the libraries to correctly loaded.
Are Hadoop native libraries for Windows available anywhere? I did a quick search and couldn't find them.
For sure. Hadoop runs on Windows Azure (HDInight), is available to be installed on Windows machines (Hortonworks), and there are instructions for building Hadoop with native libraries for Windows (http://www.srccodes.com/m/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os).
Did this answer your question? I'm interested in getting Presto to work on Windows, but could use feedback so that the correct changes are made.
Do you know if the pre-built native libraries are available from Hortonworks or another official source? None of us have a Windows machine.
Yes, there are prebuilt native libraries available. I'll send the link a little later.
It is available from Hortonworks at: http://hortonworks.com/products/releases/hdp-2-windows/
The pre-built binaries are missing bzip2 support which cause the NativeLibraryChecker to fail. They will need to be re-built to support bzip2 or the code that calls NativeLibraryChecker will need to take over the check and special case Windows.
Either is fine. bzip2 isn't widely used anyway.
Closing this as parent task prestodb/presto#2351 was also closed.