Giters

combust / mleap

MLeap: Deploy ML Pipelines to Production

Home Page:https://combust.github.io/mleap-docs/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Need Help on Contributing towards Mleap for 2.13

prashil1996 opened this issue · comments

commented

Hi Developers,

I've recently forked the main MLeap repository with the intention of contributing to support Scala 2.13. However, I'm unfamiliar with the build process and prerequisites. Could you please provide guidance on the necessary steps to make local changes and build MLeap with Scala 2.13 compatibility?

Thank you for your assistance.
Prashil

commented

Use the make targets in the repo root to build and run tests. Once you have a build that you think works locally, then can commit and push to origin to create a PR. That will trigger a full build on our CI system.

Specifically regarding scala 2.13 compatibility, please aim for cross compiling to support both scala 2.12 and 2.13. sbt has native support for this and the repo should already be set up correctly to utilize it.

commented

@jsleight Thanks for replying, also what are the different requirements for compiling the build locally?
Scala, Java, SBT?

I tried building it locally but I was getting some error. Can you help me with some steps? Any wiki?

The following were encountered after running make on the root repo:

17:38:30.945 [pool-11-thread-20] ERROR ml.dmlc.xgboost4j.java.NativeLibLoader - Failed to load xgboost4j library from jar for platform macos/aarch64
17:38:30.948 [pool-11-thread-20] ERROR ml.dmlc.xgboost4j.java.DMatrix - Failed to load native library
java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
	at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:285) ~[xgboost4j_2.12-1.7.6.jar:?]
	at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:228) ~[xgboost4j_2.12-1.7.6.jar:?]
	at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:168) ~[xgboost4j_2.12-1.7.6.jar:?]
	at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34) ~[xgboost4j_2.12-1.7.6.jar:?]
	at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69) ~[xgboost4j_2.12-1.7.6.jar:?]
	at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32) ~[xgboost4j_2.12-1.7.6.jar:?]
	at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21) ~[test-classes/:?]
	at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14) ~[test-classes/:?]
	at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128) ~[?:?]
	at jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304) ~[?:?]
	at java.lang.Class.newInstance(Class.java:725) ~[?:?]
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454) ~[scalatest-core_2.12-3.2.16.jar:3.2.16]
	at sbt.TestRunner.runTest$1(TestFramework.scala:153) ~[?:?]
	at sbt.TestRunner.run(TestFramework.scala:168) ~[?:?]
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336) ~[?:?]
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296) ~[?:?]
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336) ~[?:?]
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336) ~[?:?]
	at sbt.TestFunction.apply(TestFramework.scala:348) ~[?:?]
	at sbt.Tests$.processRunnable$1(Tests.scala:475) ~[?:?]
	at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481) ~[?:?]
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47) ~[?:?]
	at sbt.std.Transform$$anon$4.work(Transform.scala:69) ~[?:?]
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283) ~[?:?]
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) ~[?:?]
	at sbt.Execute.work(Execute.scala:292) ~[?:?]
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283) ~[?:?]
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) ~[?:?]
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[info] XGBoostPredictorClassificationModelParitySpec:
[info] ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec *** ABORTED ***
[info]   java.lang.ExceptionInInitializerError:
[info]   at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[info]   at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[info]   at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[info]   at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[info]   at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[info]   at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info]   at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[info]   at java.base/java.lang.Class.newInstance(Class.java:725)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[info]   ...
[info]   Cause: java.lang.RuntimeException: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[info]   at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:37)
[info]   at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[info]   at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[info]   at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[info]   at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[info]   at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[info]   at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info]   at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[info]   at java.base/java.lang.Class.newInstance(Class.java:725)
[info]   ...
[info]   Cause: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[info]   at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:285)
[info]   at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:228)
[info]   at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:168)
[info]   at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
[info]   at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[info]   at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[info]   at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[info]   at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[info]   at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[info]   ...
[error] java.lang.ExceptionInInitializerError
[error] 	at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[error] 	at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[error] 	at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[error] 	at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[error] 	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[error] 	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[error] 	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error] 	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[error] 	at java.base/java.lang.Class.newInstance(Class.java:725)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error] 	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
[error] 	at sbt.TestRunner.run(TestFramework.scala:168)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
[error] 	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error] 	at sbt.TestFunction.apply(TestFramework.scala:348)
[error] 	at sbt.Tests$.processRunnable$1(Tests.scala:475)
[error] 	at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481)
[error] 	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[error] 	at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:37)
[error] 	at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[error] 	at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[error] 	at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[error] 	at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[error] 	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[error] 	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[error] 	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error] 	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[error] 	at java.base/java.lang.Class.newInstance(Class.java:725)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error] 	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
[error] 	at sbt.TestRunner.run(TestFramework.scala:168)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
[error] 	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error] 	at sbt.TestFunction.apply(TestFramework.scala:348)
[error] 	at sbt.Tests$.processRunnable$1(Tests.scala:475)
[error] 	at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481)
[error] 	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Caused by: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[error] 	at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:285)
[error] 	at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:228)
[error] 	at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:168)
[error] 	at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
[error] 	at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[error] 	at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[error] 	at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[error] 	at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[error] 	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[error] 	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[error] 	at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error] 	at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[error] 	at java.base/java.lang.Class.newInstance(Class.java:725)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error] 	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
[error] 	at sbt.TestRunner.run(TestFramework.scala:168)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
[error] 	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error] 	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error] 	at sbt.TestFunction.apply(TestFramework.scala:348)
[error] 	at sbt.Tests$.processRunnable$1(Tests.scala:475)
[error] 	at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481)
[error] 	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1583)
[error] (mleap-xgboost-runtime / Test / executeTests) java.lang.ExceptionInInitializerError
[error] Total time: 302 s (05:02), completed 14-Oct-2023, 5:38:31 pm
make: *** [test_xgboost_runtime] Error 1

Any inputs on how I can resolve this?

commented

Ah

  • sbt v1.9.4
  • java 11
  • spark 3.4
    (fyi: the /project directory of sbt projects usually has all these details.)

The libxgboost4j being missing is a known thing when trying to build on mac. XGBoost's java libraries don't support mac. iirc tensorflow libraries might have this problem too.