JNI implementation

[GuilhemSempere]

Hi

We have several web applications submitting jobs to SGE using the Java-based Opal Toolkit (https://sourceforge.net/projects/opaltoolkit/) via drmaa.jar

In the process of migrating from SGE to Slurm I realized that the JNI methods do not exist in the slurm-drmaa library. Do you know of any out-of-the-box solution for submitting jobs via drmaa.jar to Slurm?

Best regards,

Guilhem

[varduefr]

Hi, I'm trying to do the same using drmaa.jar from SGE but after compiling libdrmaa.so for SLURM I'm trying to point to the compiled library in:
export DRMAA_LIBRARY_PATH=/home/vargasfr/slurm_drmaa/slurm-drmaa-1.1.2/install/lib/libdrmaa.so
export LD_LIBRARY_PATH=/home/vargasfr/slurm_drmaa/slurm-drmaa-1.1.2/install/lib/

But I'm getting an error in the Java code:
Exception in thread "main" java.lang.UnsatisfiedLinkError: com.sun.grid.drmaa.SessionImpl.nativeInit(Ljava/lang/String;)V
at com.sun.grid.drmaa.SessionImpl.nativeInit(Native Method)
at com.sun.grid.drmaa.SessionImpl.init(SessionImpl.java:291)
at Test2.main(Test2.java:16)

Did you find a solution?
Thanks!

[GuilhemSempere]

Hi

Well the solution I went for was to write my own JobManager (which doesn't use drmaa) for Opal Toolkit.

If your goal is also to make Opal work with Slurm you will probably be able to use this out of the box: https://gitlab.southgreen.fr/AGAP_ID_Java/Opal-SlurmCompatible

Otherwise maybe you can use it as an example and adapt it to your own purpose?

G.

[varduefr]

Hi Guilhem, Thanks for your help! In principle I'm builiding KNIME plugins (nodes) in Java so in my previous implementation I just used the drmaa.jar that is available for SGE (when we had SGE in the cluster) and just used their classes to create an SGE session, job template in order to submit SGE jobs to the cluster from the KNIME nodes. I've seen you have a jar folder that contains SlurmJobManager.jar, Adding this to my library folder or CLASSPATH will allow the application access SLURM and submit jobs like the drmaa.jar file? If not, as I'm not using OPAL which part from your project is doing just the link to SLURM and the job submission? Thanks! El mié, 1 sept 2021 a las 11:23, GuilhemSempere ***@***.***>) escribió:

…

Hi Well the solution I went for was to write my own JobManager (which doesn't use drmaa) for Opal Toolkit. If your goal is also to make Opal work with Slurm you will probably be able to use this out of the box: https://gitlab.southgreen.fr/AGAP_ID_Java/Opal-SlurmCompatible Otherwise maybe you can use it as an example and adapt it to your own purpose? G. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4WGY6WS4B6LEVMVHVHHCDT7XWJPANCNFSM4O55VQDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[GuilhemSempere]

This jar is the result of running build.xml, it contains just the 2 classes that needed to be added to Opal Toolkit for it to be compatible with Slurm. So this is indeed where you have to look:

https://gitlab.southgreen.fr/AGAP_ID_Java/Opal-SlurmCompatible/-/blob/master/src/edu/sdsc/nbcr/opal/manager/SlurmJobManager.java

https://gitlab.southgreen.fr/AGAP_ID_Java/Opal-SlurmCompatible/-/blob/master/src/edu/sdsc/nbcr/opal/manager/slurm/Job.java

I must specify that if your current app entirely relies on drmaa, I think you'll have much more to implement. In my case Opal was already able to talk to several non-drmaa schedulers so it was quite easy to take one of them as example and adapt the code to Slurm syntax.

Good luck ;-)

[Nate Coraor]

I'm not really familiar enough with Java to understand the details here but it seems that Java bindings for DRMAA like the one provided with SGE are not simply wrappers around the C libraries like the bindings for other languages (e.g. Python). Here is some discussion of it from a long while back for someone looking to do the same with Torque:

https://stackoverflow.com/questions/2715243/is-there-a-drmaa-java-library-that-works-with-torque-pbs

Creating a full Java DRMAA binding that is Slurm-specific is well outside the scope of this project, unless I am misunderstanding what is necessary to support a Java binding that links against slurm-drmaa.

[Nate Coraor]

Closing for out of scope, but please let me know if I've misunderstood.