Loading library with same name as system library will have an unexpected library loaded

[Bassam Helal]

Scenario:

This pertains mainly to GNU/Linux.
I want to use a local version of a library with the file name libfoo.so.
If my system has in its default lib paths (such as /lib/x86_64-linux-gnu) a library with the same name and a higher version (such as libfoo.so.3) then that library (in the system paths) will be loaded despite adding my own library in the preferred search path. My one wins only if my version is greather than or equal to the one found in the system. The system finding technically will see mine first (expected) but sees it as inferior to the system one.

Expected Behavior:

My added search paths are ALWAYS preferred, even if a better version exists in the system paths. This could be because I want to use a non versioned library of a library that happens to be found on the system. For example if I actually want to use my own libc.so, which could be different from the system's libc.so.6, or even just my own older version for some reason like libc.so.4. Obviously this example is contrived and a little ridiculous, but it may be the case that a library is, unknowingly to the consumers, actually installed on the client's machine, and with a higher version than the one distributed which may even have no version, leading to confusing errors that are tricky to debug.

Solution

The solution is relatively simple and is in Platform.Linux.locateLibrary() where towards the end of the function there is version checking. We could add a LibraryOption to give consumers preference for higher and/or system versions, if for example they're distributing some common library for compatibility but would prefer to use the system's one, which could be of a higher version.

I think a great API addition related to this, (and one I would personally find great use in) is some way to query which library am I actually using. So that I can be aware at runtime, the path of the library I am currently using.

I have begun to implement a solution to this on the javadocs branch of my fork but it will need some deep internal changes.

Currently, libraries are loaded lazily, in this case meaning when there are declared functions in the mapping interface that need to be bound to native functions in the library. Only if a function exists in that interface will JNR-FFI actually attempt to load the library and look for the address of the function with that name. In the case of an empty interface it will silently fail (or succeed), this is somewhat fine because it doesn't actually matter, the interface was empty but when you fill the interface, then the loading happens and only then will we be able to check for success, and which paths were successful etc. Also how would we be able to query these successful paths, ie which place do consumers call, Runtime is a good candidate but implementing that could be difficult. Adding another method to LoadedLibrary is expensive since that's another special function name we need to be aware of. The actual successful paths are determined in NativeLibrary so we need it to be from something that can access that.

The ideal scenario is something like:

LibC libc = LibraryLoader.loadLibrary(
            LibC.class,
            libraryOptions,
            searchPaths,
            libName
);

String libPath = Runtime.getLibraryPath(libc);

I'm going to try to do both of these myself in my fork. The main issue is easy enough but the important addition to the API is not (at least from what I can tell).

[Bassam Helal]

Let me know if this is something you think is worth pursuing and if so, assign the issue to me.

[Bassam Helal]

Also, I don't know how library versioning is done on Darwin. This site says Darwin does indeed have library versioning which we actually do not implement as of right now. My experience has been difficult with dylibs on modern MacOS so that could make things tricky for Darwin.

[Bassam Helal]

I found a working and somewhat elegant solution to the querying of successful library paths. Essentially adding a variable into NativeLibrary for successful paths. Runtime now has a new method getLoadedLibraryPaths() which contains a map of library names to successful paths that is updated upon successful library loads. When a library is loaded successfully we update the NativeLibrary's successfulPaths and we get the Runtime.getSystemRuntime() and update its local variable to reflect this as well. It has minimal cost overall and does what we want. I am yet to write tests for this but I have ran some code and it works as expected however there are 2 main issues I'm annoyed about:

• If a library gets unloaded the Map returned by getLoadedLibraryPaths() will not reflect this, for example if I use my LibFoo interface mapping in an instance libFoo and then I'm done with it, it gets GC'd and I move on. If I now query Runtime.getLoadedLibraryPaths() it will contain a reference to LibFoo something I've long forgotten about. This isn't a bad thing necessarily and the way to solve this is very convoluted, but it's a little annoyance of mine. UPDATE: Actually just putting a finalize() method into NativeLibrary to remove it from the Runtime's paths works because a library loaded using AsmLibraryLoader will always contain a strong reference to the NativeLibrary but I don't feel 100% comfortable with finalize() though jnr:jffi uses it in Library to call dlclose() so I guess it's not that bad.

• More importantly, we need a better key for the map. Currently Runtime.getSystemRuntime() returns a
  Map<List<String>, List<String>> with the key being the library's name(s) and the value being the library's successful path(s). This isn't great because we can technically load a library with the name foo multiple different times with different mapping interfaces (and even paths), the names just happened to be the same. We need a better way to uniquely identify a library. I was thinking maybe a combination of names and the mapping interface Class so library foo loaded for interface LibFooA will differ from library foo (possibly different paths even) for interface LibFooB since the mapping interfaces are different.

I know I'm being a little verbose, but I want to log my progress and changes in a detailed way so I can come back to them and for better understanding of what I'm doing for feedback and ideas.

[Charles Oliver Nutter]

@basshelal Perhaps you could stop by our Matrix chat some time to discuss this? I am going forward with releasing 2.2.4 so we can pick up typedef fixes for JRuby 9.2.18.0, but we could do a quick flip for 2.2.5 to improve this.

[Bassam Helal]

@headius Yeah no worries, I'm not too fussed about when this gets fixed/merged (as long as it eventually does)

Actually I've made huge progress regarding this in my fork and I'd say I'm 80% done with it, it's quite a few changes though, so it'll need more time to review and test etc.

I'll open a PR when I'm more comfortable with it, and I'll definitely use the Matrix chat if I need.

Cheers 😊