René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Library reboot and much more","articleBody":"Hello folks,\r\n\r\nI/we are trying to reboot the this library, the idea is to unify a lot of different leaf/sparse conventions all together for handling natives for the whole cycle, from the producer publication till the user consumption, since this is one of the cause of pain/additional overhead on end consumers and library developers, for example:\r\n- in Scenery/Sciview where we have to [manually fix](https://github.com/scenerygraphics/scenery/blob/main/build.gradle.kts#L65-L96) the pom\r\n- in the incoming Gradle Platform I have to [manually hardcode](https://github.com/scijava/pom-scijava/blob/2653c2cd723c2466d6a0ccf89811ca8f96453bed/gradle/platform/build.gradle.kts#L24) a list of the runtime dependencies to later on setup a runtime constraints for those, instead of relying on some sort of \"runtime classifier schema\"\r\n\r\nThis attempt is a chance to fix also some other issues which are somehow (strictly) related, hopefully once and for all, before the new Foreign Function API transition/JNI future limitations take over.\r\n\r\nActually all of this goes beyond the scope of `native-lib-loader`, but there isn't really a right place to discuss about all of these thematics, so I'd just over-exploit the \"Issues\" space here as a placeholder.\r\n\r\nLet me write down some of the ideas we came with, the goal is to reach a broader audience for further feedbacks, improvements, etc..\r\n\r\nThis might be interesting for some other people outside this repo other than @ctrueden (and @bmarwell?), such as @jjohannes and @Spasi\r\n\r\n### One artifact for each supported \"`OS`_`Arch`\"\r\n\r\nFirst of all, natives should be published for each OS/Arch combination, in order to avoid pulling in dependencies thus downloading useless files which are not meant in any way for the given platform, just polluting the consumer classpath, saving space, bandwidth and time on scale.\r\n\r\nThe good new is that most of libraries nowadays already follow this path, but some of them (like [`jinput`](https://repo1.maven.org/maven2/net/java/jinput/jinput/2.0.10/)), unfortunately, don't. We could help them by providing a PR ourselves or, in the worst cases, re-publishing the artifact ourselves on scijava (or central by new GAVs)? I don't know\r\n\r\n### Using the os-maven-plugin normalized os and arch values\r\n\r\nYou can see them as `enum`s [here](https://github.com/elect86/native-support/blob/master/plugin/src/main/kotlin/org/nativeSupport/enums.kt), they are copy/pasted from the [os-maven-plugin](https://github.com/trustin/os-maven-plugin/blob/master/src/main/java/kr/motd/maven/os/Detector.java#L150)\r\n\r\nGradle has already something, but it's somehow [limited](https://gist.github.com/elect86/b1f6826a4d799f989f3f72c223a6e754). They also use `macos` for `MACOS`, to say\r\n\r\n### Publishing using the [Gradle Metadata rich format](https://docs.gradle.org/current/userguide/variant_model.html#understanding-variant-selection)\r\n\r\nGradle does this natively, of course, but a Gradle plugin will make the creation of those variants even easier\r\n\r\nAnd the [`gradle-module-metadata-maven-plugin`](https://github.com/jjohannes/gradle-module-metadata-maven-plugin?tab=readme-ov-file) will produce the Gradle metadata for library developers using Maven.\r\n\r\nThese library developers using Maven and targetting other Maven consumers should declare the natives dependency in their pom using placeholders for `OS` and `Arch`.\r\nIn the Scijava ecosystem this is already done today in some places, such [here](https://github.com/flimlib/flimj-ops/blob/master/pom.xml#L138C18-L138C44)\r\nWe should come up with some convention for the classifier, something like `natives-$os-$arch` could be an idea. Lwjgl [does already](https://repo1.maven.org/maven2/org/lwjgl/lwjgl/3.3.3/) something like that, although for x64 classifiers it simply shorten to `natives-$os`.\r\n\r\n### Extracting and caching natives\r\n\r\nNatives will be extracted into a directory following:\r\n- [XDG conventions](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html), so under the user home (like `~/.cache/native-lib-loader`)\r\n- [Gradle dependency cache](https://docs.gradle.org/current/userguide/dependency_resolution.html#sec:dependency_cache) GAVH (\"`gropId`/`artifactId`/`version`/`hashSha1`\") schema, it's important to take the hashing in account, as according to the [Gradle docs](https://docs.gradle.org/current/userguide/dependency_resolution.html#sub:cache_checksum_storage):\r\n\u003e It is possible for different repositories to provide a different binary artifact in response to the same artifact identifier. This is often the case with Maven SNAPSHOT artifacts, but can also be true for any artifact which is republished without changing its identifier. By caching artifacts based on their SHA1 checksum, Gradle is able to maintain multiple versions of the same artifact. This means that when resolving against one repository Gradle will never overwrite the cached artifact file from a different repository. This is done without requiring a separate artifact file store per repository.\r\n\r\n### Consuming the natives\r\n\r\nGradle handles `.module` file natively and will properly resolve them, no work needed there, the consumer has to just set the os and arch attributes to find the right natives under the hood. A plugin could also help by setting these values automatically, from the running machine underneath.\r\nScijava already does [something similar](https://github.com/flimlib/flimj-ops/blob/efd6d84ea67652b0ff1559a36cb7c610d3773e5b/pom.xml#L138) with `scijava.natives.classifier` \u0026 company. And corresponding Maven plugin (or whatsoever) might also provide the same automatic setup underneath\r\n\r\n### Loading the natives in runtime\r\n\r\nA `NativeLoader2` (or whatsoever) will read all the underlying native libraries from the given GAVH (\"`gropId`/`artifactId`/`version`/`hashSha1`\") coordinate, extracting them if they weren't already cached before, and load them downstream safely respecting the [JNI restriction](https://docs.oracle.com/javase/6/docs/technotes/guides/jni/jni-12.html#libmanage)\r\n\u003e The same JNI native library cannot be loaded into more than one class loader.\r\n\r\nThis means that classloader must match with the classloader of the class specifying the native methods.\r\nThis is achievable by the i-th class by passing down to `NativeLoader2` the reference to `System::load`, like Lwjgl is already [doing](https://github.com/LWJGL/lwjgl3/blob/master/modules/lwjgl/opengl/src/main/java/org/lwjgl/opengl/GL.java#L80) since ages\r\n\r\nAlso, we should prefer `System::load` to `System::loadLibrary`, since as it's noted [here](https://stackoverflow.com/questions/7016391/difference-between-system-load-and-system-loadlibrary-in-java#comment127547281_7016496)\r\n\r\n\u003e The drawback of loadLibrary() is so that it returns the confusing error message \"java.lang.UnsatisfiedLinkError: myLib (Not found in java.library.path)\" even if it is found, but could not be loaded due to other reasons. In such case calling load()/loadLibrary() with the absolute path shows additional output like \"rtld: Symbol myFunc was referenced ... but a runtime definition was not found.\" which is hidden by loadLibrary() with the library basename as argument. That is the best way is to write own method which iterates through java.library.path and calls load() per item.\r\n\r\nThis could solve issues such as [this one](https://github.com/ikvmnet/ikvm/issues/448)\r\n\r\nAnd/or we may use JNA to get dependency libraries properly added as well, as noted [here](https://stackoverflow.com/a/10138434/1047713). Some field in `META-INF/MANIFEST` might serve good for this purpose\r\n\r\nHowever, the default strategy should be to load all the native present in the given artifact, in case there are more than one\r\n\r\nLooking forwards for feedbacks :)","author":{"url":"https://github.com/elect86","@type":"Person","name":"elect86"},"datePublished":"2024-04-22T09:41:00.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/50/native-lib-loader/issues/50"}