GDAL upgrade

[ghost]

Continuing the conversation started here.

I had submitted changes to GDAL to clean up the generation of the JNI interface shared libraries, to use a single library, which were accepted and incorporated (see OSGeo/gdal#286) starting in release 2.3.2. The Ubuntu 18.04 LTS contains an older version, 2.2.3, and there are 5 JNI shared libraries. The Ubuntu 18.10 version sports version 2.3.2, so it has the single libgdalalljni.so file.

Both the GisInternals binary distribution and the Ubuntu libgdal-java packages include ‘gdal.jar’, and there’s only one class that not included in both (Ubuntu GDAL 2.2.3 includes ‘AsyncReader’). I’m pretty confident that Worldwind will work with both.

One possibility If we want the ERDAS-ECW and MrSID support in Linux is to compile GDAL and enable those two formats, in addition to whatever is available in the distribution. But my brief reading of the process this morning indicates that support needs to be enabled in the core GDAL code if those two formats are to be supported. I have no idea how important Linux is to WorldWind, nor if those two formats are important. We happen to want to use both. Also, I don’t have the bandwidth to try to work with a number of different Linux editions.

NASA originally included the LizardTech MrSID modules in their GDAL release. The licensing requirements haven’t changed as far as I can tell. One can redistribute the object code for free as long as the end product’s about box credits them (MrSID License).

I don’t see the ERDAS-ECW included in the NASA release. The license is similar to the MrSID license, in that redistribution is permitted without charge for the read-only portion, but the end user should include language in the their own license that Intergraph dictates (ERDAS-ECW license).

I’ll try to put something together that’s flexible and allows the user to drop his own GDAL packages in, for both Windows and Linux with x64. For Windows, I'll include all the required DLLs from the GISInternals build, and note the source. For Linux, I would like a decision about whether the MrSID and ERDAS-ECW formats are needed. If so, I'd need to build GDAL accordingly. If not, I'd want to rely on 'gdal.jar' and 'libgdalalljni.so' as supplied with Ubuntu.

[wcmatthysen]

For Linux, I think we can go with the latest Ubuntu package (i.e. 2.3.2) to keep the maintenance as low as possible. I don't know if we want to maintain a separate GDAL package with MrSID and ECW included. We'll then have to be responsible for maintaining that if this is the case. Except if we can in some way allow the user to supply their own GDAL jar and native library. If you are up for managing a custom GDAL package, then that could also be a solution. We'll have to discuss this.

I'm also looking into this issue as I'm playing around with getting Gradle to build the project and I need to link to a GDAL jar for the project to compile. From what I can see in the maven-central repository, there is a 2.3.2 GDAL jar there. I don't see a 2.2.3 jar. If we are going to switch to Gradle, we'll need to build against one of these versions. So, I think we can compile against 2.3.2 which is included in Ubuntu 18.10 as you mentioned. See: https://mvnrepository.com/artifact/org.gdal/gdal

@emxsys, what your thoughts are on this?

[wcmatthysen]

I'm actually running into a couple of issues right now as I'm testing the Gradle build:

/worldwind-java/src/gov/nasa/worldwind/util/gdal/GDALUtils.java:66: error: cannot find symbol
private static class GDALLibraryLoader implements gdal.LibraryLoader
                                                      ^
symbol:   class LibraryLoader
location: class gdal
/worldwind-java/src/gov/nasa/worldwind/util/gdal/GDALUtils.java:126: error: cannot find symbol
                gdal.setLibraryLoader(new GDALLibraryLoader());

From what I can see, the gdal.LibraryLoader class is missing from the published GDAL jar. It looks like the GDAL jar that the WorldWind team built includes this extra class that is not in the published jar.

[gbburkhardt]

I explored working with a GDAL 2.4.0 edition, and managed to get it work on Windows.  I used the GISinternals binary release, which includes the Java interface. The GDAL code included by NASA minimized the need for shared libraries, which I have to admit that I like a lot.  The JNI shared library included all the dependencies.  My old pull request included shared libraries that were similiar -  the JNI shared library had few dependencies. But building GDAL for a large number of formats is time consuming, and a good one is available for Windows, and Ubuntu includes all but the proprietary formats.  We started looking at this again recently, since the formats included my pull request omitted some formats that we now find we need. The GDAL code itself is set up to auto-load the JNI shared library.  It makes System.loadLibrary() calls (or, with 2.3.2, only one) whenever the classes in 'gdal.jar' are loaded, via static initializers.  But for that to work: a)  The JNI shared library needs to be in a directory that's in the 'java.library.path'.  There is a hack in GDALUtils to change the 'java.library.path' on the fly, but I wonder how long that will work. b) On Windows, all the dependent DLLs need to be a directory that's in the PATH environment variable.  When the System.loadLibrary("gdalalljni") call is made, Java doesn't try to find all the dependent shared libaries - the OS does with whatever rules it uses.  For the binary distribution from GISinternals, there a couple dozen DLLs that need to be loaded. c) On Linux, the LD_LIBRARY_PATH environment variable is used to find dependencies. One would like Worldwind to simply find the GDAL libraries and load them.  That was the original philosophy and implementation. But it was aided by reducing the number of shared library dependencies. I'd like to simply use other people's work.  The most straight forward approach is with configuration files.  Please correct me if I'm wrong, but for a deployed application changing environment variables like PATH or LD_LIBRARY_PATH is messy. I don't see how to avoid using a wrapper script to invoke the executable after setting the variables. I think that for Windows, we should supply a set of DLLs (the size is small), but make it possible to use an installed set of files from GISinternals or elsewhere. My original pull request optionally made use of a list of DLLs to load from an AvKey lookup using the Configuration class.  So the list of DLLs would go in 'worldwind.xml', and it would be order dependent.  Each library could only depend on libraries loaded earlier, or system libraries. The alternative is to require users to set environment variables (PATH, LD_LIBRARY_PATH).  During development that's not so bad, but using that technique for deployment is messier.  If setting those variables were required, all the code in Worldwind that preloads the GDAL shared libraries could simply be deleted - the first time a class in 'gdal.jar' were loaded, the static initializer would load the JNI shared library, and the OS would find all the dependent libraries.  Despite the appeal of this approach, I don't like it so much because of the deployment issues.

…

On 4/16/2019 3:47 PM, Wiehann Matthysen wrote: For Linux, I think we can go with the latest Ubuntu package (i.e. 2.3.2) to keep the maintenance as low as possible. I don't know if we want to maintain a separate GDAL package with MrSID and ECW included. We'll then have to be responsible for maintaining that if this is the case. Except if we can in some way allow the user to supply their own GDAL jar and native library. If you are up for managing this, then that could also be a solution. We'll have to discuss this. I'm also looking into this issue as I'm playing around with getting Gradle to build the project and I need to link to a GDAL jar for the project to compile. From what I can see in the maven-central repository, there is a |2.3.2| GDAL jar there. I don't see a |2.2.3| jar. If we are going to switch to Gradle, we'll need to build against one of these versions. So, I think we can compile against |2.3.2| which is included in Ubuntu 18.10 as you mentioned. See: https://mvnrepository.com/artifact/org.gdal/gdal @emxsys <https://github.com/emxsys>, what your thoughts are on this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFHjlB-KxEnrA3JNDpbKHwX20A3CH-EHks5vhijHgaJpZM4cp0kN>.

[ghost]

Yeah, I don’t see that class in any of the Java interface code that’s currently being generated. When I did the upgrade to version 2.2.3, I deleted that class, probably because I couldn’t find it in the SWIG generated code. If you want use the GDAL 2.3.2, I suggest that you use code from my pull request (NASAWorldWind#146), or wait until I get the version 2.4.0 code packaged. From: Wiehann Matthysen [mailto:[email protected]] Sent: Tuesday, April 16, 2019 15:56 To: WorldWindEarth/WorldWindJava <[email protected]> Cc: Burkhardt, Glenn B Collins <[email protected]>; Author <[email protected]> Subject: [External] Re: [WorldWindEarth/WorldWindJava] GDAL upgrade (#17) I'm actually running into a couple of issues right now as I'm testing the Gradle build: /worldwind-java/src/gov/nasa/worldwind/util/gdal/GDALUtils.java:66: error: cannot find symbol private static class GDALLibraryLoader implements gdal.LibraryLoader ^ symbol: class LibraryLoader location: class gdal /worldwind-java/src/gov/nasa/worldwind/util/gdal/GDALUtils.java:126: error: cannot find symbol gdal.setLibraryLoader(new GDALLibraryLoader()); From what I can see, the gdal.LibraryLoader class is missing from the published GDAL jar. It looks like the GDAL jar that the WorldWind team built includes this extra class that is not in the published jar. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_WorldWindEarth_WorldWindJava_issues_17-23issuecomment-2D483820159&d=DwMFaQ&c=ilBQI1lupc9Y65XwNblLtw&r=aFlNQ5B_1gGdXF59InopHqCPNu3qpTYWNwT1L2Ruc04&m=O_21rIuAxYF6W7sVY4pe39RcyDHmPSfwA4siT0dUKSk&s=NKEaP0TzosO0k9Y4rqvsChdFBGY8AzHvOveoNVN0tfQ&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AfWbIllXJwcxlz-5FTJl7F892qwema5cvPks5vhirqgaJpZM4cp0kN&d=DwMFaQ&c=ilBQI1lupc9Y65XwNblLtw&r=aFlNQ5B_1gGdXF59InopHqCPNu3qpTYWNwT1L2Ruc04&m=O_21rIuAxYF6W7sVY4pe39RcyDHmPSfwA4siT0dUKSk&s=g3ucKDXp783JoJOz8ZaNWQQbXE-HsFCkRqiSR__1Hu0&e=>.

[gbburkhardt]

Please help me decide what are reasonable setup restrictions and what's not. Let me first note that Java doesn't have builtin methods to set the global environment variables of the JVM (and I think that's unreasonable), and changing the system 'java.library.path' requires a hack.

The JNI shared library(s) need to be in a directory listed in 'java.library.path'. There are a couple ways to ensure this.
a) Rely on the user to set it to include the directory prior to running a WWJ application.
b) Configure a directory in the 'worldwind.xml' file using something like
AVKey.GDAL_PATH= "gov.nasa.worldwind.avkey.GDAL.Path", and alter the Java property
'java.libary.path' using GDALUtils.alterJavaLibraryPath(). That method works in Java 11, but its use
generates an error message about "illegal reflective access", and a note that this won't work in a future
Java release.

In addition, there are pre-platform restrictions.

• Windows requires all of the dependent DLLs to be locatable via the PATH variable, or to be in the directory from which the application was loaded.
• Linux requires all dependent shared libraries to be locatable via the LD_LIBRARY_PATH variable. When it is set, the JVM prepends the list to 'java.library.path'.

I had wanted to be able to isolate all the shared libraries, at least on Windows, in a single directory for easier packaging and easier upgrades. But I'm beginning to give in on that point. I think that most Windows applications have all the required DLLs in the same directory as the application, and although there are 38 DLLs in the GISInternals release, I guess I'm willing to put up with a bit of a mess.

If those conditions are met (Windows: 'java.libary.path' set to the location of the JNI DLLs, and all DLLs in the same directory as the application or on the PATH; Linux: LD_LIBRARY_PATH set to the location of JNI shared libraries), no special library load code is needed. The GDAL class in 'gdal.jar' will automatically load the shared libraries on first use.

One would probably still want to do some searching for 'gdal_data' and 'gdalplugins' if the environment variables are not set.

My inclination is to use the builtin features of the GDAL class to manage the shared library loading, and require the user to set 'java_library_path' or 'LD_LIBRARY_PATH' properly when running an application.

[wcmatthysen]

Hi Glenn,

What I understand from your post is this: the issue that makes it difficult now is that there is no longer a single gdal DLL (or .so) file to consider, but a couple of shared DLLs as well (38 in the case of GISInternals). The reason being that we no longer build a single native library containing all of the code as this becomes too much effort to maintain in the long run (as we discussed). However, when there was only a single native DLL it could be easily located and loaded via the WorldWind GDALUtils class. I see there are methods for finding the gdal DLL, and like you mentioned, methods to append the folder of the DLL to the java.library.path variable.

From what I understand, with shared DLLs, those DLLs need to be located via the LD_LIBRARY_PATH variable. However, the main gdal DLL needs to be located via java.library.path to be loaded via System.loadLibrary().

Thus, I think for our purposes we can re-use the existing GDALUtils class to load the main gdal DLL. This will ensure that the main gdal DLL's path is added to the java.library.path variable (via GDALUtils.alterJavaLibraryPath()) and just explicitly specify that the PATH or LD_LIBRARY_PATH variables be correctly configured to contain the folder locations to the other shared libraries. We can add all of these instructions to a README file if necessary. The user has to make sure that his/her GDAL installation is correctly configured and that the LD_LIBRARY_PATH is correctly set-up so that the native libraries can be found.

With regards to gdal_data and gdalplugins, I see that there is logic to find the gdal_data folder if you look at the findGdalDataFolder() method in GDALUtils. I don't see a method to find the gdalplugins folder, so we might have to look at that.

[wcmatthysen]

With regards to the illegal reflective access message that you get in Java 11 when modifying the java.library.path property, we are probably going to get this error message with the JOGL dependencies as well. As far as I understand the JOGL native libraries are unpacked from their jars and added to a temporary directory and this temporary directory is included in the java.library.path so that the System.loadLibrary() calls can pick them up.

We are probably going to have to address all of these native library loading issues in the future as one big issue, especially in the case where Java modules come in. From what I understand (according to @sbwong) in: NASAWorldWind#1, is that JOGL is already breaking when using modules in a Java 9+ environment.

[gbburkhardt]

One definitely gets the "illegal reflective access' message when using JOGL 2.3.2 with Java 11.

[gbburkhardt]

In the version of GDAL that NASA incorporated into Worldwind, all the GDAL JNI code and all dependent libraries were linked into the single shared library, so loading was simplified. That made distribution and configuration easy, but doesn't allow for the use of GDAL libraries built by third parties, e.g., GISInternals or Ubuntu. I think we want to avoid maintaining a set of GDAL libraries for WorldWindJava, and rely on others to supply them.

[wcmatthysen]

@gbburkhardt, agreed. I think we should avoid bundling everything into a single native library. The better approach is like we discussed: to rely on the libraries distributed in GISInternals or Ubuntu.

[wcmatthysen]

@gbburkhardt, the GDAL changes have been merged into develop.